[2026-04-14T05:10:34.648Z] Started by upstream project "anvil/anvil-pipeline/PR-999" build number 22 [2026-04-14T05:10:34.648Z] originally caused by: [2026-04-14T05:10:34.648Z] GitHub pull request comment by ylei-tsubame: retest this please (https://github.com/ClusterLabs/anvil/pull/999#issuecomment-4241420647) [2026-04-14T05:10:35.255Z] Obtained pipelines/projects/anvil/build-functional-testing/Jenkinsfile from git https://github.com/kronosnet/ci-tools [2026-04-14T05:10:35.263Z] Loading library ProjectLib@main:pipelines/projects/anvil/ [2026-04-14T05:10:35.443Z] Examining kronosnet/ci-tools [2026-04-14T05:10:35.443Z] Attempting to resolve main as a branch [2026-04-14T05:10:35.572Z] Resolved main as branch main at revision 122331b302b30271242a259844a3b4a95de3e23f [2026-04-14T05:10:35.573Z] Selected Git installation does not exist. Using Default [2026-04-14T05:10:35.573Z] The recommended git tool is: NONE [2026-04-14T05:10:35.573Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-14T05:10:35.575Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-14T05:10:35.575Z] Wiping out workspace first. [2026-04-14T05:10:35.598Z] Cloning the remote Git repository [2026-04-14T05:10:35.598Z] Using shallow clone with depth 1 [2026-04-14T05:10:35.598Z] Avoid fetching tags [2026-04-14T05:10:35.598Z] Honoring refspec on initial clone [2026-04-14T05:10:35.598Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-14T05:10:35.598Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@3 # timeout=10 [2026-04-14T05:10:35.607Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-14T05:10:35.607Z] > git --version # timeout=10 [2026-04-14T05:10:35.615Z] > git --version # 'git version 2.47.3' [2026-04-14T05:10:35.615Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-14T05:10:35.621Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-14T05:10:35.799Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-14T05:10:35.806Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-14T05:10:35.814Z] Avoid second fetch [2026-04-14T05:10:35.815Z] Checking out Revision 122331b302b30271242a259844a3b4a95de3e23f (main) [2026-04-14T05:10:35.815Z] > git config core.sparsecheckout # timeout=10 [2026-04-14T05:10:35.822Z] > git checkout -f 122331b302b30271242a259844a3b4a95de3e23f # timeout=10 [2026-04-14T05:10:35.853Z] Commit message: "fn-testing: move aws to us-west-2" [2026-04-14T05:10:35.859Z] Loading library GlobalLib@main:./pipelines/libs/global/ [2026-04-14T05:10:36.019Z] Examining kronosnet/ci-tools [2026-04-14T05:10:36.019Z] Attempting to resolve main as a branch [2026-04-14T05:10:36.150Z] Resolved main as branch main at revision 122331b302b30271242a259844a3b4a95de3e23f [2026-04-14T05:10:36.151Z] Selected Git installation does not exist. Using Default [2026-04-14T05:10:36.151Z] The recommended git tool is: NONE [2026-04-14T05:10:36.151Z] using credential 93a393a1-7fe9-45ae-ade0-5a533c87d8d9 [2026-04-14T05:10:36.153Z] DEPRECATED: The 'Wipe out repository & force clone' extension is deprecated for Pipeline jobs. Pipeline users should use the deleteDir() step instead. [2026-04-14T05:10:36.153Z] Wiping out workspace first. [2026-04-14T05:10:36.178Z] Cloning the remote Git repository [2026-04-14T05:10:36.178Z] Using shallow clone with depth 1 [2026-04-14T05:10:36.178Z] Avoid fetching tags [2026-04-14T05:10:36.178Z] Honoring refspec on initial clone [2026-04-14T05:10:36.179Z] Cloning repository https://github.com/kronosnet/ci-tools.git [2026-04-14T05:10:36.179Z] > git init /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@libs/a38125f6c167b5c61aae7e05cd041ecf55e96373eebcceb3c4473bea537ec70b@5 # timeout=10 [2026-04-14T05:10:36.188Z] Fetching upstream changes from https://github.com/kronosnet/ci-tools.git [2026-04-14T05:10:36.188Z] > git --version # timeout=10 [2026-04-14T05:10:36.195Z] > git --version # 'git version 2.47.3' [2026-04-14T05:10:36.195Z] using GIT_ASKPASS to set credentials getCredUUID - github [2026-04-14T05:10:36.202Z] > git fetch --no-tags --force --progress --depth=1 -- https://github.com/kronosnet/ci-tools.git +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-14T05:10:36.376Z] > git config remote.origin.url https://github.com/kronosnet/ci-tools.git # timeout=10 [2026-04-14T05:10:36.383Z] > git config --add remote.origin.fetch +refs/heads/main:refs/remotes/origin/main # timeout=10 [2026-04-14T05:10:36.394Z] Avoid second fetch [2026-04-14T05:10:36.394Z] Checking out Revision 122331b302b30271242a259844a3b4a95de3e23f (main) [2026-04-14T05:10:36.394Z] > git config core.sparsecheckout # timeout=10 [2026-04-14T05:10:36.401Z] > git checkout -f 122331b302b30271242a259844a3b4a95de3e23f # timeout=10 [2026-04-14T05:10:36.433Z] Commit message: "fn-testing: move aws to us-west-2" [2026-04-14T05:10:36.494Z] [Pipeline] Start of Pipeline [2026-04-14T05:10:36.585Z] [Pipeline] node [2026-04-14T05:10:36.587Z] Running on anvil-ci-bm-phy03 in /srv/workspace/anvil/anvil-build-functional-testing [2026-04-14T05:10:36.588Z] [Pipeline] { [2026-04-14T05:10:36.596Z] [Pipeline] stage [2026-04-14T05:10:36.596Z] [Pipeline] { (Prep build env) [2026-04-14T05:10:36.605Z] [Pipeline] script [2026-04-14T05:10:36.606Z] [Pipeline] { [2026-04-14T05:10:36.610Z] [Pipeline] cleanWs [2026-04-14T05:10:36.671Z] [WS-CLEANUP] Deleting project workspace... [2026-04-14T05:10:36.671Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-14T05:10:36.701Z] [WS-CLEANUP] done [2026-04-14T05:10:36.725Z] [Pipeline] lock [2026-04-14T05:10:36.725Z] Trying to acquire lock on [Resource: MAINTENANCE LOCKOUT] [2026-04-14T05:10:36.725Z] Resource [MAINTENANCE LOCKOUT] did not exist. Created. [2026-04-14T05:10:36.725Z] Lock acquired on [Resource: MAINTENANCE LOCKOUT] [2026-04-14T05:10:36.725Z] [Pipeline] { [2026-04-14T05:10:36.728Z] [Pipeline] } [2026-04-14T05:10:36.728Z] Lock released on resource [Resource: MAINTENANCE LOCKOUT] [2026-04-14T05:10:36.730Z] [Pipeline] // lock [2026-04-14T05:10:36.735Z] [Pipeline] } [2026-04-14T05:10:36.738Z] [Pipeline] // script [2026-04-14T05:10:36.740Z] [Pipeline] } [2026-04-14T05:10:36.742Z] [Pipeline] // stage [2026-04-14T05:10:36.745Z] [Pipeline] stage [2026-04-14T05:10:36.746Z] [Pipeline] { (Run functional testing) [2026-04-14T05:10:36.753Z] [Pipeline] script [2026-04-14T05:10:36.754Z] [Pipeline] { [2026-04-14T05:10:36.772Z] [Pipeline] echo [2026-04-14T05:10:36.773Z] runWithTimeout 360 [2026-04-14T05:10:36.773Z] [Pipeline] timeout [2026-04-14T05:10:36.773Z] Timeout set to expire in 6 hr 0 min [2026-04-14T05:10:36.774Z] [Pipeline] { [2026-04-14T05:10:36.779Z] [Pipeline] sh [2026-04-14T05:10:37.315Z] + /root/ci-tools/ci-wrap ci-setup-anvil-bm-vm almalinux-9 enterprise pr/999 all [2026-04-14T05:10:37.315Z] CITBRANCH: main [2026-04-14T05:10:37.315Z] CITHOME: /root/ci-tools [2026-04-14T05:10:37.315Z] CITCMD: ci-setup-anvil-bm-vm [2026-04-14T05:10:37.315Z] CITCMDARGS: almalinux-9 enterprise pr/999 all [2026-04-14T05:10:37.683Z] Genering ssh keys and ssh config [2026-04-14T05:10:37.683Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:10:37.683Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:10:38.052Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:10:38.052Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:10:38.052Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:10:38.052Z] Delay build by 180 seconds to allow Jenkins to cancel the job [2026-04-14T05:13:47.882Z] Destroying VMs (if any) [2026-04-14T05:13:47.882Z] Warning: Permanently added 'an-striker01' (ED25519) to the list of known hosts. [2026-04-14T05:13:47.882Z] Disconnecting an-striker01.ci.alteeve.com from Red Hat. [2026-04-14T05:13:47.882Z] This might take a few seconds. [2026-04-14T05:13:47.882Z] [2026-04-14T05:13:47.882Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-14T05:13:47.882Z] ! Cannot disconnect from Red Hat Insights [2026-04-14T05:13:47.882Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-14T05:13:47.882Z] [2026-04-14T05:13:47.882Z] Manage your connected systems: https://red.ht/connector [2026-04-14T05:13:47.882Z] [2026-04-14T05:13:47.882Z] The following errors were encountered during disconnect: [2026-04-14T05:13:47.882Z] [2026-04-14T05:13:47.882Z] TYPE STEP ERROR [2026-04-14T05:13:47.882Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-14T05:13:47.882Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-14T05:13:51.938Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-14T05:13:51.938Z] [2026-04-14T05:13:51.938Z] 1. Register with subscription-manager [2026-04-14T05:13:51.938Z] # subscription-manager register [2026-04-14T05:13:51.938Z] [2026-04-14T05:13:51.938Z] 2. Register with insights-client [2026-04-14T05:13:51.938Z] # insights-client --register [2026-04-14T05:13:52.307Z] This system is currently not registered. [2026-04-14T05:13:54.761Z] Domain 'an-striker01' destroyed [2026-04-14T05:13:54.761Z] [2026-04-14T05:13:54.761Z] Domain 'an-striker01' has been undefined [2026-04-14T05:13:54.761Z] [2026-04-14T05:13:55.129Z] Warning: Permanently added 'an-striker02' (ED25519) to the list of known hosts. [2026-04-14T05:13:55.129Z] Disconnecting an-striker02.ci.alteeve.com from Red Hat. [2026-04-14T05:13:55.129Z] This might take a few seconds. [2026-04-14T05:13:55.129Z] [2026-04-14T05:13:55.129Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-14T05:14:00.195Z] ! Cannot disconnect from Red Hat Insights [2026-04-14T05:14:00.195Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-14T05:14:00.195Z] [2026-04-14T05:14:00.195Z] Manage your connected systems: https://red.ht/connector [2026-04-14T05:14:00.195Z] [2026-04-14T05:14:00.195Z] The following errors were encountered during disconnect: [2026-04-14T05:14:00.195Z] [2026-04-14T05:14:00.195Z] TYPE STEP ERROR [2026-04-14T05:14:00.195Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-14T05:14:00.195Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-14T05:14:04.535Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-14T05:14:04.535Z] [2026-04-14T05:14:04.535Z] 1. Register with subscription-manager [2026-04-14T05:14:04.535Z] # subscription-manager register [2026-04-14T05:14:04.535Z] [2026-04-14T05:14:04.535Z] 2. Register with insights-client [2026-04-14T05:14:04.535Z] # insights-client --register [2026-04-14T05:14:04.902Z] This system is currently not registered. [2026-04-14T05:14:07.353Z] Domain 'an-striker02' destroyed [2026-04-14T05:14:07.353Z] [2026-04-14T05:14:07.353Z] Domain 'an-striker02' has been undefined [2026-04-14T05:14:07.353Z] [2026-04-14T05:14:07.353Z] Warning: Permanently added 'an-a01n01' (ED25519) to the list of known hosts. [2026-04-14T05:14:07.728Z] Disconnecting an-a01n01.ci.alteeve.com from Red Hat. [2026-04-14T05:14:07.728Z] This might take a few seconds. [2026-04-14T05:14:07.728Z] [2026-04-14T05:14:07.728Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-14T05:14:11.782Z] ! Cannot disconnect from Red Hat Insights [2026-04-14T05:14:11.782Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-14T05:14:11.782Z] [2026-04-14T05:14:11.782Z] Manage your connected systems: https://red.ht/connector [2026-04-14T05:14:11.782Z] [2026-04-14T05:14:11.782Z] The following errors were encountered during disconnect: [2026-04-14T05:14:11.782Z] [2026-04-14T05:14:11.782Z] TYPE STEP ERROR [2026-04-14T05:14:11.782Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-14T05:14:11.782Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-14T05:14:15.836Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-14T05:14:15.836Z] [2026-04-14T05:14:15.836Z] 1. Register with subscription-manager [2026-04-14T05:14:15.836Z] # subscription-manager register [2026-04-14T05:14:15.836Z] [2026-04-14T05:14:15.836Z] 2. Register with insights-client [2026-04-14T05:14:15.836Z] # insights-client --register [2026-04-14T05:14:16.205Z] This system is currently not registered. [2026-04-14T05:14:20.720Z] Domain 'an-a01n01' destroyed [2026-04-14T05:14:20.720Z] [2026-04-14T05:14:20.720Z] Domain 'an-a01n01' has been undefined [2026-04-14T05:14:20.720Z] [2026-04-14T05:14:20.720Z] Warning: Permanently added 'an-a01n02' (ED25519) to the list of known hosts. [2026-04-14T05:14:21.088Z] Disconnecting an-a01n02.ci.alteeve.com from Red Hat. [2026-04-14T05:14:21.088Z] This might take a few seconds. [2026-04-14T05:14:21.088Z] [2026-04-14T05:14:21.088Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-14T05:14:25.152Z] ! Cannot disconnect from Red Hat Insights [2026-04-14T05:14:25.152Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-14T05:14:25.152Z] [2026-04-14T05:14:25.152Z] Manage your connected systems: https://red.ht/connector [2026-04-14T05:14:25.152Z] [2026-04-14T05:14:25.152Z] The following errors were encountered during disconnect: [2026-04-14T05:14:25.152Z] [2026-04-14T05:14:25.152Z] TYPE STEP ERROR [2026-04-14T05:14:25.152Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-14T05:14:25.152Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-14T05:14:29.216Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-14T05:14:29.216Z] [2026-04-14T05:14:29.216Z] 1. Register with subscription-manager [2026-04-14T05:14:29.216Z] # subscription-manager register [2026-04-14T05:14:29.216Z] [2026-04-14T05:14:29.216Z] 2. Register with insights-client [2026-04-14T05:14:29.216Z] # insights-client --register [2026-04-14T05:14:29.585Z] This system is currently not registered. [2026-04-14T05:14:33.640Z] Domain 'an-a01n02' destroyed [2026-04-14T05:14:33.640Z] [2026-04-14T05:14:33.640Z] Domain 'an-a01n02' has been undefined [2026-04-14T05:14:33.640Z] [2026-04-14T05:14:33.640Z] Warning: Permanently added 'an-a01dr01' (ED25519) to the list of known hosts. [2026-04-14T05:14:33.640Z] Disconnecting an-a01dr01.ci.alteeve.com from Red Hat. [2026-04-14T05:14:33.640Z] This might take a few seconds. [2026-04-14T05:14:33.640Z] [2026-04-14T05:14:33.640Z] 𐄂 Deactivated the Remote Host Configuration daemon [2026-04-14T05:14:37.691Z] ! Cannot disconnect from Red Hat Insights [2026-04-14T05:14:37.691Z] ! Cannot disconnect from Red Hat Subscription Management [2026-04-14T05:14:37.691Z] [2026-04-14T05:14:37.691Z] Manage your connected systems: https://red.ht/connector [2026-04-14T05:14:37.691Z] [2026-04-14T05:14:37.691Z] The following errors were encountered during disconnect: [2026-04-14T05:14:37.691Z] [2026-04-14T05:14:37.691Z] TYPE STEP ERROR [2026-04-14T05:14:37.691Z] ERROR insights cannot disconnect from Red Hat Insights: exit status 1 [2026-04-14T05:14:37.691Z] ERROR rhsm cannot disconnect from Red Hat Subscription Management: warning: the system is already unregistered [2026-04-14T05:14:41.332Z] This host is unregistered, please ensure that the system is registered with subscription-manager and then with insights-client. [2026-04-14T05:14:41.332Z] [2026-04-14T05:14:41.332Z] 1. Register with subscription-manager [2026-04-14T05:14:41.332Z] # subscription-manager register [2026-04-14T05:14:41.332Z] [2026-04-14T05:14:41.332Z] 2. Register with insights-client [2026-04-14T05:14:41.332Z] # insights-client --register [2026-04-14T05:14:42.117Z] This system is currently not registered. [2026-04-14T05:14:45.305Z] Domain 'an-a01dr01' destroyed [2026-04-14T05:14:45.305Z] [2026-04-14T05:14:45.305Z] Domain 'an-a01dr01' has been undefined [2026-04-14T05:14:45.305Z] [2026-04-14T05:14:45.305Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-14T05:14:45.305Z] Genering ssh keys and ssh config [2026-04-14T05:14:45.305Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:14:45.305Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:14:45.673Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:14:45.673Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:14:45.673Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:14:46.042Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-14T05:14:46.042Z] receiving incremental file list [2026-04-14T05:14:46.042Z] [2026-04-14T05:14:46.042Z] sent 27 bytes received 442 bytes 938.00 bytes/sec [2026-04-14T05:14:46.042Z] total size is 16,565,796,937 speedup is 35,321,528.65 [2026-04-14T05:14:46.042Z] Downloading latest AlmaLinux 9 cloud image [2026-04-14T05:14:46.471Z] Generating daily cache [2026-04-14T05:14:46.471Z] Updating daily cache for almalinux-9 2026-04-14 [2026-04-14T05:14:46.471Z] Destroying VMs (if any) [2026-04-14T05:14:46.471Z] Formatting 'almalinux-9-latest.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/images/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:14:46.471Z] wrote an-cache-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:14:48.288Z] [2026-04-14T05:14:48.288Z] Starting install... [2026-04-14T05:14:48.288Z] [2026-04-14T05:14:48.288Z] Creating domain... | 00:00 [2026-04-14T05:14:48.288Z] Domain creation completed. [2026-04-14T05:14:48.288Z] Waiting for nodes and cloud-init to complete full init [2026-04-14T05:14:51.478Z] cloud-init status node an-cache: [2026-04-14T05:14:51.478Z] Tue Apr 14 01:14:51 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:05.887Z] cloud-init status node an-cache: [2026-04-14T05:15:05.887Z] Tue Apr 14 01:15:02 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:13.836Z] cloud-init status node an-cache: running [2026-04-14T05:15:13.836Z] Tue Apr 14 01:15:13 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:25.141Z] cloud-init status node an-cache: running [2026-04-14T05:15:25.141Z] Tue Apr 14 01:15:24 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:36.443Z] cloud-init status node an-cache: running [2026-04-14T05:15:36.443Z] Tue Apr 14 01:15:34 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:46.680Z] cloud-init status node an-cache: running [2026-04-14T05:15:46.680Z] Tue Apr 14 01:15:45 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:15:58.018Z] cloud-init status node an-cache: running [2026-04-14T05:15:58.018Z] Tue Apr 14 01:15:56 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:16:07.460Z] cloud-init status node an-cache: running [2026-04-14T05:16:07.460Z] Tue Apr 14 01:16:07 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:16:19.420Z] cloud-init status node an-cache: running [2026-04-14T05:16:19.420Z] Tue Apr 14 01:16:17 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:16:29.268Z] cloud-init status node an-cache: running [2026-04-14T05:16:29.268Z] Tue Apr 14 01:16:28 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:16:40.553Z] cloud-init status node an-cache: running [2026-04-14T05:16:40.553Z] Tue Apr 14 01:16:39 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:16:50.946Z] cloud-init status node an-cache: running [2026-04-14T05:16:50.946Z] Tue Apr 14 01:16:50 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:02.238Z] cloud-init status node an-cache: running [2026-04-14T05:17:02.239Z] Tue Apr 14 01:17:00 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:13.523Z] cloud-init status node an-cache: running [2026-04-14T05:17:13.523Z] Tue Apr 14 01:17:11 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:23.598Z] cloud-init status node an-cache: running [2026-04-14T05:17:23.598Z] Tue Apr 14 01:17:22 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:34.956Z] cloud-init status node an-cache: running [2026-04-14T05:17:34.956Z] Tue Apr 14 01:17:33 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:44.340Z] cloud-init status node an-cache: running [2026-04-14T05:17:44.340Z] Tue Apr 14 01:17:43 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:17:56.180Z] cloud-init status node an-cache: running [2026-04-14T05:17:56.180Z] Tue Apr 14 01:17:54 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:18:05.587Z] cloud-init status node an-cache: running [2026-04-14T05:18:05.587Z] Tue Apr 14 01:18:05 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:18:16.905Z] cloud-init status node an-cache: running [2026-04-14T05:18:16.905Z] Tue Apr 14 01:18:16 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:18:27.325Z] cloud-init status node an-cache: running [2026-04-14T05:18:27.326Z] Tue Apr 14 01:18:26 EDT 2026 actual: 0 expected: 1 [2026-04-14T05:18:38.612Z] cloud-init status node an-cache: done [2026-04-14T05:18:38.612Z] Tue Apr 14 01:18:37 EDT 2026 actual: 1 expected: 1 [2026-04-14T05:20:18.162Z] bash: line 1: rhc: command not found [2026-04-14T05:20:18.162Z] bash: line 1: insights-client: command not found [2026-04-14T05:20:18.162Z] bash: line 1: subscription-manager: command not found [2026-04-14T05:20:18.162Z] Domain 'an-cache' is being shutdown [2026-04-14T05:20:18.162Z] [2026-04-14T05:20:18.947Z] Domain 'an-cache' has been undefined [2026-04-14T05:20:18.947Z] [2026-04-14T05:20:18.947Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:20:18.947Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:20:19.315Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:20:19.315Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:20:19.315Z] # github.com:22 SSH-2.0-3992d52 [2026-04-14T05:20:19.315Z] Done updating daily cache [2026-04-14T05:20:19.315Z] Creating nodes disks [2026-04-14T05:20:25.566Z] Formatting 'an-striker01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-striker01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-striker02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-striker02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01n01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01n01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01n02-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01n02-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01dr01-vol.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 backing_file=/srv/anvil/cache/almalinux-9-latest.qcow2 backing_fmt=qcow2 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Formatting 'an-a01dr01-vol2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=214748364800 lazy_refcounts=off refcount_bits=16 [2026-04-14T05:20:25.566Z] Generating an-test-deployX kickstart files/isos [2026-04-14T05:20:25.566Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy1.iso' [2026-04-14T05:20:25.566Z] Media current: stdio file, overwriteable [2026-04-14T05:20:25.566Z] Media status : is blank [2026-04-14T05:20:25.566Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1725g free [2026-04-14T05:20:25.566Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks1' [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] ISO image produced: 184 sectors [2026-04-14T05:20:25.566Z] Written to medium : 184 sectors at LBA 0 [2026-04-14T05:20:25.566Z] Writing to 'stdio:/srv/anvil/snapshots/deploy1.iso' completed successfully. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy2.iso' [2026-04-14T05:20:25.566Z] Media current: stdio file, overwriteable [2026-04-14T05:20:25.566Z] Media status : is blank [2026-04-14T05:20:25.566Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-14T05:20:25.566Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks2' [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] ISO image produced: 184 sectors [2026-04-14T05:20:25.566Z] Written to medium : 184 sectors at LBA 0 [2026-04-14T05:20:25.566Z] Writing to 'stdio:/srv/anvil/snapshots/deploy2.iso' completed successfully. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy3.iso' [2026-04-14T05:20:25.566Z] Media current: stdio file, overwriteable [2026-04-14T05:20:25.566Z] Media status : is blank [2026-04-14T05:20:25.566Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-14T05:20:25.566Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks3' [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] ISO image produced: 184 sectors [2026-04-14T05:20:25.566Z] Written to medium : 184 sectors at LBA 0 [2026-04-14T05:20:25.566Z] Writing to 'stdio:/srv/anvil/snapshots/deploy3.iso' completed successfully. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy4.iso' [2026-04-14T05:20:25.566Z] Media current: stdio file, overwriteable [2026-04-14T05:20:25.566Z] Media status : is blank [2026-04-14T05:20:25.566Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-14T05:20:25.566Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks4' [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] ISO image produced: 184 sectors [2026-04-14T05:20:25.566Z] Written to medium : 184 sectors at LBA 0 [2026-04-14T05:20:25.566Z] Writing to 'stdio:/srv/anvil/snapshots/deploy4.iso' completed successfully. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] xorriso 1.5.4 : RockRidge filesystem manipulator, libburnia project. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Drive current: -outdev 'stdio:/srv/anvil/snapshots/deploy5.iso' [2026-04-14T05:20:25.566Z] Media current: stdio file, overwriteable [2026-04-14T05:20:25.566Z] Media status : is blank [2026-04-14T05:20:25.566Z] Media summary: 0 sessions, 0 data blocks, 0 data, 1731g free [2026-04-14T05:20:25.566Z] Added to ISO image: directory '/'='/srv/anvil/configs/ks5' [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] xorriso : UPDATE : 1 files added in 1 seconds [2026-04-14T05:20:25.566Z] ISO image produced: 184 sectors [2026-04-14T05:20:25.566Z] Written to medium : 184 sectors at LBA 0 [2026-04-14T05:20:25.566Z] Writing to 'stdio:/srv/anvil/snapshots/deploy5.iso' completed successfully. [2026-04-14T05:20:25.566Z] [2026-04-14T05:20:25.566Z] Generating cloud-init configs [2026-04-14T05:20:25.566Z] Generating cloud-init metadata and seed images [2026-04-14T05:20:25.933Z] wrote /srv/anvil/snapshots/an-striker01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:20:25.933Z] wrote /srv/anvil/snapshots/an-striker02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:20:25.933Z] wrote /srv/anvil/snapshots/an-a01n01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:20:26.301Z] wrote /srv/anvil/snapshots/an-a01n02-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:20:26.301Z] wrote /srv/anvil/snapshots/an-a01dr01-seed.img with filesystem=iso9660 and diskformat=raw [2026-04-14T05:20:26.301Z] Creating VMs [2026-04-14T05:20:28.115Z] [2026-04-14T05:20:28.115Z] Starting install... [2026-04-14T05:20:28.115Z] [2026-04-14T05:20:28.115Z] Creating domain... | 00:00 [2026-04-14T05:20:28.115Z] Domain creation completed. [2026-04-14T05:20:29.378Z] [2026-04-14T05:20:29.378Z] Starting install... [2026-04-14T05:20:29.378Z] [2026-04-14T05:20:29.378Z] Creating domain... | 00:00 [2026-04-14T05:20:29.378Z] Domain creation completed. [2026-04-14T05:20:31.191Z] [2026-04-14T05:20:31.191Z] Starting install... [2026-04-14T05:20:31.191Z] [2026-04-14T05:20:31.191Z] Creating domain... | 00:00 [2026-04-14T05:20:31.191Z] Domain creation completed. [2026-04-14T05:20:33.409Z] [2026-04-14T05:20:33.409Z] Starting install... [2026-04-14T05:20:33.409Z] [2026-04-14T05:20:33.409Z] Creating domain... | 00:00 [2026-04-14T05:20:33.409Z] Domain creation completed. [2026-04-14T05:20:34.676Z] [2026-04-14T05:20:34.676Z] Starting install... [2026-04-14T05:20:34.676Z] [2026-04-14T05:20:34.676Z] Creating domain... | 00:00 [2026-04-14T05:20:34.676Z] Domain creation completed. [2026-04-14T05:20:34.676Z] Waiting for nodes and cloud-init to complete full init [2026-04-14T05:20:37.625Z] cloud-init status node an-striker01: [2026-04-14T05:20:41.101Z] cloud-init status node an-striker02: [2026-04-14T05:20:43.963Z] cloud-init status node an-a01n01: [2026-04-14T05:20:47.074Z] cloud-init status node an-a01n02: [2026-04-14T05:20:50.351Z] cloud-init status node an-a01dr01: [2026-04-14T05:20:50.351Z] Tue Apr 14 01:20:49 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:21:02.471Z] cloud-init status node an-striker01: running [2026-04-14T05:21:03.269Z] cloud-init status node an-striker02: running [2026-04-14T05:21:04.066Z] cloud-init status node an-a01n01: running [2026-04-14T05:21:05.346Z] cloud-init status node an-a01n02: running [2026-04-14T05:21:06.626Z] cloud-init status node an-a01dr01: running [2026-04-14T05:21:06.626Z] Tue Apr 14 01:21:06 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:21:18.089Z] cloud-init status node an-striker01: running [2026-04-14T05:21:24.867Z] cloud-init status node an-striker02: running [2026-04-14T05:21:24.868Z] cloud-init status node an-a01n01: running [2026-04-14T05:21:25.668Z] cloud-init status node an-a01n02: running [2026-04-14T05:21:26.533Z] cloud-init status node an-a01dr01: running [2026-04-14T05:21:26.533Z] Tue Apr 14 01:21:26 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:21:38.368Z] cloud-init status node an-striker01: running [2026-04-14T05:21:38.745Z] cloud-init status node an-striker02: running [2026-04-14T05:21:39.607Z] cloud-init status node an-a01n01: running [2026-04-14T05:21:40.406Z] cloud-init status node an-a01n02: running [2026-04-14T05:21:41.714Z] cloud-init status node an-a01dr01: running [2026-04-14T05:21:41.714Z] Tue Apr 14 01:21:41 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:21:54.180Z] cloud-init status node an-striker01: running [2026-04-14T05:21:54.180Z] cloud-init status node an-striker02: running [2026-04-14T05:21:54.180Z] cloud-init status node an-a01n01: running [2026-04-14T05:21:55.460Z] cloud-init status node an-a01n02: running [2026-04-14T05:21:56.258Z] cloud-init status node an-a01dr01: running [2026-04-14T05:21:56.258Z] Tue Apr 14 01:21:55 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:22:07.616Z] cloud-init status node an-striker01: running [2026-04-14T05:22:07.990Z] cloud-init status node an-striker02: running [2026-04-14T05:22:08.786Z] cloud-init status node an-a01n01: running [2026-04-14T05:22:10.067Z] cloud-init status node an-a01n02: running [2026-04-14T05:22:10.864Z] cloud-init status node an-a01dr01: running [2026-04-14T05:22:10.864Z] Tue Apr 14 01:22:10 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:22:22.616Z] cloud-init status node an-striker01: running [2026-04-14T05:22:22.616Z] cloud-init status node an-striker02: running [2026-04-14T05:22:23.902Z] cloud-init status node an-a01n01: running [2026-04-14T05:22:24.709Z] cloud-init status node an-a01n02: running [2026-04-14T05:22:25.515Z] cloud-init status node an-a01dr01: running [2026-04-14T05:22:25.515Z] Tue Apr 14 01:22:25 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:22:42.205Z] cloud-init status node an-striker01: running [2026-04-14T05:22:42.578Z] cloud-init status node an-striker02: running [2026-04-14T05:22:43.374Z] cloud-init status node an-a01n01: running [2026-04-14T05:22:44.655Z] cloud-init status node an-a01n02: running [2026-04-14T05:22:45.450Z] cloud-init status node an-a01dr01: running [2026-04-14T05:22:45.450Z] Tue Apr 14 01:22:45 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:22:56.784Z] cloud-init status node an-striker01: running [2026-04-14T05:22:57.153Z] cloud-init status node an-striker02: running [2026-04-14T05:22:58.419Z] cloud-init status node an-a01n01: running [2026-04-14T05:22:59.204Z] cloud-init status node an-a01n02: running [2026-04-14T05:22:59.992Z] cloud-init status node an-a01dr01: running [2026-04-14T05:22:59.992Z] Tue Apr 14 01:22:59 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:23:11.278Z] cloud-init status node an-striker01: running [2026-04-14T05:23:11.646Z] cloud-init status node an-striker02: running [2026-04-14T05:23:12.432Z] cloud-init status node an-a01n01: running [2026-04-14T05:23:13.219Z] cloud-init status node an-a01n02: running [2026-04-14T05:23:14.485Z] cloud-init status node an-a01dr01: running [2026-04-14T05:23:14.485Z] Tue Apr 14 01:23:14 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:23:25.821Z] cloud-init status node an-striker01: running [2026-04-14T05:23:26.190Z] cloud-init status node an-striker02: running [2026-04-14T05:23:27.015Z] cloud-init status node an-a01n01: running [2026-04-14T05:23:27.801Z] cloud-init status node an-a01n02: running [2026-04-14T05:23:29.065Z] cloud-init status node an-a01dr01: running [2026-04-14T05:23:29.065Z] Tue Apr 14 01:23:28 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:23:40.350Z] cloud-init status node an-striker01: running [2026-04-14T05:23:40.350Z] cloud-init status node an-striker02: running [2026-04-14T05:23:41.617Z] cloud-init status node an-a01n01: running [2026-04-14T05:23:42.411Z] cloud-init status node an-a01n02: running [2026-04-14T05:23:43.200Z] cloud-init status node an-a01dr01: running [2026-04-14T05:23:43.200Z] Tue Apr 14 01:23:42 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:23:54.021Z] cloud-init status node an-striker01: running [2026-04-14T05:23:54.808Z] cloud-init status node an-striker02: running [2026-04-14T05:23:55.697Z] cloud-init status node an-a01n01: running [2026-04-14T05:23:56.962Z] cloud-init status node an-a01n02: running [2026-04-14T05:23:57.747Z] cloud-init status node an-a01dr01: running [2026-04-14T05:23:57.747Z] Tue Apr 14 01:23:57 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:24:09.041Z] cloud-init status node an-striker01: running [2026-04-14T05:24:09.411Z] cloud-init status node an-striker02: running [2026-04-14T05:24:10.679Z] cloud-init status node an-a01n01: running [2026-04-14T05:24:11.468Z] cloud-init status node an-a01n02: running [2026-04-14T05:24:12.255Z] cloud-init status node an-a01dr01: running [2026-04-14T05:24:12.255Z] Tue Apr 14 01:24:12 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:24:23.588Z] cloud-init status node an-striker01: running [2026-04-14T05:24:23.960Z] cloud-init status node an-striker02: running [2026-04-14T05:24:25.240Z] cloud-init status node an-a01n01: running [2026-04-14T05:24:26.423Z] cloud-init status node an-a01n02: running [2026-04-14T05:24:32.725Z] cloud-init status node an-a01dr01: running [2026-04-14T05:24:32.725Z] Tue Apr 14 01:24:31 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:24:44.081Z] cloud-init status node an-striker01: running [2026-04-14T05:24:49.240Z] cloud-init status node an-striker02: running [2026-04-14T05:24:49.655Z] cloud-init status node an-a01n01: running [2026-04-14T05:24:51.908Z] cloud-init status node an-a01n02: running [2026-04-14T05:24:51.908Z] cloud-init status node an-a01dr01: running [2026-04-14T05:24:51.908Z] Tue Apr 14 01:24:51 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:25:03.307Z] cloud-init status node an-striker01: running [2026-04-14T05:25:03.307Z] cloud-init status node an-striker02: running [2026-04-14T05:25:04.162Z] cloud-init status node an-a01n01: running [2026-04-14T05:25:05.450Z] cloud-init status node an-a01n02: running [2026-04-14T05:25:06.243Z] cloud-init status node an-a01dr01: running [2026-04-14T05:25:06.243Z] Tue Apr 14 01:25:06 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:25:17.849Z] cloud-init status node an-striker01: running [2026-04-14T05:25:18.293Z] cloud-init status node an-striker02: running [2026-04-14T05:25:19.111Z] cloud-init status node an-a01n01: running [2026-04-14T05:25:20.890Z] cloud-init status node an-a01n02: running [2026-04-14T05:25:21.510Z] cloud-init status node an-a01dr01: running [2026-04-14T05:25:21.510Z] Tue Apr 14 01:25:20 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:25:33.164Z] cloud-init status node an-striker01: running [2026-04-14T05:25:33.164Z] cloud-init status node an-striker02: running [2026-04-14T05:25:34.164Z] cloud-init status node an-a01n01: running [2026-04-14T05:25:40.519Z] cloud-init status node an-a01n02: running [2026-04-14T05:25:40.889Z] cloud-init status node an-a01dr01: running [2026-04-14T05:25:40.889Z] Tue Apr 14 01:25:40 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:25:52.419Z] cloud-init status node an-striker01: running [2026-04-14T05:25:52.788Z] cloud-init status node an-striker02: running [2026-04-14T05:25:54.054Z] cloud-init status node an-a01n01: running [2026-04-14T05:25:54.842Z] cloud-init status node an-a01n02: running [2026-04-14T05:25:55.689Z] cloud-init status node an-a01dr01: running [2026-04-14T05:25:55.689Z] Tue Apr 14 01:25:55 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:26:06.978Z] cloud-init status node an-striker01: running [2026-04-14T05:26:07.346Z] cloud-init status node an-striker02: running [2026-04-14T05:26:08.609Z] cloud-init status node an-a01n01: running [2026-04-14T05:26:09.394Z] cloud-init status node an-a01n02: running [2026-04-14T05:26:10.179Z] cloud-init status node an-a01dr01: running [2026-04-14T05:26:10.179Z] Tue Apr 14 01:26:09 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:26:22.352Z] cloud-init status node an-striker01: running [2026-04-14T05:26:22.352Z] cloud-init status node an-striker02: running [2026-04-14T05:26:22.722Z] cloud-init status node an-a01n01: running [2026-04-14T05:26:24.052Z] cloud-init status node an-a01n02: running [2026-04-14T05:26:24.839Z] cloud-init status node an-a01dr01: running [2026-04-14T05:26:24.839Z] Tue Apr 14 01:26:24 EDT 2026 actual: 0 expected: 5 [2026-04-14T05:26:36.129Z] cloud-init status node an-striker01: running [2026-04-14T05:26:37.394Z] cloud-init status node an-striker02: running [2026-04-14T05:26:40.586Z] cloud-init status node an-a01n01: done [2026-04-14T05:26:42.251Z] cloud-init status node an-a01n02: running [2026-04-14T05:26:43.516Z] cloud-init status node an-a01dr01: done [2026-04-14T05:26:43.516Z] Tue Apr 14 01:26:43 EDT 2026 actual: 2 expected: 5 [2026-04-14T05:26:57.152Z] cloud-init status node an-striker01: running [2026-04-14T05:27:01.211Z] cloud-init status node an-striker02: running [2026-04-14T05:27:02.002Z] cloud-init status node an-a01n01: done [2026-04-14T05:27:03.281Z] cloud-init status node an-a01n02: done [2026-04-14T05:27:04.562Z] cloud-init status node an-a01dr01: done [2026-04-14T05:27:04.562Z] Tue Apr 14 01:27:04 EDT 2026 actual: 3 expected: 5 [2026-04-14T05:27:18.266Z] cloud-init status node an-striker01: running [2026-04-14T05:27:18.266Z] cloud-init status node an-striker02: running [2026-04-14T05:27:18.640Z] cloud-init status node an-a01n01: done [2026-04-14T05:27:19.917Z] cloud-init status node an-a01n02: done [2026-04-14T05:27:20.710Z] cloud-init status node an-a01dr01: done [2026-04-14T05:27:20.710Z] Tue Apr 14 01:27:20 EDT 2026 actual: 3 expected: 5 [2026-04-14T05:27:34.471Z] cloud-init status node an-striker01: done [2026-04-14T05:27:35.747Z] cloud-init status node an-striker02: done [2026-04-14T05:27:36.540Z] cloud-init status node an-a01n01: done [2026-04-14T05:27:38.272Z] cloud-init status node an-a01n02: done [2026-04-14T05:27:38.272Z] cloud-init status node an-a01dr01: done [2026-04-14T05:27:38.272Z] Tue Apr 14 01:27:38 EDT 2026 actual: 5 expected: 5 [2026-04-14T05:27:38.272Z] Building simengine model [2026-04-14T05:27:51.931Z] Generating anvil striker auto config [2026-04-14T05:27:54.382Z] Cloning into 'anvil-test-suite'... [2026-04-14T05:27:55.167Z] generated ansible config: [2026-04-14T05:27:55.167Z] ========================= [2026-04-14T05:27:55.167Z] [defaults] [2026-04-14T05:27:55.167Z] inventory = ansible-inventory [2026-04-14T05:27:55.167Z] forks = 500 [2026-04-14T05:27:55.167Z] any_errors_fatal = true [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:55.167Z] [ssh_connection] [2026-04-14T05:27:55.167Z] retries = 20 [2026-04-14T05:27:55.167Z] ========================= [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:55.167Z] generated ansible inventory: [2026-04-14T05:27:55.167Z] ============================ [2026-04-14T05:27:55.167Z] [strikers] [2026-04-14T05:27:55.167Z] an-striker01 [2026-04-14T05:27:55.167Z] an-striker02 [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:55.167Z] [nodes] [2026-04-14T05:27:55.167Z] an-a01n01 [2026-04-14T05:27:55.167Z] an-a01n02 [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:55.167Z] [drnodes] [2026-04-14T05:27:55.167Z] an-a01dr01 [2026-04-14T05:27:55.167Z] ============================ [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:55.167Z] Checking connectivity to nodes [2026-04-14T05:27:55.167Z] [2026-04-14T05:27:57.617Z] an-a01n01 | SUCCESS => { [2026-04-14T05:27:57.618Z] "ansible_facts": { [2026-04-14T05:27:57.618Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-14T05:27:57.618Z] }, [2026-04-14T05:27:57.618Z] "changed": false, [2026-04-14T05:27:57.618Z] "ping": "pong" [2026-04-14T05:27:57.618Z] } [2026-04-14T05:27:57.618Z] an-a01n02 | SUCCESS => { [2026-04-14T05:27:57.618Z] "ansible_facts": { [2026-04-14T05:27:57.618Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-14T05:27:57.618Z] }, [2026-04-14T05:27:57.618Z] "changed": false, [2026-04-14T05:27:57.618Z] "ping": "pong" [2026-04-14T05:27:57.618Z] } [2026-04-14T05:27:57.618Z] an-a01dr01 | SUCCESS => { [2026-04-14T05:27:57.618Z] "ansible_facts": { [2026-04-14T05:27:57.618Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-14T05:27:57.618Z] }, [2026-04-14T05:27:57.618Z] "changed": false, [2026-04-14T05:27:57.618Z] "ping": "pong" [2026-04-14T05:27:57.618Z] } [2026-04-14T05:27:57.618Z] an-striker01 | SUCCESS => { [2026-04-14T05:27:57.618Z] "ansible_facts": { [2026-04-14T05:27:57.618Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-14T05:27:57.618Z] }, [2026-04-14T05:27:57.618Z] "changed": false, [2026-04-14T05:27:57.618Z] "ping": "pong" [2026-04-14T05:27:57.618Z] } [2026-04-14T05:27:57.618Z] an-striker02 | SUCCESS => { [2026-04-14T05:27:57.618Z] "ansible_facts": { [2026-04-14T05:27:57.618Z] "discovered_interpreter_python": "/usr/bin/python3" [2026-04-14T05:27:57.618Z] }, [2026-04-14T05:27:57.618Z] "changed": false, [2026-04-14T05:27:57.618Z] "ping": "pong" [2026-04-14T05:27:57.618Z] } [2026-04-14T05:27:57.618Z] [2026-04-14T05:27:58.403Z] [2026-04-14T05:27:58.403Z] PLAY [Setup all nodes] ********************************************************* [2026-04-14T05:27:58.403Z] [2026-04-14T05:27:58.403Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:28:00.864Z] ok: [an-a01n01] [2026-04-14T05:28:00.864Z] ok: [an-a01n02] [2026-04-14T05:28:00.864Z] ok: [an-a01dr01] [2026-04-14T05:28:00.864Z] ok: [an-striker01] [2026-04-14T05:28:00.864Z] ok: [an-striker02] [2026-04-14T05:28:00.864Z] [2026-04-14T05:28:00.864Z] TASK [create /opt/MegaRAID/storcli/] ******************************************* [2026-04-14T05:28:02.131Z] changed: [an-a01n01] [2026-04-14T05:28:02.131Z] changed: [an-a01n02] [2026-04-14T05:28:02.131Z] changed: [an-a01dr01] [2026-04-14T05:28:02.131Z] changed: [an-striker01] [2026-04-14T05:28:02.131Z] changed: [an-striker02] [2026-04-14T05:28:02.131Z] [2026-04-14T05:28:02.131Z] TASK [wipe old storcli64] ****************************************************** [2026-04-14T05:28:03.222Z] ok: [an-a01n01] [2026-04-14T05:28:03.222Z] ok: [an-a01dr01] [2026-04-14T05:28:03.222Z] ok: [an-a01n02] [2026-04-14T05:28:03.222Z] ok: [an-striker02] [2026-04-14T05:28:03.222Z] ok: [an-striker01] [2026-04-14T05:28:03.222Z] [2026-04-14T05:28:03.222Z] TASK [get latest Simengine storcli64] ****************************************** [2026-04-14T05:28:05.042Z] changed: [an-a01n01] [2026-04-14T05:28:05.042Z] changed: [an-a01n02] [2026-04-14T05:28:05.042Z] changed: [an-a01dr01] [2026-04-14T05:28:05.042Z] changed: [an-striker02] [2026-04-14T05:28:05.042Z] changed: [an-striker01] [2026-04-14T05:28:05.042Z] [2026-04-14T05:28:05.042Z] PLAY [setup strikers] ********************************************************** [2026-04-14T05:28:05.042Z] [2026-04-14T05:28:05.042Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:28:06.433Z] ok: [an-striker02] [2026-04-14T05:28:06.433Z] ok: [an-striker01] [2026-04-14T05:28:06.433Z] [2026-04-14T05:28:06.433Z] TASK [copy striker autogenerated config] *************************************** [2026-04-14T05:28:08.354Z] changed: [an-striker01] [2026-04-14T05:28:08.354Z] changed: [an-striker02] [2026-04-14T05:28:08.354Z] [2026-04-14T05:28:08.354Z] PLAY [setup lvm storage] ******************************************************* [2026-04-14T05:28:08.354Z] [2026-04-14T05:28:08.354Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:28:09.704Z] ok: [an-a01n02] [2026-04-14T05:28:09.704Z] ok: [an-a01n01] [2026-04-14T05:28:09.704Z] ok: [an-a01dr01] [2026-04-14T05:28:09.704Z] [2026-04-14T05:28:09.704Z] TASK [check if vg already exists] ********************************************** [2026-04-14T05:28:10.530Z] ok: [an-a01n01] [2026-04-14T05:28:10.530Z] ok: [an-a01n02] [2026-04-14T05:28:10.530Z] ok: [an-a01dr01] [2026-04-14T05:28:10.530Z] [2026-04-14T05:28:10.530Z] TASK [vgcheck] ***************************************************************** [2026-04-14T05:28:10.530Z] skipping: [an-a01n01] [2026-04-14T05:28:10.530Z] [2026-04-14T05:28:10.530Z] TASK [create anvil-test-vg] **************************************************** [2026-04-14T05:28:11.841Z] changed: [an-a01n01] [2026-04-14T05:28:11.841Z] changed: [an-a01dr01] [2026-04-14T05:28:11.841Z] changed: [an-a01n02] [2026-04-14T05:28:11.841Z] [2026-04-14T05:28:11.841Z] PLAY [deploy anvil] ************************************************************ [2026-04-14T05:28:11.841Z] [2026-04-14T05:28:11.841Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:28:13.703Z] ok: [an-striker01] [2026-04-14T05:28:13.703Z] ok: [an-striker02] [2026-04-14T05:28:13.703Z] [2026-04-14T05:28:13.703Z] TASK [check if installation is in progress] ************************************ [2026-04-14T05:28:14.531Z] ok: [an-striker01] [2026-04-14T05:28:14.531Z] ok: [an-striker02] [2026-04-14T05:28:14.531Z] [2026-04-14T05:28:14.531Z] TASK [register installation is in progress] ************************************ [2026-04-14T05:28:16.513Z] changed: [an-striker01] [2026-04-14T05:28:16.513Z] changed: [an-striker02] [2026-04-14T05:28:16.513Z] [2026-04-14T05:28:16.513Z] TASK [start anvil deployment and configuration] ******************************** [2026-04-14T05:28:18.998Z] changed: [an-striker01] [2026-04-14T05:28:18.998Z] changed: [an-striker02] [2026-04-14T05:28:18.998Z] [2026-04-14T05:28:18.998Z] TASK [Wait up to 20 minutes for DB to be running on both strikers] ************* [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (120 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (119 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (118 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (117 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (116 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (115 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (114 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (113 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (112 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (111 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (110 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (109 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (108 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (107 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (106 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (105 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (104 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (103 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (102 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (101 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (100 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (99 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (98 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (97 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (96 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (95 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (94 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (93 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (92 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (91 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (90 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-04-14T05:34:02.308Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (89 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (88 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (87 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (86 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (85 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (84 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (83 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker02 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (82 retries left). [2026-04-14T05:35:56.524Z] FAILED - RETRYING: [an-striker01 -> localhost]: Wait up to 20 minutes for DB to be running on both strikers (82 retries left). [2026-04-14T05:35:56.524Z] ok: [an-striker02 -> localhost] [2026-04-14T05:35:56.524Z] ok: [an-striker01 -> localhost] [2026-04-14T05:35:56.524Z] [2026-04-14T05:35:56.524Z] TASK [Wait up to 20 minutes for both striker-db-status to sync] **************** [2026-04-14T05:36:46.262Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (120 retries left). [2026-04-14T05:36:46.262Z] ok: [an-striker01] [2026-04-14T05:36:46.262Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (119 retries left). [2026-04-14T05:36:46.262Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (118 retries left). [2026-04-14T05:36:46.262Z] FAILED - RETRYING: [an-striker02]: Wait up to 20 minutes for both striker-db-status to sync (117 retries left). [2026-04-14T05:36:46.262Z] ok: [an-striker02] [2026-04-14T05:36:46.262Z] [2026-04-14T05:36:46.262Z] TASK [wait up to 20 minutes for nodes to join the database] ******************** [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (120 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (119 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (118 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (117 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (116 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (115 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (114 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (113 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (112 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (111 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (110 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (109 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (108 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (107 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (106 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (105 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (104 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (103 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (102 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (101 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (100 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (99 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (98 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (97 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (96 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (95 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (94 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (93 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (92 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (91 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (90 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (89 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (88 retries left). [2026-04-14T05:42:27.289Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (87 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (86 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (85 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (84 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (83 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (82 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (81 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (80 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (79 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (78 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (77 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (76 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (75 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (74 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (73 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-14T05:45:07.095Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for nodes to join the database (72 retries left). [2026-04-14T05:45:07.095Z] ok: [an-striker02 -> localhost] [2026-04-14T05:45:07.095Z] ok: [an-striker01 -> localhost] [2026-04-14T05:45:07.095Z] [2026-04-14T05:45:07.095Z] TASK [wait up to 20 minutes for anvil nodes to form] *************************** [2026-04-14T05:45:07.095Z] ok: [an-striker02 -> localhost] [2026-04-14T05:45:07.095Z] ok: [an-striker01 -> localhost] [2026-04-14T05:45:07.095Z] [2026-04-14T05:45:07.095Z] TASK [wait up to 60 minutes for pacemaker to be configured] ******************** [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (360 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (359 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (358 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (357 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (356 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (355 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (354 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (353 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (352 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (351 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (350 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-14T05:47:16.664Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 60 minutes for pacemaker to be configured (349 retries left). [2026-04-14T05:47:16.664Z] ok: [an-striker02 -> localhost] [2026-04-14T05:47:16.664Z] ok: [an-striker01 -> localhost] [2026-04-14T05:47:16.664Z] [2026-04-14T05:47:16.664Z] TASK [wait up to 20 minutes for drbd to be configured] ************************* [2026-04-14T05:47:30.297Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-14T05:47:30.297Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (120 retries left). [2026-04-14T05:47:30.297Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-14T05:47:30.297Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for drbd to be configured (119 retries left). [2026-04-14T05:47:30.297Z] ok: [an-striker02 -> localhost] [2026-04-14T05:47:30.297Z] ok: [an-striker01 -> localhost] [2026-04-14T05:47:30.297Z] [2026-04-14T05:47:30.297Z] TASK [wait up to 20 minutes for any other configuration job to complete] ******* [2026-04-14T05:47:41.588Z] FAILED - RETRYING: [an-striker01 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-04-14T05:47:41.588Z] FAILED - RETRYING: [an-striker02 -> localhost]: wait up to 20 minutes for any other configuration job to complete (120 retries left). [2026-04-14T05:47:41.588Z] ok: [an-striker01 -> localhost] [2026-04-14T05:47:41.588Z] ok: [an-striker02 -> localhost] [2026-04-14T05:47:41.588Z] [2026-04-14T05:47:41.588Z] PLAY [check network status after deployment] *********************************** [2026-04-14T05:47:41.588Z] [2026-04-14T05:47:41.588Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:47:44.040Z] ok: [an-a01n01] [2026-04-14T05:47:44.040Z] ok: [an-a01n02] [2026-04-14T05:47:44.040Z] ok: [an-a01dr01] [2026-04-14T05:47:44.040Z] ok: [an-striker02] [2026-04-14T05:47:44.040Z] ok: [an-striker01] [2026-04-14T05:47:44.040Z] [2026-04-14T05:47:44.040Z] TASK [check network manager autoconnect is set to yes] ************************* [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=ifn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=sn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=ifn1_bridge1) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=ifn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=ifn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=ifn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=sn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=sn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=bcn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=ifn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=ifn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=bcn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=bcn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=ifn1_bridge1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=bcn1_bridge1) [2026-04-14T05:47:53.368Z] skipping: [an-a01dr01] => (item=lo) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=bcn1_bridge1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=sn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=bcn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=sn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=bcn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=ifn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=bcn1_bridge1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=ifn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=bcn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=ifn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=bcn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=bcn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=bcn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=ifn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=bcn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=sn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=ifn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=ifn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=ifn1_bridge1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=sn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=bcn1_link1) [2026-04-14T05:47:53.368Z] skipping: [an-a01n01] => (item=lo) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=bcn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-striker01] => (item=bcn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=ifn1_bond1) [2026-04-14T05:47:53.368Z] skipping: [an-striker01] => (item=lo) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=ifn1_bond1) [2026-04-14T05:47:53.368Z] skipping: [an-a01n02] => (item=lo) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=sn1_link1) [2026-04-14T05:47:53.368Z] changed: [an-a01dr01] => (item=bcn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-a01n02] => (item=bcn1_link2) [2026-04-14T05:47:53.368Z] changed: [an-a01n01] => (item=sn1_bond1) [2026-04-14T05:47:53.368Z] changed: [an-striker02] => (item=ifn1_link2) [2026-04-14T05:47:53.368Z] skipping: [an-striker02] => (item=lo) [2026-04-14T05:47:53.368Z] [2026-04-14T05:47:53.368Z] PLAY [check pacemaker status after deployment] ********************************* [2026-04-14T05:47:53.368Z] [2026-04-14T05:47:53.368Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:47:53.735Z] ok: [an-a01n02] [2026-04-14T05:47:53.735Z] ok: [an-a01n01] [2026-04-14T05:47:53.735Z] [2026-04-14T05:47:53.735Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T05:47:53.736Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T05:47:53.736Z] [2026-04-14T05:47:53.736Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T05:47:57.787Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T05:47:57.787Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T05:47:57.787Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T05:47:57.787Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T05:47:57.787Z] [2026-04-14T05:47:57.787Z] PLAY [check drbd kmod] ********************************************************* [2026-04-14T05:47:57.787Z] [2026-04-14T05:47:57.787Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:47:59.052Z] ok: [an-a01n01] [2026-04-14T05:47:59.052Z] ok: [an-a01dr01] [2026-04-14T05:47:59.052Z] ok: [an-a01n02] [2026-04-14T05:47:59.052Z] [2026-04-14T05:47:59.052Z] TASK [check if drbd kernel module is already loaded] *************************** [2026-04-14T05:47:59.839Z] ok: [an-a01n01] [2026-04-14T05:47:59.839Z] ok: [an-a01n02] [2026-04-14T05:47:59.839Z] ok: [an-a01dr01] [2026-04-14T05:47:59.839Z] [2026-04-14T05:47:59.839Z] TASK [load drbd kernel module] ************************************************* [2026-04-14T05:47:59.839Z] skipping: [an-a01n01] [2026-04-14T05:47:59.839Z] skipping: [an-a01n02] [2026-04-14T05:47:59.839Z] skipping: [an-a01dr01] [2026-04-14T05:47:59.839Z] [2026-04-14T05:47:59.839Z] TASK [check if drbd kernel module is loaded] *********************************** [2026-04-14T05:47:59.839Z] skipping: [an-a01n01] [2026-04-14T05:47:59.839Z] skipping: [an-a01n02] [2026-04-14T05:47:59.839Z] skipping: [an-a01dr01] [2026-04-14T05:47:59.839Z] [2026-04-14T05:47:59.839Z] PLAY [Copy iso images to striker01] ******************************************** [2026-04-14T05:47:59.839Z] [2026-04-14T05:47:59.839Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:48:02.289Z] ok: [an-striker01] [2026-04-14T05:48:02.289Z] [2026-04-14T05:48:02.289Z] TASK [Wipe old snapshots dir] ************************************************** [2026-04-14T05:48:03.075Z] ok: [an-striker01] [2026-04-14T05:48:03.075Z] [2026-04-14T05:48:03.075Z] TASK [Create snapshots dir] **************************************************** [2026-04-14T05:48:03.861Z] changed: [an-striker01] [2026-04-14T05:48:03.861Z] [2026-04-14T05:48:03.861Z] TASK [Copy C9S kickstart images] *********************************************** [2026-04-14T05:48:15.350Z] changed: [an-striker01] => (item=deploy1.iso) [2026-04-14T05:48:15.350Z] changed: [an-striker01] => (item=deploy2.iso) [2026-04-14T05:48:15.350Z] changed: [an-striker01] => (item=deploy3.iso) [2026-04-14T05:48:15.350Z] changed: [an-striker01] => (item=deploy4.iso) [2026-04-14T05:48:15.350Z] changed: [an-striker01] => (item=deploy5.iso) [2026-04-14T05:48:15.350Z] [2026-04-14T05:48:15.350Z] TASK [Move install isos to anvil upload dir] *********************************** [2026-04-14T05:48:15.350Z] changed: [an-striker01] [2026-04-14T05:48:15.350Z] [2026-04-14T05:48:15.350Z] TASK [Wipe old snapshots dir] ************************************************** [2026-04-14T05:48:16.137Z] changed: [an-striker01] [2026-04-14T05:48:16.137Z] [2026-04-14T05:48:16.137Z] PLAY [Check iso images propagation] ******************************************** [2026-04-14T05:48:16.137Z] [2026-04-14T05:48:16.137Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:48:19.326Z] ok: [an-a01n01] [2026-04-14T05:48:19.326Z] ok: [an-a01n02] [2026-04-14T05:48:19.326Z] ok: [an-a01dr01] [2026-04-14T05:48:19.326Z] ok: [an-striker01] [2026-04-14T05:48:19.326Z] ok: [an-striker02] [2026-04-14T05:48:19.326Z] [2026-04-14T05:48:19.326Z] TASK [wait up to 30 minutes for all images to be propagated to all nodes] ****** [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (180 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (179 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (178 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (177 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (176 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-striker01]: wait up to 30 minutes for all images to be propagated to all nodes (175 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-14T05:49:43.281Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for all images to be propagated to all nodes (174 retries left). [2026-04-14T05:49:43.281Z] ok: [an-striker01] [2026-04-14T05:49:43.281Z] ok: [an-striker02] [2026-04-14T05:49:43.281Z] ok: [an-a01n02] [2026-04-14T05:49:43.281Z] ok: [an-a01n01] [2026-04-14T05:49:43.281Z] ok: [an-a01dr01] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] PLAY [example tests] *********************************************************** [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:49:43.281Z] ok: [an-a01n01] [2026-04-14T05:49:43.281Z] ok: [an-a01n02] [2026-04-14T05:49:43.281Z] ok: [an-a01dr01] [2026-04-14T05:49:43.281Z] ok: [an-striker02] [2026-04-14T05:49:43.281Z] ok: [an-striker01] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [this test should be executed only on strikers] *************************** [2026-04-14T05:49:43.281Z] skipping: [an-a01n01] [2026-04-14T05:49:43.281Z] skipping: [an-a01n02] [2026-04-14T05:49:43.281Z] skipping: [an-a01dr01] [2026-04-14T05:49:43.281Z] ok: [an-striker01] [2026-04-14T05:49:43.281Z] ok: [an-striker02] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [this test should be executed only on non strikers] *********************** [2026-04-14T05:49:43.281Z] skipping: [an-striker01] [2026-04-14T05:49:43.281Z] skipping: [an-striker02] [2026-04-14T05:49:43.281Z] ok: [an-a01n02] [2026-04-14T05:49:43.281Z] ok: [an-a01dr01] [2026-04-14T05:49:43.281Z] ok: [an-a01n01] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [inject failure] ********************************************************** [2026-04-14T05:49:43.281Z] skipping: [an-striker01] [2026-04-14T05:49:43.281Z] skipping: [an-striker02] [2026-04-14T05:49:43.281Z] skipping: [an-a01n01] [2026-04-14T05:49:43.281Z] skipping: [an-a01n02] [2026-04-14T05:49:43.281Z] skipping: [an-a01dr01] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] PLAY [Record anvil available resources from striker01] ************************* [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:49:43.281Z] ok: [an-striker01] [2026-04-14T05:49:43.281Z] [2026-04-14T05:49:43.281Z] TASK [Record anvil available resources] **************************************** [2026-04-14T05:49:45.731Z] ok: [an-striker01] [2026-04-14T05:49:45.731Z] [2026-04-14T05:49:45.731Z] PLAY [check selinux status] **************************************************** [2026-04-14T05:49:45.731Z] [2026-04-14T05:49:45.731Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:49:48.182Z] ok: [an-a01n01] [2026-04-14T05:49:48.182Z] ok: [an-a01n02] [2026-04-14T05:49:48.182Z] ok: [an-a01dr01] [2026-04-14T05:49:48.182Z] ok: [an-striker02] [2026-04-14T05:49:48.182Z] ok: [an-striker01] [2026-04-14T05:49:48.182Z] [2026-04-14T05:49:48.182Z] TASK [check selinux status permissive (8)] ************************************* [2026-04-14T05:49:48.182Z] skipping: [an-striker01] [2026-04-14T05:49:48.182Z] skipping: [an-striker02] [2026-04-14T05:49:48.182Z] skipping: [an-a01n01] [2026-04-14T05:49:48.182Z] skipping: [an-a01n02] [2026-04-14T05:49:48.182Z] skipping: [an-a01dr01] [2026-04-14T05:49:48.182Z] [2026-04-14T05:49:48.182Z] TASK [check selinux status permissive (9)] ************************************* [2026-04-14T05:49:48.550Z] ok: [an-striker01] => { [2026-04-14T05:49:48.550Z] "ansible_selinux": { [2026-04-14T05:49:48.550Z] "config_mode": "enforcing", [2026-04-14T05:49:48.550Z] "mode": "enforcing", [2026-04-14T05:49:48.550Z] "policyvers": 33, [2026-04-14T05:49:48.550Z] "status": "enabled", [2026-04-14T05:49:48.550Z] "type": "targeted" [2026-04-14T05:49:48.550Z] }, [2026-04-14T05:49:48.550Z] "failed_when_result": false [2026-04-14T05:49:48.550Z] } [2026-04-14T05:49:48.550Z] ok: [an-striker02] => { [2026-04-14T05:49:48.550Z] "ansible_selinux": { [2026-04-14T05:49:48.550Z] "config_mode": "enforcing", [2026-04-14T05:49:48.550Z] "mode": "enforcing", [2026-04-14T05:49:48.550Z] "policyvers": 33, [2026-04-14T05:49:48.550Z] "status": "enabled", [2026-04-14T05:49:48.550Z] "type": "targeted" [2026-04-14T05:49:48.550Z] }, [2026-04-14T05:49:48.550Z] "failed_when_result": false [2026-04-14T05:49:48.550Z] } [2026-04-14T05:49:48.550Z] ok: [an-a01n01] => { [2026-04-14T05:49:48.550Z] "ansible_selinux": { [2026-04-14T05:49:48.550Z] "config_mode": "enforcing", [2026-04-14T05:49:48.550Z] "mode": "enforcing", [2026-04-14T05:49:48.550Z] "policyvers": 33, [2026-04-14T05:49:48.550Z] "status": "enabled", [2026-04-14T05:49:48.550Z] "type": "targeted" [2026-04-14T05:49:48.550Z] }, [2026-04-14T05:49:48.550Z] "failed_when_result": false [2026-04-14T05:49:48.550Z] } [2026-04-14T05:49:48.550Z] ok: [an-a01n02] => { [2026-04-14T05:49:48.550Z] "ansible_selinux": { [2026-04-14T05:49:48.550Z] "config_mode": "enforcing", [2026-04-14T05:49:48.550Z] "mode": "enforcing", [2026-04-14T05:49:48.550Z] "policyvers": 33, [2026-04-14T05:49:48.550Z] "status": "enabled", [2026-04-14T05:49:48.550Z] "type": "targeted" [2026-04-14T05:49:48.550Z] }, [2026-04-14T05:49:48.550Z] "failed_when_result": false [2026-04-14T05:49:48.550Z] } [2026-04-14T05:49:48.550Z] ok: [an-a01dr01] => { [2026-04-14T05:49:48.550Z] "ansible_selinux": { [2026-04-14T05:49:48.550Z] "config_mode": "enforcing", [2026-04-14T05:49:48.550Z] "mode": "enforcing", [2026-04-14T05:49:48.550Z] "policyvers": 33, [2026-04-14T05:49:48.550Z] "status": "enabled", [2026-04-14T05:49:48.550Z] "type": "targeted" [2026-04-14T05:49:48.550Z] }, [2026-04-14T05:49:48.550Z] "failed_when_result": false [2026-04-14T05:49:48.550Z] } [2026-04-14T05:49:48.550Z] [2026-04-14T05:49:48.550Z] PLAY [Create random file on striker01] ***************************************** [2026-04-14T05:49:48.550Z] [2026-04-14T05:49:48.550Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:49:51.000Z] ok: [an-striker01] [2026-04-14T05:49:51.000Z] [2026-04-14T05:49:51.000Z] TASK [Create archive file] ***************************************************** [2026-04-14T05:49:52.815Z] changed: [an-striker01] [2026-04-14T05:49:52.815Z] [2026-04-14T05:49:52.815Z] PLAY [Check archive file propagation] ****************************************** [2026-04-14T05:49:52.815Z] [2026-04-14T05:49:52.815Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:49:56.005Z] ok: [an-a01n01] [2026-04-14T05:49:56.005Z] ok: [an-a01dr01] [2026-04-14T05:49:56.005Z] ok: [an-a01n02] [2026-04-14T05:49:56.005Z] ok: [an-striker02] [2026-04-14T05:49:56.005Z] ok: [an-striker01] [2026-04-14T05:49:56.005Z] [2026-04-14T05:49:56.005Z] TASK [wait up to 30 minutes for archive to be propagated to all nodes] ********* [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-14T05:51:19.983Z] ok: [an-striker01] [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (180 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (179 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (178 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (177 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (176 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-striker02]: wait up to 30 minutes for archive to be propagated to all nodes (175 retries left). [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01n02]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-14T05:51:19.983Z] ok: [an-a01n01] [2026-04-14T05:51:19.983Z] FAILED - RETRYING: [an-a01dr01]: wait up to 30 minutes for archive to be propagated to all nodes (174 retries left). [2026-04-14T05:51:19.983Z] ok: [an-striker02] [2026-04-14T05:51:19.983Z] ok: [an-a01n02] [2026-04-14T05:51:19.983Z] ok: [an-a01dr01] [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] PLAY [test fencing] ************************************************************ [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:51:19.983Z] ok: [an-a01n01] [2026-04-14T05:51:19.983Z] ok: [an-a01n02] [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T05:51:19.983Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T05:51:19.983Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T05:51:19.983Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T05:51:19.983Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T05:51:19.983Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [kill node "an-a01n01"] *************************************************** [2026-04-14T05:51:19.983Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [kill node "an-a01n01"] *************************************************** [2026-04-14T05:51:19.983Z] skipping: [an-a01n02] [2026-04-14T05:51:19.983Z] changed: [an-a01n01] [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T05:51:19.983Z] skipping: [an-a01n02] [2026-04-14T05:51:19.983Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01 [2026-04-14T05:51:19.983Z] [2026-04-14T05:51:19.983Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T05:54:29.768Z] ok: [an-a01n01 -> localhost] [2026-04-14T05:54:29.768Z] [2026-04-14T05:54:29.768Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T05:54:29.768Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T05:54:29.768Z] [2026-04-14T05:54:29.768Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T05:54:32.219Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T05:54:32.219Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T05:54:32.219Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T05:54:32.219Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T05:54:32.219Z] [2026-04-14T05:54:32.219Z] TASK [kill node "an-a01n02"] *************************************************** [2026-04-14T05:54:32.219Z] included: /srv/anvil/anvil-test-suite/playbooks/common/kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T05:54:32.219Z] [2026-04-14T05:54:32.219Z] TASK [kill node "an-a01n02"] *************************************************** [2026-04-14T05:54:33.005Z] skipping: [an-a01n01] [2026-04-14T05:54:33.005Z] changed: [an-a01n02] [2026-04-14T05:54:33.005Z] [2026-04-14T05:54:33.005Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T05:54:33.005Z] skipping: [an-a01n01] [2026-04-14T05:54:33.005Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n02 [2026-04-14T05:54:33.005Z] [2026-04-14T05:54:33.005Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T05:57:58.093Z] ok: [an-a01n02 -> localhost] [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T05:57:58.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T05:57:58.093Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T05:57:58.093Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T05:57:58.093Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T05:57:58.093Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] PLAY [advanced server tests] *************************************************** [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T05:57:58.093Z] ok: [an-a01n02] [2026-04-14T05:57:58.093Z] ok: [an-a01n01] [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T05:57:58.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T05:57:58.093Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T05:57:58.093Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T05:57:58.093Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T05:57:58.093Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [check anvil-provision-server storage group] ****************************** [2026-04-14T05:57:58.093Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-14T05:57:58.093Z] [2026-04-14T05:57:58.093Z] TASK [get anvil-provision-server storage group] ******************************** [2026-04-14T05:57:58.880Z] changed: [an-a01n01] [2026-04-14T05:57:58.880Z] changed: [an-a01n02] [2026-04-14T05:57:58.880Z] [2026-04-14T05:57:58.880Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T05:57:58.880Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-14T05:57:58.880Z] [2026-04-14T05:57:58.880Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T05:58:05.136Z] changed: [an-a01n01] [2026-04-14T05:58:05.136Z] changed: [an-a01n02] [2026-04-14T05:58:05.136Z] [2026-04-14T05:58:05.136Z] TASK [deploy servers] ********************************************************** [2026-04-14T05:58:05.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T05:58:05.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T05:58:05.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T05:58:05.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T05:58:05.136Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T05:58:05.136Z] [2026-04-14T05:58:05.136Z] TASK [check if server an-test-deploy1 exists] ********************************** [2026-04-14T05:58:05.921Z] skipping: [an-a01n02] [2026-04-14T05:58:05.921Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy1\n", "delta": "0:00:00.894679", "end": "2026-04-14 05:58:05.603476", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 05:58:04.708797", "stderr": "Error: resource or tag id 'an-test-deploy1' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy1' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T05:58:05.921Z] ...ignoring [2026-04-14T05:58:05.921Z] [2026-04-14T05:58:05.921Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T05:58:15.242Z] skipping: [an-a01n02] [2026-04-14T05:58:15.242Z] changed: [an-a01n01] [2026-04-14T05:58:15.242Z] [2026-04-14T05:58:15.242Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker] *** [2026-04-14T05:58:15.242Z] skipping: [an-a01n01] [2026-04-14T05:58:15.242Z] skipping: [an-a01n02] [2026-04-14T05:58:15.242Z] [2026-04-14T05:58:15.242Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced] *** [2026-04-14T05:58:15.242Z] skipping: [an-a01n01] [2026-04-14T05:58:15.242Z] skipping: [an-a01n02] [2026-04-14T05:58:15.242Z] [2026-04-14T05:58:15.242Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation] *** [2026-04-14T05:58:15.242Z] skipping: [an-a01n01] [2026-04-14T05:58:15.242Z] skipping: [an-a01n02] [2026-04-14T05:58:15.242Z] [2026-04-14T05:58:15.243Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation] *** [2026-04-14T05:58:15.243Z] skipping: [an-a01n01] [2026-04-14T05:58:15.243Z] skipping: [an-a01n02] [2026-04-14T05:58:15.243Z] [2026-04-14T05:58:15.243Z] TASK [check if server an-test-deploy2 exists] ********************************** [2026-04-14T05:58:16.029Z] skipping: [an-a01n02] [2026-04-14T05:58:16.029Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.883023", "end": "2026-04-14 05:58:15.744616", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 05:58:14.861593", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T05:58:16.029Z] ...ignoring [2026-04-14T05:58:16.029Z] [2026-04-14T05:58:16.029Z] TASK [deploy server an-test-deploy2 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T05:58:25.351Z] skipping: [an-a01n02] [2026-04-14T05:58:25.351Z] changed: [an-a01n01] [2026-04-14T05:58:25.351Z] [2026-04-14T05:58:25.351Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker] *** [2026-04-14T05:58:25.351Z] skipping: [an-a01n01] [2026-04-14T05:58:25.351Z] skipping: [an-a01n02] [2026-04-14T05:58:25.351Z] [2026-04-14T05:58:25.351Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced] *** [2026-04-14T05:58:25.351Z] skipping: [an-a01n01] [2026-04-14T05:58:25.351Z] skipping: [an-a01n02] [2026-04-14T05:58:25.351Z] [2026-04-14T05:58:25.351Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation] *** [2026-04-14T05:58:25.351Z] skipping: [an-a01n01] [2026-04-14T05:58:25.351Z] skipping: [an-a01n02] [2026-04-14T05:58:25.351Z] [2026-04-14T05:58:25.351Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation] *** [2026-04-14T05:58:25.351Z] skipping: [an-a01n01] [2026-04-14T05:58:25.351Z] skipping: [an-a01n02] [2026-04-14T05:58:25.351Z] [2026-04-14T05:58:25.351Z] TASK [check if server an-test-deploy3 exists] ********************************** [2026-04-14T05:58:26.615Z] skipping: [an-a01n02] [2026-04-14T05:58:26.615Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.843542", "end": "2026-04-14 05:58:26.422964", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 05:58:25.579422", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T05:58:26.615Z] ...ignoring [2026-04-14T05:58:26.615Z] [2026-04-14T05:58:26.615Z] TASK [deploy server an-test-deploy3 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T05:58:35.950Z] skipping: [an-a01n02] [2026-04-14T05:58:35.950Z] changed: [an-a01n01] [2026-04-14T05:58:35.950Z] [2026-04-14T05:58:35.950Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker] *** [2026-04-14T05:58:35.950Z] skipping: [an-a01n01] [2026-04-14T05:58:35.950Z] skipping: [an-a01n02] [2026-04-14T05:58:35.950Z] [2026-04-14T05:58:35.950Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced] *** [2026-04-14T05:58:35.950Z] skipping: [an-a01n01] [2026-04-14T05:58:35.950Z] skipping: [an-a01n02] [2026-04-14T05:58:35.950Z] [2026-04-14T05:58:35.950Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation] *** [2026-04-14T05:58:35.950Z] skipping: [an-a01n01] [2026-04-14T05:58:35.950Z] skipping: [an-a01n02] [2026-04-14T05:58:35.950Z] [2026-04-14T05:58:35.950Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation] *** [2026-04-14T05:58:35.950Z] skipping: [an-a01n01] [2026-04-14T05:58:35.950Z] skipping: [an-a01n02] [2026-04-14T05:58:35.950Z] [2026-04-14T05:58:35.950Z] TASK [check if server an-test-deploy4 exists] ********************************** [2026-04-14T05:58:37.390Z] skipping: [an-a01n02] [2026-04-14T05:58:37.390Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.882214", "end": "2026-04-14 05:58:37.038036", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 05:58:36.155822", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T05:58:37.390Z] ...ignoring [2026-04-14T05:58:37.390Z] [2026-04-14T05:58:37.390Z] TASK [deploy server an-test-deploy4 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T05:58:46.710Z] skipping: [an-a01n02] [2026-04-14T05:58:46.710Z] changed: [an-a01n01] [2026-04-14T05:58:46.710Z] [2026-04-14T05:58:46.710Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker] *** [2026-04-14T05:58:46.710Z] skipping: [an-a01n01] [2026-04-14T05:58:46.710Z] skipping: [an-a01n02] [2026-04-14T05:58:46.710Z] [2026-04-14T05:58:46.710Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced] *** [2026-04-14T05:58:46.710Z] skipping: [an-a01n01] [2026-04-14T05:58:46.710Z] skipping: [an-a01n02] [2026-04-14T05:58:46.710Z] [2026-04-14T05:58:46.710Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation] *** [2026-04-14T05:58:46.710Z] skipping: [an-a01n01] [2026-04-14T05:58:46.710Z] skipping: [an-a01n02] [2026-04-14T05:58:46.710Z] [2026-04-14T05:58:46.710Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation] *** [2026-04-14T05:58:46.710Z] skipping: [an-a01n01] [2026-04-14T05:58:46.710Z] skipping: [an-a01n02] [2026-04-14T05:58:46.710Z] [2026-04-14T05:58:46.710Z] TASK [check if server an-test-deploy5 exists] ********************************** [2026-04-14T05:58:48.525Z] skipping: [an-a01n02] [2026-04-14T05:58:48.525Z] fatal: [an-a01n01]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.049349", "end": "2026-04-14 05:58:48.211294", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 05:58:47.161945", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T05:58:48.525Z] ...ignoring [2026-04-14T05:58:48.525Z] [2026-04-14T05:58:48.525Z] TASK [deploy server an-test-deploy5 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T05:58:59.816Z] skipping: [an-a01n02] [2026-04-14T05:58:59.816Z] changed: [an-a01n01] [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to be started in pacemaker] *** [2026-04-14T05:58:59.816Z] skipping: [an-a01n01] [2026-04-14T05:58:59.816Z] skipping: [an-a01n02] [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} drbd storage to be synced] *** [2026-04-14T05:58:59.816Z] skipping: [an-a01n01] [2026-04-14T05:58:59.816Z] skipping: [an-a01n02] [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to start installation] *** [2026-04-14T05:58:59.816Z] skipping: [an-a01n01] [2026-04-14T05:58:59.816Z] skipping: [an-a01n02] [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [Wait up to {{ timeout }} minutes for an-test-deploy{{ servernum }} to complete installation] *** [2026-04-14T05:58:59.816Z] skipping: [an-a01n01] [2026-04-14T05:58:59.816Z] skipping: [an-a01n02] [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [wait servers to be started in pacemaker] ********************************* [2026-04-14T05:58:59.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T05:58:59.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T05:58:59.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T05:58:59.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T05:58:59.816Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T05:58:59.816Z] [2026-04-14T05:58:59.816Z] TASK [Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker] **** [2026-04-14T05:59:28.224Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-14T05:59:28.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (360 retries left). [2026-04-14T05:59:28.224Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-14T05:59:28.224Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy1 to be started in pacemaker (359 retries left). [2026-04-14T05:59:28.224Z] ok: [an-a01n01] [2026-04-14T05:59:28.224Z] ok: [an-a01n02] [2026-04-14T05:59:28.224Z] [2026-04-14T05:59:28.224Z] TASK [Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker] **** [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (360 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (359 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (358 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (357 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (356 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (355 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (354 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (353 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (352 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (351 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (350 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left). [2026-04-14T06:01:52.835Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to be started in pacemaker (349 retries left). [2026-04-14T06:01:52.835Z] ok: [an-a01n02] [2026-04-14T06:01:52.835Z] ok: [an-a01n01] [2026-04-14T06:01:52.835Z] [2026-04-14T06:01:52.835Z] TASK [Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker] **** [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (360 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (359 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (358 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (357 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (356 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (355 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (354 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (353 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (352 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (351 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (350 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (349 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (348 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left). [2026-04-14T06:04:47.509Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to be started in pacemaker (347 retries left). [2026-04-14T06:04:47.509Z] ok: [an-a01n02] [2026-04-14T06:04:47.509Z] ok: [an-a01n01] [2026-04-14T06:04:47.509Z] [2026-04-14T06:04:47.509Z] TASK [Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker] **** [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (360 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (359 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (358 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (357 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (356 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (355 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (354 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (353 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (352 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (351 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (350 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (349 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-14T06:07:12.117Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to be started in pacemaker (348 retries left). [2026-04-14T06:07:12.117Z] ok: [an-a01n02] [2026-04-14T06:07:12.117Z] ok: [an-a01n01] [2026-04-14T06:07:12.117Z] [2026-04-14T06:07:12.117Z] TASK [Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker] **** [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (360 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (359 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (358 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (357 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (356 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-14T06:10:22.052Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (355 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (354 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (353 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (352 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (351 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (350 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (349 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (348 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (347 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (346 retries left). [2026-04-14T06:10:22.053Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to be started in pacemaker (345 retries left). [2026-04-14T06:10:22.053Z] ok: [an-a01n01] [2026-04-14T06:10:22.053Z] ok: [an-a01n02] [2026-04-14T06:10:22.053Z] [2026-04-14T06:10:22.053Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:10:22.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:10:22.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:10:22.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:10:22.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:10:22.053Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:10:22.053Z] [2026-04-14T06:10:22.053Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:10:23.318Z] ok: [an-a01n02] [2026-04-14T06:10:23.318Z] ok: [an-a01n01] [2026-04-14T06:10:23.318Z] [2026-04-14T06:10:23.318Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:10:24.105Z] ok: [an-a01n02] [2026-04-14T06:10:24.105Z] ok: [an-a01n01] [2026-04-14T06:10:24.105Z] [2026-04-14T06:10:24.105Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:10:25.370Z] ok: [an-a01n02] [2026-04-14T06:10:25.370Z] ok: [an-a01n01] [2026-04-14T06:10:25.370Z] [2026-04-14T06:10:25.370Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:10:26.158Z] ok: [an-a01n02] [2026-04-14T06:10:26.158Z] ok: [an-a01n01] [2026-04-14T06:10:26.158Z] [2026-04-14T06:10:26.158Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (360 retries left). [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (359 retries left). [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-14T06:11:00.212Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced (358 retries left). [2026-04-14T06:11:00.212Z] ok: [an-a01n02] [2026-04-14T06:11:00.212Z] ok: [an-a01n01] [2026-04-14T06:11:00.212Z] [2026-04-14T06:11:00.212Z] TASK [wait servers to respond to ping] ***************************************** [2026-04-14T06:11:00.580Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:11:00.580Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:11:00.580Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:11:00.580Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:11:00.580Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:11:00.580Z] [2026-04-14T06:11:00.580Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ping] ************ [2026-04-14T06:11:01.367Z] ok: [an-a01n02] [2026-04-14T06:11:01.367Z] ok: [an-a01n01] [2026-04-14T06:11:01.367Z] [2026-04-14T06:11:01.367Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ping] ************ [2026-04-14T06:11:02.154Z] ok: [an-a01n01] [2026-04-14T06:11:02.154Z] ok: [an-a01n02] [2026-04-14T06:11:02.154Z] [2026-04-14T06:11:02.154Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ping] ************ [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (360 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (359 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (358 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (357 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (356 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (355 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (354 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (353 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (352 retries left). [2026-04-14T06:13:11.660Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ping (352 retries left). [2026-04-14T06:13:11.660Z] ok: [an-a01n02] [2026-04-14T06:13:11.660Z] ok: [an-a01n01] [2026-04-14T06:13:11.660Z] [2026-04-14T06:13:11.660Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ping] ************ [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (360 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (359 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (358 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (357 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (356 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (355 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (354 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (353 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (352 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (351 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (350 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (349 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-14T06:16:21.459Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ping (348 retries left). [2026-04-14T06:16:21.459Z] ok: [an-a01n02] [2026-04-14T06:16:21.459Z] ok: [an-a01n01] [2026-04-14T06:16:21.459Z] [2026-04-14T06:16:21.459Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ping] ************ [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (360 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (359 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (358 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (357 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (356 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (355 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (354 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (353 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (352 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (351 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-14T06:18:45.923Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ping (350 retries left). [2026-04-14T06:18:45.923Z] ok: [an-a01n02] [2026-04-14T06:18:45.923Z] ok: [an-a01n01] [2026-04-14T06:18:45.923Z] [2026-04-14T06:18:45.923Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:18:45.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:18:45.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:18:45.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:18:45.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:18:45.923Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:18:45.923Z] [2026-04-14T06:18:45.923Z] TASK [Wait up to 60 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:18:45.923Z] ok: [an-a01n02] [2026-04-14T06:18:45.923Z] ok: [an-a01n01] [2026-04-14T06:18:45.923Z] [2026-04-14T06:18:45.923Z] TASK [Wait up to 60 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (360 retries left). [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (359 retries left). [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-04-14T06:19:44.571Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 to respond to ssh (358 retries left). [2026-04-14T06:19:44.571Z] ok: [an-a01n02] [2026-04-14T06:19:44.571Z] ok: [an-a01n01] [2026-04-14T06:19:44.571Z] [2026-04-14T06:19:44.571Z] TASK [Wait up to 60 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (360 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (359 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (358 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (357 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (356 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (355 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (354 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (353 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (352 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (351 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (350 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (349 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (348 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (347 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (346 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (345 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (344 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (344 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (343 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (343 retries left). [2026-04-14T06:23:09.623Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy3 to respond to ssh (342 retries left). [2026-04-14T06:23:09.623Z] ok: [an-a01n02] [2026-04-14T06:23:09.623Z] ok: [an-a01n01] [2026-04-14T06:23:09.623Z] [2026-04-14T06:23:09.623Z] TASK [Wait up to 60 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (360 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (359 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (358 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (357 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (356 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (355 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (354 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (353 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (352 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (351 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (350 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-04-14T06:25:34.163Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy4 to respond to ssh (349 retries left). [2026-04-14T06:25:34.163Z] ok: [an-a01n02] [2026-04-14T06:25:34.163Z] ok: [an-a01n01] [2026-04-14T06:25:34.163Z] [2026-04-14T06:25:34.163Z] TASK [Wait up to 60 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (360 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (359 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (358 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (357 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (356 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (355 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (354 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (353 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (352 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-14T06:27:43.499Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy5 to respond to ssh (351 retries left). [2026-04-14T06:27:43.499Z] ok: [an-a01n02] [2026-04-14T06:27:43.499Z] ok: [an-a01n01] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [fence with servers test] ************************************************* [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/common/fence_with_servers.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [fence with servers iteration 1] ****************************************** [2026-04-14T06:27:43.499Z] ok: [an-a01n01] [2026-04-14T06:27:43.499Z] ok: [an-a01n02] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [check servers status] **************************************************** [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:27:43.499Z] ok: [an-a01n02] [2026-04-14T06:27:43.499Z] ok: [an-a01n01] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:27:43.499Z] ok: [an-a01n02] [2026-04-14T06:27:43.499Z] ok: [an-a01n01] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [set target] ************************************************************** [2026-04-14T06:27:43.499Z] ok: [an-a01n01] [2026-04-14T06:27:43.499Z] ok: [an-a01n02] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [will kill node with servers] ********************************************* [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [kill node "an-a01n01"] *************************************************** [2026-04-14T06:27:43.499Z] skipping: [an-a01n02] [2026-04-14T06:27:43.499Z] changed: [an-a01n01] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:27:43.499Z] skipping: [an-a01n01] [2026-04-14T06:27:43.499Z] skipping: [an-a01n02] [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [wait for node to be offline] ********************************************* [2026-04-14T06:27:43.499Z] skipping: [an-a01n01] [2026-04-14T06:27:43.499Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-14T06:27:43.499Z] [2026-04-14T06:27:43.499Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline] ****************** [2026-04-14T06:28:42.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-14T06:28:42.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-14T06:28:42.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-14T06:28:42.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-14T06:28:42.150Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-14T06:28:42.150Z] ok: [an-a01n02] [2026-04-14T06:28:42.150Z] [2026-04-14T06:28:42.150Z] TASK [check servers status] **************************************************** [2026-04-14T06:28:42.150Z] skipping: [an-a01n01] [2026-04-14T06:28:42.150Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-14T06:28:42.150Z] [2026-04-14T06:28:42.150Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (114 retries left). [2026-04-14T06:30:21.233Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (113 retries left). [2026-04-14T06:30:21.233Z] ok: [an-a01n02] [2026-04-14T06:30:21.233Z] [2026-04-14T06:30:21.233Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:30:23.686Z] ok: [an-a01n02] [2026-04-14T06:30:23.686Z] [2026-04-14T06:30:23.686Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] => (item=1) [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] => (item=2) [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] => (item=3) [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] => (item=4) [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] => (item=5) [2026-04-14T06:30:23.686Z] skipping: [an-a01n01] [2026-04-14T06:30:23.686Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-14T06:30:23.686Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-14T06:30:23.686Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-14T06:30:23.686Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-14T06:30:23.686Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-14T06:30:23.686Z] [2026-04-14T06:30:23.686Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:30:24.951Z] ok: [an-a01n02] [2026-04-14T06:30:24.951Z] [2026-04-14T06:30:24.951Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:30:25.737Z] ok: [an-a01n02] [2026-04-14T06:30:25.737Z] [2026-04-14T06:30:25.737Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:30:26.524Z] ok: [an-a01n02] [2026-04-14T06:30:26.524Z] [2026-04-14T06:30:26.524Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:30:27.309Z] ok: [an-a01n02] [2026-04-14T06:30:27.309Z] [2026-04-14T06:30:27.309Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:30:28.096Z] ok: [an-a01n02] [2026-04-14T06:30:28.096Z] [2026-04-14T06:30:28.096Z] TASK [wait for node to be back online] ***************************************** [2026-04-14T06:30:28.464Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:30:28.464Z] [2026-04-14T06:30:28.464Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:33:53.407Z] ok: [an-a01n02 -> localhost] [2026-04-14T06:33:53.407Z] ok: [an-a01n01 -> localhost] [2026-04-14T06:33:53.407Z] [2026-04-14T06:33:53.407Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:33:53.407Z] [2026-04-14T06:33:53.407Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:33:53.407Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:33:53.407Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:33:53.407Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:33:53.407Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:33:53.407Z] [2026-04-14T06:33:53.407Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:33:53.407Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:33:53.407Z] [2026-04-14T06:33:53.407Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:33:53.407Z] ok: [an-a01n01] [2026-04-14T06:33:53.407Z] ok: [an-a01n02] [2026-04-14T06:33:53.407Z] [2026-04-14T06:33:53.407Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:34:07.041Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-04-14T06:34:07.041Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (360 retries left). [2026-04-14T06:34:07.041Z] FAILED - RETRYING: [an-a01n02]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-04-14T06:34:07.041Z] FAILED - RETRYING: [an-a01n01]: Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced (359 retries left). [2026-04-14T06:34:07.041Z] ok: [an-a01n02] [2026-04-14T06:34:07.041Z] ok: [an-a01n01] [2026-04-14T06:34:07.041Z] [2026-04-14T06:34:07.041Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:34:07.827Z] ok: [an-a01n01] [2026-04-14T06:34:07.827Z] ok: [an-a01n02] [2026-04-14T06:34:07.827Z] [2026-04-14T06:34:07.827Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:34:08.197Z] ok: [an-a01n01] [2026-04-14T06:34:08.197Z] ok: [an-a01n02] [2026-04-14T06:34:08.197Z] [2026-04-14T06:34:08.197Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:34:09.101Z] ok: [an-a01n01] [2026-04-14T06:34:09.101Z] ok: [an-a01n02] [2026-04-14T06:34:09.101Z] [2026-04-14T06:34:09.101Z] TASK [fence with servers iteration 2] ****************************************** [2026-04-14T06:34:09.101Z] ok: [an-a01n01] [2026-04-14T06:34:09.101Z] ok: [an-a01n02] [2026-04-14T06:34:09.101Z] [2026-04-14T06:34:09.101Z] TASK [check servers status] **************************************************** [2026-04-14T06:34:09.101Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:34:09.101Z] [2026-04-14T06:34:09.101Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:34:11.554Z] ok: [an-a01n01] [2026-04-14T06:34:11.554Z] ok: [an-a01n02] [2026-04-14T06:34:11.554Z] [2026-04-14T06:34:11.554Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:34:13.370Z] ok: [an-a01n01] [2026-04-14T06:34:13.370Z] ok: [an-a01n02] [2026-04-14T06:34:13.370Z] [2026-04-14T06:34:13.370Z] TASK [set target] ************************************************************** [2026-04-14T06:34:13.370Z] ok: [an-a01n01] [2026-04-14T06:34:13.370Z] ok: [an-a01n02] [2026-04-14T06:34:13.370Z] [2026-04-14T06:34:13.370Z] TASK [will kill node with servers] ********************************************* [2026-04-14T06:34:13.740Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:34:13.740Z] [2026-04-14T06:34:13.740Z] TASK [kill node "an-a01n02"] *************************************************** [2026-04-14T06:34:14.110Z] skipping: [an-a01n01] [2026-04-14T06:34:14.110Z] changed: [an-a01n02] [2026-04-14T06:34:14.110Z] [2026-04-14T06:34:14.110Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T06:34:14.110Z] skipping: [an-a01n01] [2026-04-14T06:34:14.110Z] skipping: [an-a01n02] [2026-04-14T06:34:14.110Z] [2026-04-14T06:34:14.110Z] TASK [wait for node to be offline] ********************************************* [2026-04-14T06:34:14.478Z] skipping: [an-a01n02] [2026-04-14T06:34:14.478Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-14T06:34:14.478Z] [2026-04-14T06:34:14.478Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline] ****************** [2026-04-14T06:35:24.700Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-14T06:35:24.700Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-14T06:35:24.700Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-14T06:35:24.700Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-14T06:35:24.700Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-14T06:35:24.700Z] ok: [an-a01n01] [2026-04-14T06:35:24.700Z] [2026-04-14T06:35:24.701Z] TASK [check servers status] **************************************************** [2026-04-14T06:35:24.701Z] skipping: [an-a01n02] [2026-04-14T06:35:24.701Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-14T06:35:24.701Z] [2026-04-14T06:35:24.701Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:36:23.235Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-14T06:36:23.235Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-14T06:36:23.235Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-14T06:36:23.235Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-14T06:36:23.235Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-14T06:36:23.235Z] ok: [an-a01n01] [2026-04-14T06:36:23.235Z] [2026-04-14T06:36:23.235Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:36:24.501Z] ok: [an-a01n01] [2026-04-14T06:36:24.501Z] [2026-04-14T06:36:24.501Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] => (item=1) [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] => (item=2) [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] => (item=3) [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] => (item=4) [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] => (item=5) [2026-04-14T06:36:24.870Z] skipping: [an-a01n02] [2026-04-14T06:36:24.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-14T06:36:24.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-14T06:36:24.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-14T06:36:24.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-14T06:36:24.870Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-14T06:36:24.870Z] [2026-04-14T06:36:24.870Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:36:44.780Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-14T06:36:44.780Z] ok: [an-a01n01] [2026-04-14T06:36:44.780Z] [2026-04-14T06:36:44.780Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:36:44.780Z] ok: [an-a01n01] [2026-04-14T06:36:44.780Z] [2026-04-14T06:36:44.780Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:36:46.049Z] ok: [an-a01n01] [2026-04-14T06:36:46.049Z] [2026-04-14T06:36:46.049Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:36:47.317Z] ok: [an-a01n01] [2026-04-14T06:36:47.317Z] [2026-04-14T06:36:47.317Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:36:48.581Z] ok: [an-a01n01] [2026-04-14T06:36:48.581Z] [2026-04-14T06:36:48.581Z] TASK [wait for node to be back online] ***************************************** [2026-04-14T06:36:48.581Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:36:48.581Z] [2026-04-14T06:36:48.581Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T06:40:13.636Z] ok: [an-a01n01 -> localhost] [2026-04-14T06:40:13.636Z] ok: [an-a01n02 -> localhost] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:40:13.636Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:40:13.636Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:40:13.636Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:40:13.636Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=1) [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=2) [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=3) [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=4) [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n02, an-a01n01 => (item=5) [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [fence with servers iteration 3] ****************************************** [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [check servers status] **************************************************** [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [set target] ************************************************************** [2026-04-14T06:40:13.636Z] ok: [an-a01n01] [2026-04-14T06:40:13.636Z] ok: [an-a01n02] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [will kill node with servers] ********************************************* [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [kill node "an-a01n01"] *************************************************** [2026-04-14T06:40:13.636Z] skipping: [an-a01n02] [2026-04-14T06:40:13.636Z] changed: [an-a01n01] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:40:13.636Z] skipping: [an-a01n01] [2026-04-14T06:40:13.636Z] skipping: [an-a01n02] [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait for node to be offline] ********************************************* [2026-04-14T06:40:13.636Z] skipping: [an-a01n01] [2026-04-14T06:40:13.636Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-14T06:40:13.636Z] [2026-04-14T06:40:13.636Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline] ****************** [2026-04-14T06:41:23.734Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-14T06:41:23.734Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-14T06:41:23.734Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-14T06:41:23.734Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-14T06:41:23.734Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-14T06:41:23.734Z] ok: [an-a01n02] [2026-04-14T06:41:23.734Z] [2026-04-14T06:41:23.734Z] TASK [check servers status] **************************************************** [2026-04-14T06:41:23.734Z] skipping: [an-a01n01] [2026-04-14T06:41:23.734Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-14T06:41:23.734Z] [2026-04-14T06:41:23.734Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-14T06:42:33.867Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (115 retries left). [2026-04-14T06:42:33.867Z] ok: [an-a01n02] [2026-04-14T06:42:33.867Z] [2026-04-14T06:42:33.867Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:42:33.867Z] ok: [an-a01n02] [2026-04-14T06:42:33.867Z] [2026-04-14T06:42:33.867Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] => (item=1) [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] => (item=2) [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] => (item=3) [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] => (item=4) [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] => (item=5) [2026-04-14T06:42:33.867Z] skipping: [an-a01n01] [2026-04-14T06:42:33.867Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-14T06:42:33.867Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-14T06:42:33.867Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-14T06:42:33.867Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-14T06:42:33.867Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-14T06:42:33.867Z] [2026-04-14T06:42:33.867Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:42:33.867Z] ok: [an-a01n02] [2026-04-14T06:42:33.867Z] [2026-04-14T06:42:33.867Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:42:34.655Z] ok: [an-a01n02] [2026-04-14T06:42:34.655Z] [2026-04-14T06:42:34.655Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:42:35.442Z] ok: [an-a01n02] [2026-04-14T06:42:35.442Z] [2026-04-14T06:42:35.442Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:42:36.229Z] ok: [an-a01n02] [2026-04-14T06:42:36.229Z] [2026-04-14T06:42:36.229Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:42:38.045Z] ok: [an-a01n02] [2026-04-14T06:42:38.045Z] [2026-04-14T06:42:38.045Z] TASK [wait for node to be back online] ***************************************** [2026-04-14T06:42:38.413Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:42:38.413Z] [2026-04-14T06:42:38.413Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:46:03.319Z] ok: [an-a01n01 -> localhost] [2026-04-14T06:46:03.319Z] ok: [an-a01n02 -> localhost] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:46:03.319Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:46:03.319Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:46:03.319Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:46:03.319Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [fence with servers iteration 4] ****************************************** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [check servers status] **************************************************** [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [set target] ************************************************************** [2026-04-14T06:46:03.319Z] ok: [an-a01n01] [2026-04-14T06:46:03.319Z] ok: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [will kill node with servers] ********************************************* [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [kill node "an-a01n02"] *************************************************** [2026-04-14T06:46:03.319Z] skipping: [an-a01n01] [2026-04-14T06:46:03.319Z] changed: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T06:46:03.319Z] skipping: [an-a01n01] [2026-04-14T06:46:03.319Z] skipping: [an-a01n02] [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait for node to be offline] ********************************************* [2026-04-14T06:46:03.319Z] skipping: [an-a01n02] [2026-04-14T06:46:03.319Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n01 [2026-04-14T06:46:03.319Z] [2026-04-14T06:46:03.319Z] TASK [wait up to 10 minutes for node an-a01n02 to be offline] ****************** [2026-04-14T06:47:13.562Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (60 retries left). [2026-04-14T06:47:13.562Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (59 retries left). [2026-04-14T06:47:13.562Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (58 retries left). [2026-04-14T06:47:13.562Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (57 retries left). [2026-04-14T06:47:13.562Z] FAILED - RETRYING: [an-a01n01]: wait up to 10 minutes for node an-a01n02 to be offline (56 retries left). [2026-04-14T06:47:13.562Z] ok: [an-a01n01] [2026-04-14T06:47:13.562Z] [2026-04-14T06:47:13.562Z] TASK [check servers status] **************************************************** [2026-04-14T06:47:13.562Z] skipping: [an-a01n02] [2026-04-14T06:47:13.562Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01 [2026-04-14T06:47:13.562Z] [2026-04-14T06:47:13.562Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:48:12.213Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-14T06:48:12.213Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-14T06:48:12.213Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-14T06:48:12.213Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-14T06:48:12.213Z] FAILED - RETRYING: [an-a01n01]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-14T06:48:12.213Z] ok: [an-a01n01] [2026-04-14T06:48:12.213Z] [2026-04-14T06:48:12.213Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:48:12.213Z] ok: [an-a01n01] [2026-04-14T06:48:12.213Z] [2026-04-14T06:48:12.214Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] => (item=1) [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] => (item=2) [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] => (item=3) [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] => (item=4) [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] => (item=5) [2026-04-14T06:48:12.214Z] skipping: [an-a01n02] [2026-04-14T06:48:12.214Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=1) [2026-04-14T06:48:12.214Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=2) [2026-04-14T06:48:12.214Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=3) [2026-04-14T06:48:12.214Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=4) [2026-04-14T06:48:12.214Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n01 => (item=5) [2026-04-14T06:48:12.214Z] [2026-04-14T06:48:12.214Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:48:25.848Z] FAILED - RETRYING: [an-a01n01]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-14T06:48:25.848Z] ok: [an-a01n01] [2026-04-14T06:48:25.848Z] [2026-04-14T06:48:25.848Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:48:25.848Z] ok: [an-a01n01] [2026-04-14T06:48:25.848Z] [2026-04-14T06:48:25.848Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:48:25.848Z] ok: [an-a01n01] [2026-04-14T06:48:25.848Z] [2026-04-14T06:48:25.848Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:48:26.216Z] ok: [an-a01n01] [2026-04-14T06:48:26.216Z] [2026-04-14T06:48:26.216Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:48:27.004Z] ok: [an-a01n01] [2026-04-14T06:48:27.004Z] [2026-04-14T06:48:27.005Z] TASK [wait for node to be back online] ***************************************** [2026-04-14T06:48:27.005Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:48:27.005Z] [2026-04-14T06:48:27.005Z] TASK [wait for node "an-a01n02" to be back online] ***************************** [2026-04-14T06:51:51.918Z] ok: [an-a01n01 -> localhost] [2026-04-14T06:51:51.918Z] ok: [an-a01n02 -> localhost] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:51:51.918Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:51:51.918Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:51:51.918Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [fence with servers iteration 5] ****************************************** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [check servers status] **************************************************** [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [set target] ************************************************************** [2026-04-14T06:51:51.918Z] ok: [an-a01n01] [2026-04-14T06:51:51.918Z] ok: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [will kill node with servers] ********************************************* [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./kill_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [kill node "an-a01n01"] *************************************************** [2026-04-14T06:51:51.918Z] skipping: [an-a01n02] [2026-04-14T06:51:51.918Z] changed: [an-a01n01] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:51:51.918Z] skipping: [an-a01n01] [2026-04-14T06:51:51.918Z] skipping: [an-a01n02] [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait for node to be offline] ********************************************* [2026-04-14T06:51:51.918Z] skipping: [an-a01n01] [2026-04-14T06:51:51.918Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./offline_node.yml for an-a01n02 [2026-04-14T06:51:51.918Z] [2026-04-14T06:51:51.918Z] TASK [wait up to 10 minutes for node an-a01n01 to be offline] ****************** [2026-04-14T06:53:02.164Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (60 retries left). [2026-04-14T06:53:02.164Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (59 retries left). [2026-04-14T06:53:02.164Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (58 retries left). [2026-04-14T06:53:02.164Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (57 retries left). [2026-04-14T06:53:02.164Z] FAILED - RETRYING: [an-a01n02]: wait up to 10 minutes for node an-a01n01 to be offline (56 retries left). [2026-04-14T06:53:02.164Z] ok: [an-a01n02] [2026-04-14T06:53:02.164Z] [2026-04-14T06:53:02.164Z] TASK [check servers status] **************************************************** [2026-04-14T06:53:02.164Z] skipping: [an-a01n01] [2026-04-14T06:53:02.164Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./server_started.yml for an-a01n02 [2026-04-14T06:53:02.164Z] [2026-04-14T06:53:02.164Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:54:00.696Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (120 retries left). [2026-04-14T06:54:00.696Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (119 retries left). [2026-04-14T06:54:00.696Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (118 retries left). [2026-04-14T06:54:00.696Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (117 retries left). [2026-04-14T06:54:00.696Z] FAILED - RETRYING: [an-a01n02]: wait up to 20 minutes for servers to be Started (116 retries left). [2026-04-14T06:54:00.696Z] ok: [an-a01n02] [2026-04-14T06:54:00.696Z] [2026-04-14T06:54:00.696Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:54:00.696Z] ok: [an-a01n02] [2026-04-14T06:54:00.696Z] [2026-04-14T06:54:00.696Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] => (item=1) [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] => (item=2) [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] => (item=3) [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] => (item=4) [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] => (item=5) [2026-04-14T06:54:00.696Z] skipping: [an-a01n01] [2026-04-14T06:54:00.696Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=1) [2026-04-14T06:54:00.696Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=2) [2026-04-14T06:54:00.696Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=3) [2026-04-14T06:54:00.696Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=4) [2026-04-14T06:54:00.696Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_ssh.yml for an-a01n02 => (item=5) [2026-04-14T06:54:00.696Z] [2026-04-14T06:54:00.696Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T06:54:14.330Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to respond to ssh (60 retries left). [2026-04-14T06:54:14.330Z] ok: [an-a01n02] [2026-04-14T06:54:14.330Z] [2026-04-14T06:54:14.330Z] TASK [Wait up to 10 minutes for an-test-deploy2 to respond to ssh] ************* [2026-04-14T06:54:14.330Z] ok: [an-a01n02] [2026-04-14T06:54:14.330Z] [2026-04-14T06:54:14.330Z] TASK [Wait up to 10 minutes for an-test-deploy3 to respond to ssh] ************* [2026-04-14T06:54:14.330Z] ok: [an-a01n02] [2026-04-14T06:54:14.330Z] [2026-04-14T06:54:14.330Z] TASK [Wait up to 10 minutes for an-test-deploy4 to respond to ssh] ************* [2026-04-14T06:54:15.232Z] ok: [an-a01n02] [2026-04-14T06:54:15.232Z] [2026-04-14T06:54:15.232Z] TASK [Wait up to 10 minutes for an-test-deploy5 to respond to ssh] ************* [2026-04-14T06:54:16.020Z] ok: [an-a01n02] [2026-04-14T06:54:16.020Z] [2026-04-14T06:54:16.020Z] TASK [wait for node to be back online] ***************************************** [2026-04-14T06:54:16.020Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_node.yml for an-a01n01, an-a01n02 [2026-04-14T06:54:16.020Z] [2026-04-14T06:54:16.020Z] TASK [wait for node "an-a01n01" to be back online] ***************************** [2026-04-14T06:57:41.065Z] ok: [an-a01n01 -> localhost] [2026-04-14T06:57:41.065Z] ok: [an-a01n02 -> localhost] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:57:41.065Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:57:41.065Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:57:41.065Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:57:41.065Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/tests/functional-tests/../../common/./wait_server_drbd_sync.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Wait up to 60 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Wait up to 60 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Wait up to 60 minutes for an-test-deploy3 drbd storage to be synced] ***** [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Wait up to 60 minutes for an-test-deploy4 drbd storage to be synced] ***** [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Wait up to 60 minutes for an-test-deploy5 drbd storage to be synced] ***** [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] PLAY [basic server tests] ****************************************************** [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T06:57:41.065Z] ok: [an-a01n01] [2026-04-14T06:57:41.065Z] ok: [an-a01n02] [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:57:41.065Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:57:41.065Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:57:41.065Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:57:41.065Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [check anvil-provision-server storage group] ****************************** [2026-04-14T06:57:41.065Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:41.065Z] [2026-04-14T06:57:41.065Z] TASK [get anvil-provision-server storage group] ******************************** [2026-04-14T06:57:47.320Z] changed: [an-a01n01] [2026-04-14T06:57:47.320Z] changed: [an-a01n02] [2026-04-14T06:57:47.320Z] [2026-04-14T06:57:47.320Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T06:57:47.320Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:47.320Z] [2026-04-14T06:57:47.320Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T06:57:52.386Z] changed: [an-a01n01] [2026-04-14T06:57:52.386Z] changed: [an-a01n02] [2026-04-14T06:57:52.386Z] [2026-04-14T06:57:52.386Z] TASK [deploy server 1 from "an-a01n01"] **************************************** [2026-04-14T06:57:52.386Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:52.386Z] [2026-04-14T06:57:52.386Z] TASK [check if server an-test-deploy1 exists] ********************************** [2026-04-14T06:57:53.172Z] skipping: [an-a01n02] [2026-04-14T06:57:53.172Z] ok: [an-a01n01] [2026-04-14T06:57:53.172Z] [2026-04-14T06:57:53.172Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T06:57:53.571Z] skipping: [an-a01n01] [2026-04-14T06:57:53.571Z] skipping: [an-a01n02] [2026-04-14T06:57:53.571Z] [2026-04-14T06:57:53.571Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker] **** [2026-04-14T06:57:53.571Z] skipping: [an-a01n01] [2026-04-14T06:57:53.571Z] skipping: [an-a01n02] [2026-04-14T06:57:53.571Z] [2026-04-14T06:57:53.571Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T06:57:53.571Z] skipping: [an-a01n01] [2026-04-14T06:57:53.571Z] skipping: [an-a01n02] [2026-04-14T06:57:53.571Z] [2026-04-14T06:57:53.571Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation] ********* [2026-04-14T06:57:53.571Z] skipping: [an-a01n01] [2026-04-14T06:57:53.571Z] skipping: [an-a01n02] [2026-04-14T06:57:53.571Z] [2026-04-14T06:57:53.571Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation] ****** [2026-04-14T06:57:53.571Z] skipping: [an-a01n01] [2026-04-14T06:57:53.571Z] skipping: [an-a01n02] [2026-04-14T06:57:53.571Z] [2026-04-14T06:57:53.571Z] TASK [deploy server 2 from "an-a01n02"] **************************************** [2026-04-14T06:57:53.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:53.939Z] [2026-04-14T06:57:53.939Z] TASK [check if server an-test-deploy2 exists] ********************************** [2026-04-14T06:57:55.204Z] skipping: [an-a01n01] [2026-04-14T06:57:55.204Z] ok: [an-a01n02] [2026-04-14T06:57:55.204Z] [2026-04-14T06:57:55.204Z] TASK [deploy server an-test-deploy2 from an-a01n02 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T06:57:55.572Z] skipping: [an-a01n01] [2026-04-14T06:57:55.572Z] skipping: [an-a01n02] [2026-04-14T06:57:55.572Z] [2026-04-14T06:57:55.572Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be started in pacemaker] **** [2026-04-14T06:57:55.572Z] skipping: [an-a01n01] [2026-04-14T06:57:55.572Z] skipping: [an-a01n02] [2026-04-14T06:57:55.572Z] [2026-04-14T06:57:55.572Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be synced] ***** [2026-04-14T06:57:55.572Z] skipping: [an-a01n01] [2026-04-14T06:57:55.572Z] skipping: [an-a01n02] [2026-04-14T06:57:55.572Z] [2026-04-14T06:57:55.572Z] TASK [Wait up to 10 minutes for an-test-deploy2 to start installation] ********* [2026-04-14T06:57:55.572Z] skipping: [an-a01n01] [2026-04-14T06:57:55.572Z] skipping: [an-a01n02] [2026-04-14T06:57:55.572Z] [2026-04-14T06:57:55.572Z] TASK [Wait up to 10 minutes for an-test-deploy2 to complete installation] ****** [2026-04-14T06:57:55.572Z] skipping: [an-a01n01] [2026-04-14T06:57:55.572Z] skipping: [an-a01n02] [2026-04-14T06:57:55.572Z] [2026-04-14T06:57:55.572Z] TASK [check servers status] **************************************************** [2026-04-14T06:57:55.939Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-14T06:57:55.939Z] [2026-04-14T06:57:55.939Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T06:57:57.756Z] ok: [an-a01n01] [2026-04-14T06:57:57.756Z] ok: [an-a01n02] [2026-04-14T06:57:57.756Z] [2026-04-14T06:57:57.756Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T06:58:00.323Z] ok: [an-a01n02] [2026-04-14T06:58:00.324Z] ok: [an-a01n01] [2026-04-14T06:58:00.324Z] [2026-04-14T06:58:00.324Z] PLAY [delete extra server tests] *********************************************** [2026-04-14T06:58:00.324Z] [2026-04-14T06:58:00.324Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T06:58:02.139Z] ok: [an-a01n01] [2026-04-14T06:58:02.139Z] ok: [an-a01n02] [2026-04-14T06:58:02.139Z] [2026-04-14T06:58:02.139Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T06:58:02.139Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T06:58:02.139Z] [2026-04-14T06:58:02.139Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T06:58:06.196Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T06:58:06.196Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T06:58:06.196Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T06:58:06.196Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T06:58:06.196Z] [2026-04-14T06:58:06.196Z] TASK [delete servers] ********************************************************** [2026-04-14T06:58:06.196Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T06:58:06.196Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T06:58:06.196Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T06:58:06.196Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T06:58:06.196Z] [2026-04-14T06:58:06.196Z] TASK [check if server an-test-deploy2 exists] ********************************** [2026-04-14T06:58:08.018Z] skipping: [an-a01n01] [2026-04-14T06:58:08.018Z] ok: [an-a01n02] [2026-04-14T06:58:08.018Z] [2026-04-14T06:58:08.018Z] TASK [delete server an-test-deploy2 from an-a01n02] **************************** [2026-04-14T06:58:17.339Z] skipping: [an-a01n01] [2026-04-14T06:58:17.339Z] changed: [an-a01n02] [2026-04-14T06:58:17.339Z] [2026-04-14T06:58:17.339Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker] ***** [2026-04-14T06:58:42.100Z] skipping: [an-a01n01] [2026-04-14T06:58:42.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (60 retries left). [2026-04-14T06:58:42.100Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker (59 retries left). [2026-04-14T06:58:42.100Z] ok: [an-a01n02] [2026-04-14T06:58:42.100Z] [2026-04-14T06:58:42.100Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed] **** [2026-04-14T06:58:42.100Z] skipping: [an-a01n01] [2026-04-14T06:58:42.100Z] ok: [an-a01n02] [2026-04-14T06:58:42.100Z] [2026-04-14T06:58:42.100Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed] ****** [2026-04-14T06:58:42.100Z] skipping: [an-a01n01] [2026-04-14T06:58:42.100Z] ok: [an-a01n02] [2026-04-14T06:58:42.100Z] [2026-04-14T06:58:42.100Z] TASK [check if server an-test-deploy3 exists] ********************************** [2026-04-14T06:58:43.915Z] skipping: [an-a01n01] [2026-04-14T06:58:43.915Z] ok: [an-a01n02] [2026-04-14T06:58:43.915Z] [2026-04-14T06:58:43.915Z] TASK [delete server an-test-deploy3 from an-a01n02] **************************** [2026-04-14T06:58:53.238Z] skipping: [an-a01n01] [2026-04-14T06:58:53.238Z] changed: [an-a01n02] [2026-04-14T06:58:53.238Z] [2026-04-14T06:58:53.238Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker] ***** [2026-04-14T07:00:04.396Z] skipping: [an-a01n01] [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (60 retries left). [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (59 retries left). [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (58 retries left). [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (57 retries left). [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (56 retries left). [2026-04-14T07:00:04.396Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker (55 retries left). [2026-04-14T07:00:04.396Z] ok: [an-a01n02] [2026-04-14T07:00:04.396Z] [2026-04-14T07:00:04.396Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed] **** [2026-04-14T07:00:04.396Z] skipping: [an-a01n01] [2026-04-14T07:00:04.396Z] ok: [an-a01n02] [2026-04-14T07:00:04.396Z] [2026-04-14T07:00:04.396Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed] ****** [2026-04-14T07:00:04.396Z] skipping: [an-a01n01] [2026-04-14T07:00:04.396Z] ok: [an-a01n02] [2026-04-14T07:00:04.396Z] [2026-04-14T07:00:04.396Z] TASK [check if server an-test-deploy4 exists] ********************************** [2026-04-14T07:00:06.256Z] skipping: [an-a01n01] [2026-04-14T07:00:06.256Z] ok: [an-a01n02] [2026-04-14T07:00:06.256Z] [2026-04-14T07:00:06.256Z] TASK [delete server an-test-deploy4 from an-a01n02] **************************** [2026-04-14T07:00:14.314Z] skipping: [an-a01n01] [2026-04-14T07:00:14.314Z] changed: [an-a01n02] [2026-04-14T07:00:14.314Z] [2026-04-14T07:00:14.314Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker] ***** [2026-04-14T07:01:14.097Z] skipping: [an-a01n01] [2026-04-14T07:01:14.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (60 retries left). [2026-04-14T07:01:14.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (59 retries left). [2026-04-14T07:01:14.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (58 retries left). [2026-04-14T07:01:14.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (57 retries left). [2026-04-14T07:01:14.097Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker (56 retries left). [2026-04-14T07:01:14.097Z] ok: [an-a01n02] [2026-04-14T07:01:14.097Z] [2026-04-14T07:01:14.097Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed] **** [2026-04-14T07:01:14.097Z] skipping: [an-a01n01] [2026-04-14T07:01:14.097Z] ok: [an-a01n02] [2026-04-14T07:01:14.097Z] [2026-04-14T07:01:14.097Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed] ****** [2026-04-14T07:01:14.884Z] skipping: [an-a01n01] [2026-04-14T07:01:14.884Z] ok: [an-a01n02] [2026-04-14T07:01:14.884Z] [2026-04-14T07:01:14.884Z] TASK [check if server an-test-deploy5 exists] ********************************** [2026-04-14T07:01:16.710Z] skipping: [an-a01n01] [2026-04-14T07:01:16.710Z] ok: [an-a01n02] [2026-04-14T07:01:16.710Z] [2026-04-14T07:01:16.710Z] TASK [delete server an-test-deploy5 from an-a01n02] **************************** [2026-04-14T07:01:24.617Z] skipping: [an-a01n01] [2026-04-14T07:01:24.617Z] changed: [an-a01n02] [2026-04-14T07:01:24.617Z] [2026-04-14T07:01:24.617Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker] ***** [2026-04-14T07:02:26.328Z] skipping: [an-a01n01] [2026-04-14T07:02:26.328Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (60 retries left). [2026-04-14T07:02:26.328Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (59 retries left). [2026-04-14T07:02:26.328Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (58 retries left). [2026-04-14T07:02:26.328Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (57 retries left). [2026-04-14T07:02:26.328Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker (56 retries left). [2026-04-14T07:02:26.328Z] ok: [an-a01n02] [2026-04-14T07:02:26.328Z] [2026-04-14T07:02:26.328Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed] **** [2026-04-14T07:02:26.328Z] skipping: [an-a01n01] [2026-04-14T07:02:26.328Z] ok: [an-a01n02] [2026-04-14T07:02:26.328Z] [2026-04-14T07:02:26.328Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed] ****** [2026-04-14T07:02:26.328Z] skipping: [an-a01n01] [2026-04-14T07:02:26.328Z] ok: [an-a01n02] [2026-04-14T07:02:26.328Z] [2026-04-14T07:02:26.328Z] PLAY [create server for DR testing] ******************************************** [2026-04-14T07:02:26.328Z] [2026-04-14T07:02:26.328Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:02:27.914Z] ok: [an-a01n02] [2026-04-14T07:02:27.914Z] ok: [an-a01n01] [2026-04-14T07:02:27.914Z] [2026-04-14T07:02:27.914Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T07:02:27.914Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T07:02:27.914Z] [2026-04-14T07:02:27.914Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T07:02:32.060Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T07:02:32.060Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T07:02:32.060Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T07:02:32.060Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T07:02:32.060Z] [2026-04-14T07:02:32.060Z] TASK [check anvil-provision-server storage group] ****************************** [2026-04-14T07:02:32.060Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-14T07:02:32.060Z] [2026-04-14T07:02:32.060Z] TASK [get anvil-provision-server storage group] ******************************** [2026-04-14T07:02:37.142Z] changed: [an-a01n02] [2026-04-14T07:02:37.142Z] changed: [an-a01n01] [2026-04-14T07:02:37.142Z] [2026-04-14T07:02:37.142Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T07:02:37.142Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-14T07:02:37.142Z] [2026-04-14T07:02:37.142Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T07:02:42.346Z] changed: [an-a01n02] [2026-04-14T07:02:42.346Z] changed: [an-a01n01] [2026-04-14T07:02:42.346Z] [2026-04-14T07:02:42.346Z] TASK [deploy server 1 from "an-a01n01"] **************************************** [2026-04-14T07:02:42.746Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-14T07:02:42.746Z] [2026-04-14T07:02:42.746Z] TASK [check if server an-test-deploy1 exists] ********************************** [2026-04-14T07:02:44.014Z] skipping: [an-a01n02] [2026-04-14T07:02:44.014Z] ok: [an-a01n01] [2026-04-14T07:02:44.014Z] [2026-04-14T07:02:44.014Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T07:02:44.382Z] skipping: [an-a01n01] [2026-04-14T07:02:44.382Z] skipping: [an-a01n02] [2026-04-14T07:02:44.382Z] [2026-04-14T07:02:44.382Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker] **** [2026-04-14T07:02:44.382Z] skipping: [an-a01n01] [2026-04-14T07:02:44.382Z] skipping: [an-a01n02] [2026-04-14T07:02:44.382Z] [2026-04-14T07:02:44.382Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T07:02:44.382Z] skipping: [an-a01n01] [2026-04-14T07:02:44.382Z] skipping: [an-a01n02] [2026-04-14T07:02:44.382Z] [2026-04-14T07:02:44.382Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation] ********* [2026-04-14T07:02:44.382Z] skipping: [an-a01n01] [2026-04-14T07:02:44.382Z] skipping: [an-a01n02] [2026-04-14T07:02:44.382Z] [2026-04-14T07:02:44.382Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation] ****** [2026-04-14T07:02:44.382Z] skipping: [an-a01n01] [2026-04-14T07:02:44.382Z] skipping: [an-a01n02] [2026-04-14T07:02:44.382Z] [2026-04-14T07:02:44.382Z] TASK [check servers status] **************************************************** [2026-04-14T07:02:44.751Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-14T07:02:44.751Z] [2026-04-14T07:02:44.751Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T07:02:46.570Z] ok: [an-a01n01] [2026-04-14T07:02:46.570Z] ok: [an-a01n02] [2026-04-14T07:02:46.570Z] [2026-04-14T07:02:46.570Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T07:02:49.028Z] ok: [an-a01n02] [2026-04-14T07:02:49.028Z] ok: [an-a01n01] [2026-04-14T07:02:49.028Z] [2026-04-14T07:02:49.028Z] PLAY [DR tests] **************************************************************** [2026-04-14T07:02:49.028Z] [2026-04-14T07:02:49.028Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:02:50.853Z] ok: [an-a01dr01] [2026-04-14T07:02:50.853Z] [2026-04-14T07:02:50.853Z] TASK [check DR is correctly deployed] ****************************************** [2026-04-14T07:02:54.917Z] changed: [an-a01dr01] [2026-04-14T07:02:54.917Z] [2026-04-14T07:02:54.917Z] TASK [check DR host is registered in the DB] *********************************** [2026-04-14T07:02:58.987Z] changed: [an-a01dr01] [2026-04-14T07:02:58.987Z] [2026-04-14T07:02:58.987Z] TASK [check no DR hosts are linked] ******************************************** [2026-04-14T07:03:02.822Z] changed: [an-a01dr01] [2026-04-14T07:03:02.822Z] [2026-04-14T07:03:02.822Z] TASK [link DR host to anvil] *************************************************** [2026-04-14T07:03:07.626Z] changed: [an-a01dr01] [2026-04-14T07:03:07.626Z] [2026-04-14T07:03:07.626Z] TASK [verify DR host is linked to anvil] *************************************** [2026-04-14T07:03:12.869Z] changed: [an-a01dr01] [2026-04-14T07:03:12.869Z] [2026-04-14T07:03:12.869Z] TASK [link DR vg to storage group] ********************************************* [2026-04-14T07:03:19.360Z] changed: [an-a01dr01] [2026-04-14T07:03:19.360Z] [2026-04-14T07:03:19.360Z] TASK [verify DR vg is linked to storage group] ********************************* [2026-04-14T07:03:22.563Z] changed: [an-a01dr01] [2026-04-14T07:03:22.563Z] [2026-04-14T07:03:22.563Z] TASK [protect an-test-deploy1 short-throw] ************************************* [2026-04-14T07:03:27.429Z] changed: [an-a01dr01] [2026-04-14T07:03:27.429Z] [2026-04-14T07:03:27.429Z] TASK [verify an-test-deploy1 DR drbd status short-throw] *********************** [2026-04-14T07:03:27.429Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-14T07:03:27.429Z] [2026-04-14T07:03:27.429Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created] *** [2026-04-14T07:04:01.628Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-14T07:04:01.628Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-14T07:04:01.628Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-14T07:04:01.628Z] ok: [an-a01dr01] [2026-04-14T07:04:01.628Z] [2026-04-14T07:04:01.628Z] TASK [Check correct drbd protocol is configured] ******************************* [2026-04-14T07:04:01.628Z] changed: [an-a01dr01] [2026-04-14T07:04:01.628Z] [2026-04-14T07:04:01.628Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T07:06:12.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-14T07:06:12.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-14T07:06:12.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-14T07:06:12.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-14T07:06:12.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-14T07:06:12.043Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-14T07:06:12.043Z] ok: [an-a01dr01] [2026-04-14T07:06:12.043Z] [2026-04-14T07:06:12.043Z] TASK [verify disconnect/update/connect features short-throw] ******************* [2026-04-14T07:06:12.043Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-14T07:06:12.043Z] [2026-04-14T07:06:12.043Z] TASK [Verify an-test-deploy1 drbd storage is synced] *************************** [2026-04-14T07:06:12.043Z] ok: [an-a01dr01] [2026-04-14T07:06:12.043Z] [2026-04-14T07:06:12.043Z] TASK [Disconnect an-test-deploy1 drbd storage] ********************************* [2026-04-14T07:06:12.043Z] changed: [an-a01dr01] [2026-04-14T07:06:12.043Z] [2026-04-14T07:06:12.043Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected] *** [2026-04-14T07:06:35.806Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-14T07:06:35.806Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-14T07:06:35.806Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (118 retries left). [2026-04-14T07:06:35.806Z] ok: [an-a01dr01] [2026-04-14T07:06:35.806Z] [2026-04-14T07:06:35.806Z] TASK [Inject random data to an-test-deploy1] *********************************** [2026-04-14T07:08:46.250Z] changed: [an-a01dr01 -> localhost] [2026-04-14T07:08:46.250Z] [2026-04-14T07:08:46.250Z] TASK [Update an-test-deploy1 drbd storage] ************************************* [2026-04-14T07:08:47.037Z] changed: [an-a01dr01] [2026-04-14T07:08:47.037Z] [2026-04-14T07:08:47.037Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected] *** [2026-04-14T07:09:11.732Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-14T07:09:11.732Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-14T07:09:11.732Z] ok: [an-a01dr01] [2026-04-14T07:09:11.732Z] [2026-04-14T07:09:11.732Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected] *** [2026-04-14T07:09:31.503Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-14T07:09:31.503Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-14T07:09:31.503Z] ok: [an-a01dr01] [2026-04-14T07:09:31.503Z] [2026-04-14T07:09:31.503Z] TASK [Reconnect an-test-deploy1 drbd storage] ********************************** [2026-04-14T07:09:35.591Z] changed: [an-a01dr01] [2026-04-14T07:09:35.591Z] [2026-04-14T07:09:35.591Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected] *** [2026-04-14T07:09:59.776Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-14T07:09:59.776Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-14T07:09:59.776Z] ok: [an-a01dr01] [2026-04-14T07:09:59.776Z] [2026-04-14T07:09:59.776Z] TASK [remove an-test-deploy1 protection short-throw] *************************** [2026-04-14T07:10:02.289Z] changed: [an-a01dr01] [2026-04-14T07:10:02.289Z] [2026-04-14T07:10:02.289Z] TASK [verify an-test-deploy1 DR drbd clean status short-throw] ***************** [2026-04-14T07:10:02.684Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-14T07:10:02.684Z] [2026-04-14T07:10:02.684Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed] **** [2026-04-14T07:10:51.997Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-14T07:10:51.997Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-14T07:10:51.997Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-14T07:10:51.997Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (57 retries left). [2026-04-14T07:10:51.997Z] ok: [an-a01dr01] [2026-04-14T07:10:51.997Z] [2026-04-14T07:10:51.997Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed] ****** [2026-04-14T07:10:51.997Z] ok: [an-a01dr01] [2026-04-14T07:10:51.997Z] [2026-04-14T07:10:51.997Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed] *** [2026-04-14T07:11:21.420Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-14T07:11:21.420Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-14T07:11:21.420Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-04-14T07:11:21.420Z] ok: [an-a01dr01] [2026-04-14T07:11:21.420Z] [2026-04-14T07:11:21.420Z] TASK [check drbd sync status on nodes after disconnect short-throw] ************ [2026-04-14T07:11:21.420Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-14T07:11:21.420Z] [2026-04-14T07:11:21.420Z] TASK [protect an-test-deploy1 sync] ******************************************** [2026-04-14T07:11:24.632Z] changed: [an-a01dr01] [2026-04-14T07:11:24.632Z] [2026-04-14T07:11:24.632Z] TASK [verify an-test-deploy1 DR drbd status sync] ****************************** [2026-04-14T07:11:24.632Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_sync.yml for an-a01dr01 [2026-04-14T07:11:24.632Z] [2026-04-14T07:11:24.632Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created] *** [2026-04-14T07:11:59.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (120 retries left). [2026-04-14T07:11:59.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (119 retries left). [2026-04-14T07:11:59.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage resource to be created (118 retries left). [2026-04-14T07:11:59.218Z] ok: [an-a01dr01] [2026-04-14T07:11:59.218Z] [2026-04-14T07:11:59.218Z] TASK [Check correct drbd protocol is configured] ******************************* [2026-04-14T07:11:59.218Z] changed: [an-a01dr01] [2026-04-14T07:11:59.218Z] [2026-04-14T07:11:59.218Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (120 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (119 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (118 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (117 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (116 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (115 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (114 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (113 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (112 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (111 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (110 retries left). [2026-04-14T07:14:08.673Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be synced (109 retries left). [2026-04-14T07:14:08.673Z] ok: [an-a01dr01] [2026-04-14T07:14:08.673Z] [2026-04-14T07:14:08.673Z] TASK [verify disconnect/update/connect features sync] ************************** [2026-04-14T07:14:08.673Z] included: /srv/anvil/anvil-test-suite/playbooks/common/check_server_drbd_dr_management.yml for an-a01dr01 [2026-04-14T07:14:08.673Z] [2026-04-14T07:14:08.673Z] TASK [Verify an-test-deploy1 drbd storage is synced] *************************** [2026-04-14T07:14:08.673Z] ok: [an-a01dr01] [2026-04-14T07:14:08.673Z] [2026-04-14T07:14:08.673Z] TASK [Disconnect an-test-deploy1 drbd storage] ********************************* [2026-04-14T07:14:11.894Z] changed: [an-a01dr01] [2026-04-14T07:14:11.894Z] [2026-04-14T07:14:11.894Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected] *** [2026-04-14T07:14:35.763Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-14T07:14:35.763Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (119 retries left). [2026-04-14T07:14:35.763Z] ok: [an-a01dr01] [2026-04-14T07:14:35.763Z] [2026-04-14T07:14:35.763Z] TASK [Inject random data to an-test-deploy1] *********************************** [2026-04-14T07:16:45.507Z] changed: [an-a01dr01 -> localhost] [2026-04-14T07:16:45.507Z] [2026-04-14T07:16:45.507Z] TASK [Update an-test-deploy1 drbd storage] ************************************* [2026-04-14T07:16:45.507Z] changed: [an-a01dr01] [2026-04-14T07:16:45.507Z] [2026-04-14T07:16:45.507Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected] *** [2026-04-14T07:17:09.513Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-14T07:17:09.513Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-14T07:17:09.513Z] ok: [an-a01dr01] [2026-04-14T07:17:09.513Z] [2026-04-14T07:17:09.513Z] TASK [Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected] *** [2026-04-14T07:17:19.042Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 minutes for an-test-deploy1 drbd storage to be disconnected (120 retries left). [2026-04-14T07:17:19.042Z] ok: [an-a01dr01] [2026-04-14T07:17:19.042Z] [2026-04-14T07:17:19.042Z] TASK [Reconnect an-test-deploy1 drbd storage] ********************************** [2026-04-14T07:17:22.917Z] changed: [an-a01dr01] [2026-04-14T07:17:22.917Z] [2026-04-14T07:17:22.917Z] TASK [Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected] *** [2026-04-14T07:17:57.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (120 retries left). [2026-04-14T07:17:57.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (119 retries left). [2026-04-14T07:17:57.218Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 20 timeout for an-test-deploy1 drbd storage to be (re)connected (118 retries left). [2026-04-14T07:17:57.218Z] ok: [an-a01dr01] [2026-04-14T07:17:57.218Z] [2026-04-14T07:17:57.218Z] TASK [remove an-test-deploy1 protection sync] ********************************** [2026-04-14T07:18:00.444Z] changed: [an-a01dr01] [2026-04-14T07:18:00.444Z] [2026-04-14T07:18:00.444Z] TASK [verify an-test-deploy1 DR drbd clean status sync] ************************ [2026-04-14T07:18:00.444Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_dr_clean.yml for an-a01dr01 [2026-04-14T07:18:00.444Z] [2026-04-14T07:18:00.444Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed] **** [2026-04-14T07:18:34.956Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (60 retries left). [2026-04-14T07:18:34.956Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (59 retries left). [2026-04-14T07:18:34.956Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed (58 retries left). [2026-04-14T07:18:34.956Z] ok: [an-a01dr01] [2026-04-14T07:18:34.956Z] [2026-04-14T07:18:34.956Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed] ****** [2026-04-14T07:18:34.956Z] ok: [an-a01dr01] [2026-04-14T07:18:34.956Z] [2026-04-14T07:18:34.956Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed] *** [2026-04-14T07:19:09.032Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (60 retries left). [2026-04-14T07:19:09.032Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (59 retries left). [2026-04-14T07:19:09.032Z] FAILED - RETRYING: [an-a01dr01]: Wait up to 10 minutes for an-test-deploy1 drbd storage resource to be removed (58 retries left). [2026-04-14T07:19:09.032Z] ok: [an-a01dr01] [2026-04-14T07:19:09.032Z] [2026-04-14T07:19:09.032Z] TASK [check drbd sync status on nodes after disconnect sync] ******************* [2026-04-14T07:19:09.032Z] changed: [an-a01dr01 -> an-a01n01] [2026-04-14T07:19:09.032Z] [2026-04-14T07:19:09.032Z] TASK [unlink DR vg from storage group] ***************************************** [2026-04-14T07:19:13.594Z] changed: [an-a01dr01] [2026-04-14T07:19:13.594Z] [2026-04-14T07:19:13.594Z] TASK [verify DR vg is unlinked from storage group] ***************************** [2026-04-14T07:19:16.799Z] changed: [an-a01dr01] [2026-04-14T07:19:16.800Z] [2026-04-14T07:19:16.800Z] TASK [unlink DR host from anvil] *********************************************** [2026-04-14T07:19:21.952Z] changed: [an-a01dr01] [2026-04-14T07:19:21.952Z] [2026-04-14T07:19:21.952Z] TASK [check no DR hosts are linked] ******************************************** [2026-04-14T07:19:26.060Z] changed: [an-a01dr01] [2026-04-14T07:19:26.060Z] [2026-04-14T07:19:26.060Z] PLAY [upgrade server test] ***************************************************** [2026-04-14T07:19:26.060Z] [2026-04-14T07:19:26.060Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:19:27.332Z] ok: [an-a01n01] [2026-04-14T07:19:27.332Z] ok: [an-a01n02] [2026-04-14T07:19:27.332Z] [2026-04-14T07:19:27.332Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T07:19:27.332Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T07:19:27.332Z] [2026-04-14T07:19:27.332Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T07:19:31.777Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T07:19:31.777Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T07:19:31.777Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T07:19:31.777Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T07:19:31.777Z] [2026-04-14T07:19:31.777Z] TASK [check anvil-provision-server storage group] ****************************** [2026-04-14T07:19:31.777Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_storage_group.yml for an-a01n01, an-a01n02 [2026-04-14T07:19:31.777Z] [2026-04-14T07:19:31.777Z] TASK [get anvil-provision-server storage group] ******************************** [2026-04-14T07:19:36.851Z] changed: [an-a01n02] [2026-04-14T07:19:36.851Z] changed: [an-a01n01] [2026-04-14T07:19:36.851Z] [2026-04-14T07:19:36.851Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T07:19:36.851Z] included: /srv/anvil/anvil-test-suite/playbooks/common/get_install_iso.yml for an-a01n01, an-a01n02 [2026-04-14T07:19:36.851Z] [2026-04-14T07:19:36.851Z] TASK [check anvil-provision-server install isos] ******************************* [2026-04-14T07:19:41.928Z] changed: [an-a01n02] [2026-04-14T07:19:41.928Z] changed: [an-a01n01] [2026-04-14T07:19:41.928Z] [2026-04-14T07:19:41.928Z] TASK [deploy server 1 from "an-a01n01"] **************************************** [2026-04-14T07:19:41.928Z] included: /srv/anvil/anvil-test-suite/playbooks/common/deploy_server.yml for an-a01n01, an-a01n02 [2026-04-14T07:19:41.928Z] [2026-04-14T07:19:41.928Z] TASK [check if server an-test-deploy1 exists] ********************************** [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] ok: [an-a01n01] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [deploy server an-test-deploy1 from an-a01n01 on storage 93b1fb44-51ba-4d2d-a523-9ad7738918ce] *** [2026-04-14T07:19:43.748Z] skipping: [an-a01n01] [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker] **** [2026-04-14T07:19:43.748Z] skipping: [an-a01n01] [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T07:19:43.748Z] skipping: [an-a01n01] [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [Wait up to 10 minutes for an-test-deploy1 to start installation] ********* [2026-04-14T07:19:43.748Z] skipping: [an-a01n01] [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [Wait up to 10 minutes for an-test-deploy1 to complete installation] ****** [2026-04-14T07:19:43.748Z] skipping: [an-a01n01] [2026-04-14T07:19:43.748Z] skipping: [an-a01n02] [2026-04-14T07:19:43.748Z] [2026-04-14T07:19:43.748Z] TASK [check servers status] **************************************************** [2026-04-14T07:19:44.118Z] included: /srv/anvil/anvil-test-suite/playbooks/common/server_started.yml for an-a01n01, an-a01n02 [2026-04-14T07:19:44.118Z] [2026-04-14T07:19:44.118Z] TASK [wait up to 20 minutes for servers to be Started] ************************* [2026-04-14T07:19:45.936Z] ok: [an-a01n02] [2026-04-14T07:19:45.936Z] ok: [an-a01n01] [2026-04-14T07:19:45.936Z] [2026-04-14T07:19:45.936Z] TASK [check servers are started all on the same node] ************************** [2026-04-14T07:19:47.756Z] ok: [an-a01n01] [2026-04-14T07:19:47.756Z] ok: [an-a01n02] [2026-04-14T07:19:47.756Z] [2026-04-14T07:19:47.756Z] PLAY [enable upgrade repo] ***************************************************** [2026-04-14T07:19:47.756Z] [2026-04-14T07:19:47.756Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:19:50.231Z] ok: [an-a01n02] [2026-04-14T07:19:50.231Z] ok: [an-a01n01] [2026-04-14T07:19:50.231Z] ok: [an-a01dr01] [2026-04-14T07:19:50.231Z] ok: [an-striker02] [2026-04-14T07:19:50.231Z] ok: [an-striker01] [2026-04-14T07:19:50.231Z] [2026-04-14T07:19:50.231Z] TASK [enable anvil test-update repo] ******************************************* [2026-04-14T07:19:53.475Z] changed: [an-a01n02] [2026-04-14T07:19:53.475Z] changed: [an-a01dr01] [2026-04-14T07:19:53.475Z] changed: [an-a01n01] [2026-04-14T07:19:53.475Z] changed: [an-striker02] [2026-04-14T07:19:53.475Z] changed: [an-striker01] [2026-04-14T07:19:53.475Z] [2026-04-14T07:19:53.475Z] TASK [store current uptime] **************************************************** [2026-04-14T07:19:53.475Z] ok: [an-striker01] [2026-04-14T07:19:53.475Z] ok: [an-striker02] [2026-04-14T07:19:53.475Z] ok: [an-a01n01] [2026-04-14T07:19:53.475Z] ok: [an-a01n02] [2026-04-14T07:19:53.475Z] ok: [an-a01dr01] [2026-04-14T07:19:53.475Z] [2026-04-14T07:19:53.475Z] PLAY [start upgrade process] *************************************************** [2026-04-14T07:19:53.475Z] [2026-04-14T07:19:53.475Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:19:55.300Z] ok: [an-striker01] [2026-04-14T07:19:55.300Z] [2026-04-14T07:19:55.300Z] TASK [upgrade and reboot all nodes but an-striker01] *************************** [2026-04-14T07:43:00.795Z] changed: [an-striker01] [2026-04-14T07:43:00.795Z] [2026-04-14T07:43:00.795Z] TASK [reboot striker01] ******************************************************** [2026-04-14T07:43:29.417Z] changed: [an-striker01] [2026-04-14T07:43:29.417Z] [2026-04-14T07:43:29.417Z] PLAY [reconnect all nodes] ***************************************************** [2026-04-14T07:43:29.417Z] [2026-04-14T07:43:29.417Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:43:35.676Z] ok: [an-a01n02] [2026-04-14T07:43:35.676Z] ok: [an-a01dr01] [2026-04-14T07:43:35.676Z] ok: [an-a01n01] [2026-04-14T07:43:35.676Z] ok: [an-striker02] [2026-04-14T07:43:35.676Z] ok: [an-striker01] [2026-04-14T07:43:35.676Z] [2026-04-14T07:43:35.676Z] TASK [wait for node "an-striker01" to be back online] ************************** [2026-04-14T07:43:35.676Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_node.yml for an-striker01, an-striker02, an-a01n01, an-a01n02, an-a01dr01 [2026-04-14T07:43:35.676Z] [2026-04-14T07:43:35.676Z] TASK [wait for node "all" to be back online] *********************************** [2026-04-14T07:46:46.416Z] ok: [an-striker01 -> localhost] [2026-04-14T07:46:46.416Z] ok: [an-a01n01 -> localhost] [2026-04-14T07:46:46.416Z] ok: [an-striker02 -> localhost] [2026-04-14T07:46:46.416Z] ok: [an-a01n02 -> localhost] [2026-04-14T07:46:46.416Z] ok: [an-a01dr01 -> localhost] [2026-04-14T07:46:46.416Z] [2026-04-14T07:46:46.416Z] TASK [check nodes uptime after reboot] ***************************************** [2026-04-14T07:46:47.204Z] changed: [an-a01n02] [2026-04-14T07:46:47.204Z] changed: [an-a01dr01] [2026-04-14T07:46:47.204Z] changed: [an-a01n01] [2026-04-14T07:46:47.204Z] changed: [an-striker02] [2026-04-14T07:46:47.204Z] changed: [an-striker01] [2026-04-14T07:46:47.204Z] [2026-04-14T07:46:47.204Z] PLAY [check all nodes status after upgrade] ************************************ [2026-04-14T07:46:47.204Z] [2026-04-14T07:46:47.204Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:46:49.019Z] ok: [an-a01dr01] [2026-04-14T07:46:49.019Z] ok: [an-a01n02] [2026-04-14T07:46:49.019Z] ok: [an-striker02] [2026-04-14T07:46:49.019Z] ok: [an-a01n01] [2026-04-14T07:46:49.019Z] ok: [an-striker01] [2026-04-14T07:46:49.019Z] [2026-04-14T07:46:49.019Z] TASK [check perl-Log-Journald package version] ********************************* [2026-04-14T07:46:52.213Z] changed: [an-a01n02] [2026-04-14T07:46:52.213Z] changed: [an-a01dr01] [2026-04-14T07:46:52.213Z] changed: [an-a01n01] [2026-04-14T07:46:52.213Z] changed: [an-striker02] [2026-04-14T07:46:52.213Z] changed: [an-striker01] [2026-04-14T07:46:52.213Z] [2026-04-14T07:46:52.213Z] PLAY [check nodes status after upgrade] **************************************** [2026-04-14T07:46:52.213Z] [2026-04-14T07:46:52.213Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:46:53.479Z] ok: [an-a01n02] [2026-04-14T07:46:53.479Z] ok: [an-a01dr01] [2026-04-14T07:46:53.479Z] ok: [an-a01n01] [2026-04-14T07:46:53.479Z] [2026-04-14T07:46:53.479Z] TASK [check kmod-drdb package version] ***************************************** [2026-04-14T07:46:55.295Z] changed: [an-a01dr01] [2026-04-14T07:46:55.295Z] changed: [an-a01n02] [2026-04-14T07:46:55.295Z] changed: [an-a01n01] [2026-04-14T07:46:55.295Z] [2026-04-14T07:46:55.295Z] PLAY [check hosted node status after upgrade] ********************************** [2026-04-14T07:46:55.295Z] [2026-04-14T07:46:55.295Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:46:56.561Z] ok: [an-a01n02] [2026-04-14T07:46:56.561Z] ok: [an-a01n01] [2026-04-14T07:46:56.561Z] [2026-04-14T07:46:56.561Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T07:46:56.929Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T07:46:56.929Z] [2026-04-14T07:46:56.929Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T07:47:01.438Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T07:47:01.438Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T07:47:01.438Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T07:47:01.438Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T07:47:01.438Z] [2026-04-14T07:47:01.438Z] TASK [wait servers to be started in pacemaker] ********************************* [2026-04-14T07:47:01.438Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_in_pacemaker.yml for an-a01n01, an-a01n02 [2026-04-14T07:47:01.438Z] [2026-04-14T07:47:01.438Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be started in pacemaker] **** [2026-04-14T07:47:02.702Z] ok: [an-a01n01] [2026-04-14T07:47:02.702Z] ok: [an-a01n02] [2026-04-14T07:47:02.702Z] [2026-04-14T07:47:02.702Z] TASK [wait servers drbd storage to be synced] ********************************** [2026-04-14T07:47:02.702Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_drbd_sync.yml for an-a01n01, an-a01n02 [2026-04-14T07:47:02.702Z] [2026-04-14T07:47:02.702Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be synced] ***** [2026-04-14T07:47:03.071Z] ok: [an-a01n02] [2026-04-14T07:47:03.071Z] ok: [an-a01n01] [2026-04-14T07:47:03.071Z] [2026-04-14T07:47:03.071Z] TASK [wait servers to respond to ping] ***************************************** [2026-04-14T07:47:03.441Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ping.yml for an-a01n01, an-a01n02 [2026-04-14T07:47:03.441Z] [2026-04-14T07:47:03.441Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ping] ************ [2026-04-14T07:47:05.256Z] ok: [an-a01n01] [2026-04-14T07:47:05.256Z] ok: [an-a01n02] [2026-04-14T07:47:05.256Z] [2026-04-14T07:47:05.256Z] TASK [wait servers to respond to ssh] ****************************************** [2026-04-14T07:47:05.256Z] included: /srv/anvil/anvil-test-suite/playbooks/common/wait_server_ssh.yml for an-a01n01, an-a01n02 [2026-04-14T07:47:05.256Z] [2026-04-14T07:47:05.256Z] TASK [Wait up to 10 minutes for an-test-deploy1 to respond to ssh] ************* [2026-04-14T07:47:06.043Z] ok: [an-a01n01] [2026-04-14T07:47:06.043Z] ok: [an-a01n02] [2026-04-14T07:47:06.043Z] [2026-04-14T07:47:06.043Z] PLAY [test anvil package self update] ****************************************** [2026-04-14T07:47:06.043Z] [2026-04-14T07:47:06.043Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:47:07.861Z] ok: [an-a01dr01] [2026-04-14T07:47:07.861Z] ok: [an-a01n02] [2026-04-14T07:47:07.861Z] ok: [an-a01n01] [2026-04-14T07:47:07.861Z] ok: [an-striker01] [2026-04-14T07:47:07.861Z] ok: [an-striker02] [2026-04-14T07:47:07.861Z] [2026-04-14T07:47:07.861Z] TASK [stop anvil and scan-core services] *************************************** [2026-04-14T07:47:11.053Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-14T07:47:11.053Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-14T07:47:11.053Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-14T07:47:11.053Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-14T07:47:11.053Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-14T07:47:11.053Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-14T07:47:11.053Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-14T07:47:11.053Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-14T07:47:11.053Z] changed: [an-striker01] => (item=scancore.service) [2026-04-14T07:47:11.053Z] changed: [an-striker02] => (item=scancore.service) [2026-04-14T07:47:11.053Z] [2026-04-14T07:47:11.053Z] PLAY [test striker package self update] **************************************** [2026-04-14T07:47:11.053Z] [2026-04-14T07:47:11.053Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:47:12.317Z] ok: [an-striker01] [2026-04-14T07:47:12.317Z] ok: [an-striker02] [2026-04-14T07:47:12.317Z] [2026-04-14T07:47:12.317Z] TASK [reinstall anvil-core and anvil-striker] ********************************** [2026-04-14T07:47:32.443Z] changed: [an-striker02] [2026-04-14T07:47:32.443Z] changed: [an-striker01] [2026-04-14T07:47:32.443Z] [2026-04-14T07:47:32.443Z] PLAY [test node package self update] ******************************************* [2026-04-14T07:47:32.443Z] [2026-04-14T07:47:32.443Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:47:32.443Z] ok: [an-a01n02] [2026-04-14T07:47:32.443Z] ok: [an-a01n01] [2026-04-14T07:47:32.443Z] [2026-04-14T07:47:32.443Z] TASK [reinstall anvil-core and anvil-node] ************************************* [2026-04-14T07:47:48.890Z] changed: [an-a01n01] [2026-04-14T07:47:48.890Z] changed: [an-a01n02] [2026-04-14T07:47:48.890Z] [2026-04-14T07:47:48.890Z] PLAY [test dr package self update] ********************************************* [2026-04-14T07:47:48.890Z] [2026-04-14T07:47:48.890Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:47:49.677Z] ok: [an-a01dr01] [2026-04-14T07:47:49.677Z] [2026-04-14T07:47:49.677Z] TASK [reinstall anvil-core and anvil-dr] *************************************** [2026-04-14T07:48:06.090Z] changed: [an-a01dr01] [2026-04-14T07:48:06.090Z] [2026-04-14T07:48:06.090Z] PLAY [test anvil package self update] ****************************************** [2026-04-14T07:48:06.090Z] [2026-04-14T07:48:06.090Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:48:06.772Z] ok: [an-a01dr01] [2026-04-14T07:48:06.772Z] ok: [an-striker02] [2026-04-14T07:48:06.772Z] ok: [an-a01n02] [2026-04-14T07:48:06.772Z] ok: [an-striker01] [2026-04-14T07:48:06.772Z] ok: [an-a01n01] [2026-04-14T07:48:06.772Z] [2026-04-14T07:48:06.772Z] TASK [start anvil and scan-core services] ************************************** [2026-04-14T07:48:09.230Z] changed: [an-a01n02] => (item=anvil-daemon.service) [2026-04-14T07:48:09.230Z] changed: [an-a01n01] => (item=anvil-daemon.service) [2026-04-14T07:48:09.230Z] changed: [an-a01dr01] => (item=anvil-daemon.service) [2026-04-14T07:48:09.230Z] changed: [an-striker02] => (item=anvil-daemon.service) [2026-04-14T07:48:09.230Z] changed: [an-striker01] => (item=anvil-daemon.service) [2026-04-14T07:48:09.230Z] changed: [an-a01n02] => (item=scancore.service) [2026-04-14T07:48:09.230Z] changed: [an-a01n01] => (item=scancore.service) [2026-04-14T07:48:09.230Z] changed: [an-a01dr01] => (item=scancore.service) [2026-04-14T07:48:09.230Z] changed: [an-striker02] => (item=scancore.service) [2026-04-14T07:48:09.230Z] changed: [an-striker01] => (item=scancore.service) [2026-04-14T07:48:09.230Z] [2026-04-14T07:48:09.230Z] PLAY [delete server tests] ***************************************************** [2026-04-14T07:48:09.230Z] [2026-04-14T07:48:09.230Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:48:11.047Z] ok: [an-a01n02] [2026-04-14T07:48:11.047Z] ok: [an-a01n01] [2026-04-14T07:48:11.047Z] [2026-04-14T07:48:11.047Z] TASK [check cluster is in good state] ****************************************** [2026-04-14T07:48:11.416Z] included: /srv/anvil/anvil-test-suite/playbooks/common/cluster_good.yml for an-a01n01, an-a01n02 [2026-04-14T07:48:11.416Z] [2026-04-14T07:48:11.416Z] TASK [wait up to 10 minutes for nodes to be in pacemaker cluster] ************** [2026-04-14T07:48:15.471Z] ok: [an-a01n02] => (item=an-a01n01) [2026-04-14T07:48:15.471Z] ok: [an-a01n01] => (item=an-a01n01) [2026-04-14T07:48:15.471Z] ok: [an-a01n02] => (item=an-a01n02) [2026-04-14T07:48:15.471Z] ok: [an-a01n01] => (item=an-a01n02) [2026-04-14T07:48:15.471Z] [2026-04-14T07:48:15.471Z] TASK [delete servers] ********************************************************** [2026-04-14T07:48:15.471Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=1) [2026-04-14T07:48:15.471Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=2) [2026-04-14T07:48:15.471Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=3) [2026-04-14T07:48:15.471Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=4) [2026-04-14T07:48:15.471Z] included: /srv/anvil/anvil-test-suite/playbooks/common/delete_server.yml for an-a01n01, an-a01n02 => (item=5) [2026-04-14T07:48:15.471Z] [2026-04-14T07:48:15.471Z] TASK [check if server an-test-deploy1 exists] ********************************** [2026-04-14T07:48:17.287Z] skipping: [an-a01n01] [2026-04-14T07:48:17.287Z] ok: [an-a01n02] [2026-04-14T07:48:17.287Z] [2026-04-14T07:48:17.287Z] TASK [delete server an-test-deploy1 from an-a01n02] **************************** [2026-04-14T07:48:26.620Z] skipping: [an-a01n01] [2026-04-14T07:48:26.620Z] changed: [an-a01n02] [2026-04-14T07:48:26.620Z] [2026-04-14T07:48:26.620Z] TASK [Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker] ***** [2026-04-14T07:49:15.731Z] skipping: [an-a01n01] [2026-04-14T07:49:15.731Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (60 retries left). [2026-04-14T07:49:15.731Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (59 retries left). [2026-04-14T07:49:15.731Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (58 retries left). [2026-04-14T07:49:15.731Z] FAILED - RETRYING: [an-a01n02]: Wait up to 10 minutes for an-test-deploy1 to be delete in pacemaker (57 retries left). [2026-04-14T07:49:15.731Z] ok: [an-a01n02] [2026-04-14T07:49:15.731Z] [2026-04-14T07:49:15.731Z] TASK [Wait up to 10 minutes for an-test-deploy1 drbd storage to be removed] **** [2026-04-14T07:49:15.731Z] skipping: [an-a01n01] [2026-04-14T07:49:15.731Z] ok: [an-a01n02] [2026-04-14T07:49:15.731Z] [2026-04-14T07:49:15.731Z] TASK [Wait up to 10 minutes for an-test-deploy1 lvm volume to be removed] ****** [2026-04-14T07:49:15.731Z] skipping: [an-a01n01] [2026-04-14T07:49:15.731Z] ok: [an-a01n02] [2026-04-14T07:49:15.731Z] [2026-04-14T07:49:15.731Z] TASK [check if server an-test-deploy2 exists] ********************************** [2026-04-14T07:49:16.100Z] skipping: [an-a01n01] [2026-04-14T07:49:16.100Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy2\n", "delta": "0:00:00.998348", "end": "2026-04-14 07:49:15.748790", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 07:49:14.750442", "stderr": "Error: resource or tag id 'an-test-deploy2' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy2' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T07:49:16.100Z] ...ignoring [2026-04-14T07:49:16.100Z] [2026-04-14T07:49:16.100Z] TASK [delete server an-test-deploy2 from an-a01n02] **************************** [2026-04-14T07:49:16.100Z] skipping: [an-a01n01] [2026-04-14T07:49:16.100Z] skipping: [an-a01n02] [2026-04-14T07:49:16.100Z] [2026-04-14T07:49:16.100Z] TASK [Wait up to 10 minutes for an-test-deploy2 to be delete in pacemaker] ***** [2026-04-14T07:49:16.100Z] skipping: [an-a01n01] [2026-04-14T07:49:16.100Z] skipping: [an-a01n02] [2026-04-14T07:49:16.100Z] [2026-04-14T07:49:16.100Z] TASK [Wait up to 10 minutes for an-test-deploy2 drbd storage to be removed] **** [2026-04-14T07:49:16.100Z] skipping: [an-a01n01] [2026-04-14T07:49:16.100Z] skipping: [an-a01n02] [2026-04-14T07:49:16.100Z] [2026-04-14T07:49:16.100Z] TASK [Wait up to 10 minutes for an-test-deploy2 lvm volume to be removed] ****** [2026-04-14T07:49:16.100Z] skipping: [an-a01n01] [2026-04-14T07:49:16.100Z] skipping: [an-a01n02] [2026-04-14T07:49:16.100Z] [2026-04-14T07:49:16.100Z] TASK [check if server an-test-deploy3 exists] ********************************** [2026-04-14T07:49:17.915Z] skipping: [an-a01n01] [2026-04-14T07:49:17.915Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy3\n", "delta": "0:00:00.843706", "end": "2026-04-14 07:49:17.378608", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 07:49:16.534902", "stderr": "Error: resource or tag id 'an-test-deploy3' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy3' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T07:49:17.915Z] ...ignoring [2026-04-14T07:49:17.915Z] [2026-04-14T07:49:17.915Z] TASK [delete server an-test-deploy3 from an-a01n02] **************************** [2026-04-14T07:49:17.915Z] skipping: [an-a01n01] [2026-04-14T07:49:17.915Z] skipping: [an-a01n02] [2026-04-14T07:49:17.915Z] [2026-04-14T07:49:17.915Z] TASK [Wait up to 10 minutes for an-test-deploy3 to be delete in pacemaker] ***** [2026-04-14T07:49:17.915Z] skipping: [an-a01n01] [2026-04-14T07:49:17.915Z] skipping: [an-a01n02] [2026-04-14T07:49:17.915Z] [2026-04-14T07:49:17.915Z] TASK [Wait up to 10 minutes for an-test-deploy3 drbd storage to be removed] **** [2026-04-14T07:49:17.915Z] skipping: [an-a01n01] [2026-04-14T07:49:17.915Z] skipping: [an-a01n02] [2026-04-14T07:49:17.915Z] [2026-04-14T07:49:17.915Z] TASK [Wait up to 10 minutes for an-test-deploy3 lvm volume to be removed] ****** [2026-04-14T07:49:17.915Z] skipping: [an-a01n01] [2026-04-14T07:49:17.915Z] skipping: [an-a01n02] [2026-04-14T07:49:17.915Z] [2026-04-14T07:49:17.915Z] TASK [check if server an-test-deploy4 exists] ********************************** [2026-04-14T07:49:19.828Z] skipping: [an-a01n01] [2026-04-14T07:49:19.828Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy4\n", "delta": "0:00:00.919750", "end": "2026-04-14 07:49:19.230569", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 07:49:18.310819", "stderr": "Error: resource or tag id 'an-test-deploy4' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy4' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T07:49:19.828Z] ...ignoring [2026-04-14T07:49:19.828Z] [2026-04-14T07:49:19.828Z] TASK [delete server an-test-deploy4 from an-a01n02] **************************** [2026-04-14T07:49:19.828Z] skipping: [an-a01n01] [2026-04-14T07:49:19.828Z] skipping: [an-a01n02] [2026-04-14T07:49:19.828Z] [2026-04-14T07:49:19.828Z] TASK [Wait up to 10 minutes for an-test-deploy4 to be delete in pacemaker] ***** [2026-04-14T07:49:19.828Z] skipping: [an-a01n01] [2026-04-14T07:49:19.828Z] skipping: [an-a01n02] [2026-04-14T07:49:19.828Z] [2026-04-14T07:49:19.828Z] TASK [Wait up to 10 minutes for an-test-deploy4 drbd storage to be removed] **** [2026-04-14T07:49:19.828Z] skipping: [an-a01n01] [2026-04-14T07:49:19.828Z] skipping: [an-a01n02] [2026-04-14T07:49:19.828Z] [2026-04-14T07:49:19.828Z] TASK [Wait up to 10 minutes for an-test-deploy4 lvm volume to be removed] ****** [2026-04-14T07:49:19.828Z] skipping: [an-a01n01] [2026-04-14T07:49:19.828Z] skipping: [an-a01n02] [2026-04-14T07:49:19.828Z] [2026-04-14T07:49:19.828Z] TASK [check if server an-test-deploy5 exists] ********************************** [2026-04-14T07:49:21.646Z] skipping: [an-a01n01] [2026-04-14T07:49:21.646Z] fatal: [an-a01n02]: FAILED! => {"changed": false, "cmd": "pcs status resources an-test-deploy5\n", "delta": "0:00:01.023221", "end": "2026-04-14 07:49:21.117044", "msg": "non-zero return code", "rc": 1, "start": "2026-04-14 07:49:20.093823", "stderr": "Error: resource or tag id 'an-test-deploy5' not found", "stderr_lines": ["Error: resource or tag id 'an-test-deploy5' not found"], "stdout": "", "stdout_lines": []} [2026-04-14T07:49:21.646Z] ...ignoring [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] TASK [delete server an-test-deploy5 from an-a01n02] **************************** [2026-04-14T07:49:21.646Z] skipping: [an-a01n01] [2026-04-14T07:49:21.646Z] skipping: [an-a01n02] [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] TASK [Wait up to 10 minutes for an-test-deploy5 to be delete in pacemaker] ***** [2026-04-14T07:49:21.646Z] skipping: [an-a01n01] [2026-04-14T07:49:21.646Z] skipping: [an-a01n02] [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] TASK [Wait up to 10 minutes for an-test-deploy5 drbd storage to be removed] **** [2026-04-14T07:49:21.646Z] skipping: [an-a01n01] [2026-04-14T07:49:21.646Z] skipping: [an-a01n02] [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] TASK [Wait up to 10 minutes for an-test-deploy5 lvm volume to be removed] ****** [2026-04-14T07:49:21.646Z] skipping: [an-a01n01] [2026-04-14T07:49:21.646Z] skipping: [an-a01n02] [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] PLAY [Check anvil available resources from striker01] ************************** [2026-04-14T07:49:21.646Z] [2026-04-14T07:49:21.646Z] TASK [Gathering Facts] ********************************************************* [2026-04-14T07:49:24.097Z] ok: [an-striker01] [2026-04-14T07:49:24.097Z] [2026-04-14T07:49:24.097Z] TASK [Check anvil available resources] ***************************************** [2026-04-14T07:54:35.831Z] ok: [an-striker01] [2026-04-14T07:54:35.831Z] [2026-04-14T07:54:35.831Z] PLAY RECAP ********************************************************************* [2026-04-14T07:54:35.831Z] an-a01dr01 : ok=90 changed=38 unreachable=0 failed=0 skipped=5 rescued=0 ignored=0 [2026-04-14T07:54:35.831Z] an-a01n01 : ok=314 changed=28 unreachable=0 failed=0 skipped=109 rescued=0 ignored=5 [2026-04-14T07:54:35.831Z] an-a01n02 : ok=345 changed=27 unreachable=0 failed=0 skipped=90 rescued=0 ignored=4 [2026-04-14T07:54:35.831Z] an-striker01 : ok=57 changed=19 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-14T07:54:35.831Z] an-striker02 : ok=42 changed=12 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0 [2026-04-14T07:54:35.831Z] [2026-04-14T07:54:35.831Z] tar: Removing leading `/' from member names [2026-04-14T07:54:35.831Z] Warning: Permanently added '[anvil-ci-proxy]:8086' (ED25519) to the list of known hosts. [2026-04-14T07:54:35.831Z] /root/ci-tools [2026-04-14T07:54:35.831Z] Complete anvil logs available here for download: http://anvil-ci-repo.ci.alteeve.com/testing-logs/311.tar.gz [2026-04-14T07:54:35.926Z] [Pipeline] } [2026-04-14T07:54:35.930Z] [Pipeline] // timeout [2026-04-14T07:54:35.935Z] [Pipeline] } [2026-04-14T07:54:35.938Z] [Pipeline] // script [2026-04-14T07:54:35.941Z] [Pipeline] } [2026-04-14T07:54:35.943Z] [Pipeline] // stage [2026-04-14T07:54:35.948Z] [Pipeline] stage [2026-04-14T07:54:35.948Z] [Pipeline] { (Declarative: Post Actions) [2026-04-14T07:54:35.957Z] [Pipeline] cleanWs [2026-04-14T07:54:36.017Z] [WS-CLEANUP] Deleting project workspace... [2026-04-14T07:54:36.017Z] [WS-CLEANUP] Deferred wipeout is disabled by the job configuration... [2026-04-14T07:54:36.047Z] [WS-CLEANUP] done [2026-04-14T07:54:36.056Z] [Pipeline] script [2026-04-14T07:54:36.060Z] [Pipeline] { [2026-04-14T07:54:36.087Z] [Pipeline] stage [2026-04-14T07:54:36.087Z] [Pipeline] { (Copying logs) [2026-04-14T07:54:36.132Z] [Pipeline] node [2026-04-14T07:54:36.145Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing [2026-04-14T07:54:36.146Z] [Pipeline] { [2026-04-14T07:54:36.217Z] [Pipeline] node [2026-04-14T07:54:36.229Z] Running on Jenkins in /var/lib/jenkins/workspace/anvil/anvil-build-functional-testing@2 [2026-04-14T07:54:36.230Z] [Pipeline] { [2026-04-14T07:54:36.233Z] [Pipeline] sh [2026-04-14T07:54:36.489Z] + mkdir -p /var/lib/jenkins/locks [2026-04-14T07:54:36.500Z] [Pipeline] echo [2026-04-14T07:54:36.500Z] RWLock: FD for lock log_archive in stage projectFinishUp is 370 [2026-04-14T07:54:36.503Z] [Pipeline] sleep [2026-04-14T07:54:36.504Z] No need to sleep any longer [2026-04-14T07:54:36.507Z] [Pipeline] echo [2026-04-14T07:54:36.507Z] RWLock: log_archive in stage projectFinishUp locked for WRITE [2026-04-14T07:54:36.510Z] [Pipeline] } [2026-04-14T07:54:36.513Z] [Pipeline] // node [2026-04-14T07:54:36.518Z] [Pipeline] echo [2026-04-14T07:54:36.518Z] Copying logs to /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/311/ [2026-04-14T07:54:36.519Z] [Pipeline] sh [2026-04-14T07:54:36.776Z] + mkdir -p /var/www/ci.kronosnet.org/job/anvil/job/anvil-build-functional-testing/311/ [2026-04-14T07:54:36.778Z] [Pipeline] sh