CORS-4336: Add CI jobs for AWS European Sovereign Cloud (EUSC) by liweinan · Pull Request #75568 · openshift/release

liweinan · 2026-03-02T18:04:40Z

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

openshift-ci-robot · 2026-03-02T18:04:45Z

@liweinan: This pull request references CORS-4336 which is a valid jira issue.

Details

In response to this:

Implement continuous integration support for AWS EUSC partition (aws-eusc) in eusc-de-east-1 region. Includes cluster profile definition, service endpoints configuration, custom AMI handling, and periodic test jobs.

This enables OpenShift testing on AWS's new European Sovereign Cloud infrastructure, which requires explicit endpoint configuration and custom RHCOS AMIs not available in public regions.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci-robot · 2026-03-02T18:06:05Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[125]: invalid cluster profile "aws-eusc-qe"
 * tests[126]: invalid cluster profile "aws-eusc-qe"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml

ci-operator/step-registry/cluster-profiles/cluster-profiles-config.yaml

...ipi/private/provision/cucushift-installer-rehearse-aws-eusc-ipi-private-provision-chain.yaml

ci-operator/step-registry/ipi/conf/aws/eusc-ami/ipi-conf-aws-eusc-ami-commands.sh

yunjiang29 · 2026-03-05T02:26:01Z

@liweinan as we discussed offline, for the new partition we need three types of cluster:

common IPI cluster
private cluster
disconnected (private) cluster
and based on above basic cluster, we also need to cover STS, custom KMS key, FIPS and minimum permission, you can refer to existing jobs.

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__amd64-nightly.yaml

liweinan · 2026-03-05T02:38:24Z

@yunjiang29 Thanks for the review! I'll refactor this PR today.

openshift-ci-robot · 2026-03-05T12:18:26Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan · 2026-03-06T03:16:23Z

@yunjiang29 Thanks for the detailed review! I'll update the PR recordingly.

openshift-ci-robot · 2026-03-06T06:37:31Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

Address yunfei's review comments on PR openshift#75568: 1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. AMI configuration fix: - Replace deprecated compute.platform.aws.amiID field - Use platform.aws.defaultMachinePlatform.amiID instead

openshift-ci-robot · 2026-03-06T06:40:37Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. AMI configuration fix: - Replace deprecated compute.platform.aws.amiID field - Use platform.aws.defaultMachinePlatform.amiID instead 4. Generalize step registry components for reusability: - Enhance ipi-conf-aws-custom-endpoints to support multiple AWS partitions * Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com) * Support amazonaws.eu for EUSC, amazonaws.com.cn for China * Allow full URLs for maximum flexibility - Make ipi-conf-aws-eusc-ami more generic * Support AWS_CUSTOM_AMI_ID for general use * Maintain AWS_EUSC_AMI_ID for backward compatibility * Can be used for EUSC, China, GovCloud, or custom AMI scenarios - Use generic steps in EUSC provision chain with partition-specific config - Remove obsolete ipi-conf-aws-eusc-endpoints (replaced by generic version)

openshift-ci-robot · 2026-03-06T07:00:15Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

...erator/step-registry/ipi/conf/aws/custom-endpoints/ipi-conf-aws-custom-endpoints-commands.sh

1. Job naming convention: - Rename jobs from -f60 to -f7 suffix (non-destructive tests) - Update cron schedule to standard f7 pattern: 7,14,23,30 2. Private cluster configuration: - Add complete private cluster setup with bastion host - Add VPC, security groups, and proxy configuration - Set PUBLISH=Internal for private cluster access - Add minimal IAM permission provisioning - Follow pattern from cucushift-installer-rehearse-aws-ipi-private-provision 3. Generalize step registry components for maximum reusability: a) Enhance ipi-conf-aws-custom-endpoints for all AWS partitions: - Add AWS_DOMAIN_SUFFIX env var (defaults to amazonaws.com) - Support amazonaws.eu (EUSC), amazonaws.com.cn (China) - Allow full URLs for maximum flexibility - Remove obsolete ipi-conf-aws-eusc-endpoints step b) Extend ipi-conf-aws to support custom AMI configuration: - Add AWS_AMI_ID env var for custom RHCOS AMI - Useful for EUSC, China, GovCloud, or any partition without public AMIs - Fix deprecated amiID field -> defaultMachinePlatform.amiID - Auto-detection still works for C2S/SC2S - Remove obsolete ipi-conf-aws-eusc-ami step c) EUSC provision chain now uses only generic steps with env config This refactoring reduces code duplication (net -59 lines) and makes step components reusable across all AWS partitions.

openshift-ci-robot · 2026-03-06T07:12:44Z

@liweinan, pj-rehearse: unable to determine affected jobs. This could be due to a branch that needs to be rebased. ERROR:

could not load configuration from candidate revision of release repo: failed to load ci-operator configuration from release repo: invalid ci-operator config: configuration has 2 errors:

 * tests[126]: invalid cluster profile "aws-eusc"
 * tests[127]: invalid cluster profile "aws-eusc"

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

liweinan · 2026-03-10T13:59:36Z

Relative PRs merged: #75441 / openshift/ci-tools#4973

ci-operator/step-registry/ipi/conf/aws/ipi-conf-aws-commands.sh

yunjiang29 · 2026-03-20T06:43:36Z

...t/openshift-tests-private/openshift-openshift-tests-private-release-4.22__multi-nightly.yaml

    test:
    - chain: openshift-e2e-test-qe
    workflow: baremetal-lab-upi
+- as: aws-eusc-ipi-byo-kms-etcd-encryption-fips-tp-arm-f7


@liweinan this part looks good, please copy these configs to openshift-openshift-tests-private-release-4.23__multi-nightly.yaml and openshift-openshift-tests-private-release-5.0__multi-nightly.yaml as well

patrickdillon · 2026-03-20T17:59:57Z

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

/pj-rehearse pull-ci-openshift-installer-main-e2e-aws-eusc-techpreview

openshift-ci-robot · 2026-03-20T17:59:59Z

@patrickdillon: now processing your pj-rehearse request. Please allow up to 10 minutes for jobs to trigger or cancel.

tthvo · 2026-03-20T19:14:41Z

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

I guess the credentials are not yet ready? The installer is asking for the access key and secrets in the ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview.

level=warning msg=Found override for release image (registry.build10.ci.openshift.org/ci-op-n8lpz8gb/release@sha256:02c954c753c39c185094a8253214485c9dcd4aa17202bf680bb158d06db15032). Release Image Architecture is unknown
? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25hlevel=error msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: [platform.aws.region: Internal error: failed to get list of regions: failed to create EC2 client: EOF, controlPlane.platform.aws: Internal error: failed to create EC2 client: EOF, compute[0].platform.aws: Internal error: failed to create EC2 client: EOF]
Create manifests exit code: 3

liweinan · 2026-03-21T16:08:20Z

I did a cluster bot build with the installer PR included:

https://prow.ci.openshift.org/view/gs/test-platform-results/logs/release-openshift-origin-installer-launch-aws-modern/2035274299501187072

vagrant@10:~/works$ ./openshift-install version
./openshift-install 4.22.0-0.nightly-2026-03-21-034605
built from commit 2958dba95307cf75e6c5fca1e99b247383625789
release image registry.ci.openshift.org/ocp/release@sha256:219df7aaa9001236a6cfe37d404fba75bf5a65157dbe1f1fd4385d677796ae95
release architecture amd64

I'll use it for local testing firstly.

liweinan · 2026-03-21T16:14:57Z

@tthvo's initial EUSC PR has merged, so we might actually be able to rehearse this, if credentials are setup:

I guess the credentials are not yet ready? The installer is asking for the access key and secrets in the ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview.

level=warning msg=Found override for release image (registry.build10.ci.openshift.org/ci-op-n8lpz8gb/release@sha256:02c954c753c39c185094a8253214485c9dcd4aa17202bf680bb158d06db15032). Release Image Architecture is unknown
? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25h? AWS Access Key ID [? for help] �[?25l�[?25l�7�8�[?25h�[?25hlevel=error msg=failed to fetch Master Machines: failed to load asset "Install Config": failed to create install config: [platform.aws.region: Internal error: failed to get list of regions: failed to create EC2 client: EOF, controlPlane.platform.aws: Internal error: failed to create EC2 client: EOF, compute[0].platform.aws: Internal error: failed to create EC2 client: EOF]
Create manifests exit code: 3

@yunjiang29 Do you have any idea on this?

liweinan · 2026-03-21T16:21:28Z

@tthvo @patrickdillon btw, this PR is not ready for testing because it hasn't configured AMI_ID yet.

And needs IMAGE override:

vagrant@10:~/works$   export OPENSHIFT_INSTALL_RELEASE_IMAGE_OVERRIDE=registry.build11.ci.openshift.org/ci-ln-wc39d1t/release:latest

Implement comprehensive CI infrastructure for AWS EUSC partition in eusc-de-east-1 region. Job coverage (9 jobs): - Common IPI: aws-eusc-ipi-f7, aws-eusc-ipi-f28-destructive, aws-eusc-ipi-fips-f7 - Private: aws-eusc-ipi-private-f7, aws-eusc-ipi-private-f28-destructive, aws-eusc-ipi-private-fips-f7 - Disconnected: aws-eusc-ipi-disconnected-private-f7 - STS: aws-eusc-ipi-sts-f7 - KMS: aws-eusc-ipi-byo-kms-f7 Key features: - Dynamic service endpoint auto-detection from AWS API - Split AMI variables (CONTROL_PLANE_AMI, COMPUTE_AMI) for flexible configuration - Complete private cluster deprovision cleanup (bastion, security groups, stacks, S3) - Support for FIPS-enabled clusters - Disconnected (air-gapped) private cluster support - STS (Security Token Service) authentication with OIDC - Custom KMS key encryption for etcd - Both non-destructive (f7) and destructive (f28) test variants Technical implementation: - Cluster profile: aws-eusc with automatic region detection - Custom RHCOS AMI support for control plane and compute nodes separately - Endpoint auto-detection from AWS API (no hardcoded values) - Manual credentials mode for CCO - Minimal IAM permissions - Mirror registry for disconnected environments - Backward compatible with existing AWS partitions Workflows created: - cucushift-installer-rehearse-aws-eusc-ipi (common IPI) - cucushift-installer-rehearse-aws-eusc-ipi-private (private cluster) - cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private (disconnected) - cucushift-installer-rehearse-aws-eusc-ipi-sts (STS authentication) - cucushift-installer-rehearse-aws-eusc-ipi-byo-kms (custom KMS key) Signed-off-by: Wei Li <weli@redhat.com>

@yunjiang29

This refactors the AWS European Sovereign Cloud (EUSC) CI configuration to maximize reuse of standard AWS workflows and reduce maintenance burden. Changes based on @yunjiang29's review feedback: - Reduced from 9 to 6 jobs following the pattern: 3 cluster types × 2 test types - Improved FIPS coverage from 1/9 (11%) to 2/6 (33%) jobs: * aws-eusc-ipi-fips-f7 (IPI + FIPS) * aws-eusc-ipi-private-sts-fips-f7 (Private + STS + FIPS) - Combined features across jobs: * aws-eusc-ipi-f28-destructive (destructive testing) * aws-eusc-ipi-private-mini-perm-f28 (Private + minimal permissions) * aws-eusc-ipi-disc-priv-kms-f7 (Disconnected + KMS) * aws-eusc-ipi-disc-priv-f28 (Disconnected destructive) - All jobs cover: FIPS, STS, KMS, minimal permissions across 3 cluster types - Deleted 15 EUSC-specific files, created 8 new ones (net reduction: -7 files) - Maximized reuse of standard AWS workflows: * Basic IPI: reuses cucushift-installer-rehearse-aws-ipi-deprovision * Private: reuses cucushift-installer-rehearse-aws-ipi-private-deprovision * Disconnected: reuses cucushift-installer-rehearse-aws-ipi-disconnected-private-provision * Private-STS: reuses cucushift-installer-rehearse-aws-ipi-private-cco-manual-security-token-service - EUSC-specific changes limited to: * Inserting ipi-conf-aws-custom-endpoints ref for service endpoint configuration * Custom provision chain for disconnected-private-kms (combines disconnected + KMS) - Deleted all EUSC-specific deprovision chains (reuse standard chains) - Removed unnecessary byo-kms and STS specific directory structures 1. Custom endpoints (ipi-conf-aws-custom-endpoints-commands.sh): - Removed auto-detection logic for AWS_DOMAIN_SUFFIX - Simplified to use environment variable or default to "amazonaws.com" - Removed Route53 endpoint configuration (global service) - Designed for easy removal when installer adds native EUSC support 2. AMI configuration (ipi-conf-aws-commands.sh): - Simplified from split variables (CONTROL_PLANE_AMI/COMPUTE_AMI) to single CONTROL_PLANE_AMI - Preserved C2S/SC2S auto-detection logic - Removed complex heredoc patching, kept simple approach - Updated documentation for clarity 1. **Minimize EUSC-specific code**: Only 8 workflow files vs 15 previously 2. **Maximize standard workflow reuse**: Follows USGov pattern, not C2S pattern 3. **Prepare for future evolution**: Custom endpoints easy to remove when installer supports EUSC natively 4. **FIPS coverage aligned with USGov**: 33% vs USGov's 18%, not C2S's 100% - make update completed successfully - All 6 jobs generated in ci-operator/jobs/.../periodics.yaml - Step registry validation passed Addresses: openshift#75568

Address review feedback to simplify configuration scripts: - Remove AWS_DOMAIN_SUFFIX from step parameters (use cluster profile) - Support separate CONTROL_PLANE_AMI and COMPUTE_AMI configuration - Replace yq-go with yq v4 for YAML manipulation - Eliminate unnecessary fallback logic, rely on correct parameter passing - Remove intermediate variables (RHCOS_AMI) in C2S auto-detection These changes follow existing script patterns and maintain compatibility with C2S/SC2S auto-detection while enabling flexible AMI configuration for partitions like EUSC.

Fixes CI check error that enforces OWNERS files for all component configuration directories.

- Update BASE_DOMAIN from qe.devcluster.openshift.com to ci-eusc.devcluster.openshift.com for all AWS EUSC CI jobs to use the dedicated delegated subdomain for CI/QE account - Add 8 multi-arch EUSC CI jobs in openshift-tests-private release-4.22 multi-nightly: * BYO KMS encryption with FIPS (ARM f7, AMD f28-destructive) * Disconnected private (ARM f7, AMD f28-destructive) * Private STS (ARM f7, AMD f28-destructive) * Custom DNS with minimal permissions (ARM f7, AMD f28-destructive) - Add e2e-aws-eusc-techpreview jobs to openshift/installer configs: * release-4.22, release-4.23, release-5.0, and main - Add installer repo to aws-eusc cluster profile owners - Restore version info comments in ipi-conf-aws-commands.sh All jobs use cluster_profile: aws-eusc with BASE_DOMAIN: ci-eusc.devcluster.openshift.com and FEATURE_SET: TechPreviewNoUpgrade.

The installer now configures service endpoints implicitly for EUSC partition, so manual endpoint configuration via ipi-conf-aws-custom-endpoints is no longer needed. Changes: - Remove ipi-conf-aws-custom-endpoints from all 5 EUSC workflow files - Update documentation to reflect implicit endpoint configuration - Simplify workflow by relying on installer's built-in EUSC support This addresses review feedback from yunjiang29 that the installer handles endpoints automatically for special AWS partitions like EUSC.

Update generated Prow job configurations after rebasing to the latest origin/main. Changes include: - Updated cluster assignments to match current build cluster distribution - EUSC jobs properly integrated with latest job generation logic

Delete 4 EUSC-specific workflows and 2 provision chains, replacing them with standard AWS workflows. This reduces maintenance burden and ensures consistency with standard AWS job configurations. Changes: - Delete cucushift-installer-rehearse-aws-eusc-ipi workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-private workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-private-sts workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private workflow - Delete cucushift-installer-rehearse-aws-eusc-ipi provision chain - Delete cucushift-installer-rehearse-aws-eusc-ipi-private provision chain Modified 5 jobs to use standard AWS workflows: - aws-eusc-ipi-fips-f7 → cucushift-installer-rehearse-aws-ipi - aws-eusc-ipi-f28-destructive → cucushift-installer-rehearse-aws-ipi - aws-eusc-ipi-private-sts-fips-f7 → aws-ipi-private-cco-manual-security-token-service - aws-eusc-ipi-private-mini-perm-f28 → cucushift-installer-rehearse-aws-ipi-private - aws-eusc-ipi-disc-priv-f28 → cucushift-installer-rehearse-aws-ipi-disconnected-private All modified jobs now include: - cluster_profile: aws-eusc (handles region and AMI configuration) - COMPUTE_NODE_TYPE: m5.xlarge - CONTROL_PLANE_INSTANCE_TYPE: m6i.xlarge Preserved for further discussion: - cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms (unique combination not available in standard AWS workflows) Result: -300 lines, 100% workflow reuse for modified jobs

- Delete last EUSC-specific workflow: cucushift-installer-rehearse-aws-eusc-ipi-disconnected-private-kms - Delete associated provision chain - Update aws-eusc-ipi-disc-priv-kms-f7 job to use standard cucushift-installer-rehearse-aws-ipi-disconnected-private workflow - Add COMPUTE_NODE_TYPE and CONTROL_PLANE_INSTANCE_TYPE env vars to the job All EUSC jobs now use standard AWS workflows with cluster_profile: aws-eusc. This completes the refactoring based on review feedback.

Two critical bug fixes in ipi-conf-aws-commands.sh: 1. Fix CONTROL_PLANE_AMI being unconditionally overwritten - Before: Always fetched from GitHub in C2S/SC2S environments - After: Only auto-detect if user hasn't provided CONTROL_PLANE_AMI - Impact: Users can now override AMI for control plane nodes 2. Fix COMPUTE_AMI being unconditionally overwritten - Before: COMPUTE_AMI="${CONTROL_PLANE_AMI}" (always overwrites) - After: COMPUTE_AMI="${COMPUTE_AMI:-${CONTROL_PLANE_AMI}}" (respects user value) - Impact: Users can now specify different AMIs for compute nodes Both fixes are 100% backward compatible with existing jobs. All current C2S/SC2S jobs don't set these env vars, so behavior unchanged.

Changes per yunjiang29's review comments: 1. Remove all 6 EUSC jobs from amd64-nightly.yaml - All EUSC jobs now run against multi-nightly payload only - ARM for non-destructive (f7), AMD for destructive (f28) 2. Fix ipi-conf-aws-commands.sh for C2S/SC2S: - Restore version info comment: "# custom rhcos ami for non-public regions" - Restore inline comments: "# 4.9 and below" and "# 4.10 and above" - Add COMPUTE_AMI and echo in C2S block - Remove unreasonable default COMPUTE_AMI logic outside C2S block 3. Fix multi-nightly.yaml jobs: a) Rename KMS job to include "etcd" and meet 61-char limit: aws-eusc-ipi-byo-kms-encryption-fips-tp-amd-f28-destructive → aws-eusc-ipi-byo-kms-etcd-encryption-fips-tp-f28-destructive b) Fix KMS config for destructive job: ENABLE_AWS_KMS_KEY_COMPUTE/CONTROL_PLANE: yes → no ENABLE_AWS_KMS_KEY_DEFAULT_MACHINE: no → yes c) Add -mini-perm to STS job names (they use AWS_INSTALL_USE_MINIMAL_PERMISSIONS): aws-eusc-ipi-private-sts-tp-arm-f7 → aws-eusc-ipi-private-sts-mini-perm-tp-arm-f7 aws-eusc-ipi-private-sts-tp-amd-f28-destructive → aws-eusc-ipi-private-sts-mini-perm-tp-amd-f28-destructive Result: - 8 EUSC jobs in multi-nightly (4 ARM f7 + 4 AMD f28-destructive) - 4 installer presubmit jobs (unchanged) - 0 EUSC jobs in amd64-nightly - Total: 12 EUSC jobs (down from 18)

Fix the AMI configuration condition to check both CONTROL_PLANE_AMI and COMPUTE_AMI are empty before auto-fetching RHCOS AMIs for C2S/SC2S regions. Add AWS EUSC CI jobs to release-4.23 and release-5.0 multi-nightly configs. Regenerate jobs after rebase to latest main

openshift-ci-robot · 2026-03-21T17:16:04Z

[REHEARSALNOTIFIER]
@liweinan: the pj-rehearse plugin accommodates running rehearsal tests for the changes in this PR. Expand 'Interacting with pj-rehearse' for usage details. The following rehearsable tests have been affected by this change:

Test name	Repo	Type	Reason
pull-ci-openshift-cluster-kube-scheduler-operator-main-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-main-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-main-e2e-aws-operator-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-main-e2e-aws-operator-parallel-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-main-e2e-aws-preferred-host-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-5.0-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-5.0-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-5.0-e2e-aws-operator-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-5.0-e2e-aws-operator-parallel-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-5.0-e2e-aws-preferred-host-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.23-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.23-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.23-e2e-aws-operator-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.23-e2e-aws-operator-parallel-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.23-e2e-aws-preferred-host-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.22-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.22-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.22-e2e-aws-operator-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.22-e2e-aws-operator-parallel-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.22-e2e-aws-preferred-host-serial-ote	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.21-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.21-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.20-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.20-e2e-aws-operator-preferred-host	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed
pull-ci-openshift-cluster-kube-scheduler-operator-release-4.19-e2e-aws-operator	openshift/cluster-kube-scheduler-operator	presubmit	Registry content changed

A total of 16687 jobs have been affected by this change. The above listing is non-exhaustive and limited to 25 jobs.

A full list of affected jobs can be found here

Interacting with pj-rehearse

Comment: /pj-rehearse to run up to 5 rehearsals
Comment: /pj-rehearse skip to opt-out of rehearsals
Comment: /pj-rehearse {test-name}, with each test separated by a space, to run one or more specific rehearsals
Comment: /pj-rehearse more to run up to 10 rehearsals
Comment: /pj-rehearse max to run up to 25 rehearsals
Comment: /pj-rehearse auto-ack to run up to 5 rehearsals, and add the rehearsals-ack label on success
Comment: /pj-rehearse list to get an up-to-date list of affected jobs
Comment: /pj-rehearse abort to abort all active rehearsals
Comment: /pj-rehearse network-access-allowed to allow rehearsals of tests that have the restrict_network_access field set to false. This must be executed by an openshift org member who is not the PR author

Once you are satisfied with the results of the rehearsals, comment: /pj-rehearse ack to unblock merge. When the rehearsals-ack label is present on your PR, merge will no longer be blocked by rehearsals.
If you would like the rehearsals-ack label removed, comment: /pj-rehearse reject to re-block merging.

openshift-ci · 2026-03-21T17:39:19Z

@liweinan: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/rehearse/openshift/installer/main/e2e-aws-eusc-techpreview	`a61aea4`	link	unknown	`/pj-rehearse pull-ci-openshift-installer-main-e2e-aws-eusc-techpreview`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

liweinan · 2026-03-21T17:54:46Z

this existing AMI's volumeSize is just 2GB which causes installation failed:

vagrant@10:~/works$ aws ec2 describe-images \
  --region eusc-de-east-1 \
  --image-ids ami-0c5e051fb7dc39f8d \
  --query 'Images[0].BlockDeviceMappings[0].Ebs' \
  --output json
{
    "DeleteOnTermination": true,
    "SnapshotId": "snap-0262beb125ed7dbec",
    "VolumeSize": 2,
    "VolumeType": "gp2",
    "Encrypted": false
}

I created a new AMI for using:

vagrant@10:~/works$ aws ec2 describe-images   --region eusc-de-east-1   --image-ids ami-0b78302f83217d149   --query 'Images[0].BlockDeviceMappings[0].Ebs'   --output json
{
    "DeleteOnTermination": true,
    "SnapshotId": "snap-03f7b6922d584d3c9",
    "VolumeSize": 120,
    "VolumeType": "gp3",
    "Encrypted": false
}

liweinan · 2026-03-21T20:21:31Z

install config:

apiVersion: v1
baseDomain: ci-eusc.devcluster.openshift.com
metadata:
  name: weli-eusc
platform:
  aws:
    amiID: ami-0b78302f83217d149
    region: eusc-de-east-1
pullSecret: xxx

install succeed:

vagrant@10:~/works$ ./openshift-install create cluster
INFO ipFamily is not specified in install-config; defaulting to "IPv4"
INFO Adding default service endpoints for region eusc-de-east-1
WARNING Found override for release image (registry.build11.ci.openshift.org/ci-ln-wc39d1t/release:latest). Release Image Architecture is unknown
INFO Credentials loaded from the AWS config using "SharedConfigCredentials: /home/vagrant/.aws/credentials" provider
INFO elbv2 endpoint is empty, using elb endpoint: https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu
INFO Successfully populated MCS CA cert information: root-ca 2036-03-18T19:38:17Z 2026-03-21T19:38:17Z
INFO Successfully populated MCS TLS cert information: root-ca 2036-03-18T19:38:17Z 2026-03-21T19:38:17Z
WARNING Found override for release image (registry.build11.ci.openshift.org/ci-ln-wc39d1t/release:latest). Please be warned, this is not advised
INFO Consuming Install Config from target directory
INFO Adding clusters...
INFO Creating infrastructure resources...
INFO Reconciling IAM roles for control-plane and compute nodes
INFO Creating IAM role for master
INFO Creating IAM role for worker
INFO Started local control plane with envtest
INFO Stored kubeconfig for envtest in: /home/vagrant/works/.clusterapi_output/envtest.kubeconfig
INFO Running process: Cluster API with args [-v=2 --diagnostics-address=0 --health-addr=127.0.0.1:44881 --webhook-port=38945 --webhook-cert-dir=/tmp/envtest-serving-certs-2930572155 --kubeconfig=/home/vagrant/works/.clusterapi_output/envtest.kubeconfig]
INFO Running process: aws infrastructure provider with args [-v=4 --diagnostics-address=0 --health-addr=127.0.0.1:40961 --webhook-port=35715 --webhook-cert-dir=/tmp/envtest-serving-certs-2698599822 --feature-gates=BootstrapFormatIgnition=true,ExternalResourceGC=true,TagUnmanagedNetworkResources=false,EKS=false,MachinePool=false --service-endpoints=eusc-de-east-1:ec2=https://ec2.eusc-de-east-1.amazonaws.eu,elasticloadbalancing=https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu,s3=https://s3.eusc-de-east-1.amazonaws.eu,route53=https://route53.amazonaws.eu,iam=https://iam.eusc-de-east-1.amazonaws.eu,sts=https://sts.eusc-de-east-1.amazonaws.eu,tagging=https://tagging.eusc-de-east-1.amazonaws.eu --kubeconfig=/home/vagrant/works/.clusterapi_output/envtest.kubeconfig]
INFO Creating infra manifests...
INFO Created manifest *v1.Namespace, namespace= name=openshift-cluster-api-guests
INFO Created manifest *v1beta2.AWSClusterControllerIdentity, namespace= name=default
I0321 19:38:55.081540   43988 warning_handler.go:65] "cluster.x-k8s.io/v1beta1 Cluster is deprecated; use cluster.x-k8s.io/v1beta2 Cluster" logger="KubeAPIWarningLogger"
INFO Created manifest *v1beta1.Cluster, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv
INFO Created manifest *v1beta2.AWSCluster, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv
INFO Done creating infra manifests
INFO Creating kubeconfig entry for capi cluster weli-eusc-s92nv
INFO Waiting up to 15m0s (until 7:53PM UTC) for network infrastructure to become ready...
INFO Network infrastructure is ready
INFO Creating Route53 records for control plane load balancer
INFO Created private Hosted Zone
INFO Created manifest *v1beta2.AWSMachine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-bootstrap
INFO Created manifest *v1beta2.AWSMachine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-0
INFO Created manifest *v1beta2.AWSMachine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-1
INFO Created manifest *v1beta2.AWSMachine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-2
I0321 19:45:03.631181   43988 warning_handler.go:65] "cluster.x-k8s.io/v1beta1 Machine is deprecated; use cluster.x-k8s.io/v1beta2 Machine" logger="KubeAPIWarningLogger"
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-bootstrap
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-0
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-1
INFO Created manifest *v1beta1.Machine, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master-2
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-bootstrap
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-master
INFO Created manifest *v1.Secret, namespace=openshift-cluster-api-guests name=weli-eusc-s92nv-worker
INFO Waiting up to 15m0s (until 8:00PM UTC) for machines [weli-eusc-s92nv-bootstrap weli-eusc-s92nv-master-0 weli-eusc-s92nv-master-1 weli-eusc-s92nv-master-2] to provision...
INFO Control-plane machines are ready
INFO Cluster API resources have been created. Waiting for cluster to become ready...
INFO Waiting up to 20m0s (until 8:05PM UTC) for the Kubernetes API at https://api.weli-eusc.ci-eusc.devcluster.openshift.com:6443...
INFO API v1.35.2 up
INFO Waiting up to 45m0s (until 8:44PM UTC) for bootstrapping to complete...
INFO Waiting for the bootstrap etcd member to be removed...
INFO Bootstrap etcd member has been removed
INFO Destroying the bootstrap resources...
INFO Waiting up to 5m0s for bootstrap machine deletion openshift-cluster-api-guests/weli-eusc-s92nv-bootstrap...
INFO Shutting down local Cluster API controllers...
INFO Stopped controller: Cluster API
INFO Stopped controller: aws infrastructure provider
INFO Shutting down local Cluster API control plane...
INFO Local Cluster API system has completed operations
INFO Finished destroying bootstrap resources
INFO Waiting up to 40m0s (until 8:47PM UTC) for the cluster at https://api.weli-eusc.ci-eusc.devcluster.openshift.com:6443 to initialize...
INFO Waiting up to 30m0s (until 8:50PM UTC) to ensure each cluster operator has finished progressing...
INFO All cluster operators have completed progressing
INFO Checking to see if there is a route at openshift-console/console...
INFO Install complete!
INFO To access the cluster as the system:admin user when using 'oc', run
INFO     export KUBECONFIG=/home/vagrant/works/auth/kubeconfig
INFO Access the OpenShift web-console here: https://console-openshift-console.apps.weli-eusc.ci-eusc.devcluster.openshift.com
INFO Login to the console with user: "kubeadmin", and password: "xxx"
INFO Time elapsed: 42m49s

liweinan · 2026-03-21T20:31:12Z

PR #1360 Verification Report - EUSC Ingress Operator Support

Test Summary

Status: ✅ PASSED
Date: 2026-03-21
Cluster Version: 4.22.0-0-2026-03-21-085233-test-ci-ln-wc39d1t-latest
EUSC Region: eusc-de-east-1
Cluster Name: weli-eusc

Test Environment

Platform: AWS European Sovereign Cloud (EUSC)
Region: eusc-de-east-1
Base Domain: ci-eusc.devcluster.openshift.com
Cluster Domain: apps.weli-eusc.ci-eusc.devcluster.openshift.com
Installation Method: IPI (Installer Provisioned Infrastructure)
OpenShift Version: 4.22.0 (nightly build from 2026-03-21)

Verification Results

✅ 1. Ingress Operator Status

Operator Pod: Running (2/2 containers)

NAME                                READY   STATUS    RESTARTS      AGE
ingress-operator-569c494dd4-67dlc   2/2     Running   5 (16m ago)   29m

Version:

versions:
- name: operator
  version: 4.22.0-0-2026-03-21-085233-test-ci-ln-wc39d1t-latest

✅ 2. EUSC Custom Endpoints Configuration

Key Finding: All EUSC-specific custom endpoints are correctly configured.

From Ingress Operator logs:

2026-03-21T20:08:00.644Z INFO operator.dns dns/controller.go:669
  using region from operator config {"region name": "eusc-de-east-1"}

2026-03-21T20:08:00.644Z INFO operator.dns dns/controller.go:669
  Found elb custom endpoint {"url": "https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu"}

2026-03-21T20:08:00.644Z INFO operator.dns dns/controller.go:669
  Found elb v2 custom endpoint {"url": "https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu"}

2026-03-21T20:08:00.644Z INFO operator.dns dns/controller.go:669
  Found route53 custom endpoint {"url": "https://route53.amazonaws.eu"}

2026-03-21T20:08:00.644Z INFO operator.dns dns/controller.go:669
  Found resourcegroupstaggingapi custom endpoint {"url": "https://tagging.eusc-de-east-1.amazonaws.eu"}

Verified Configuration:

Service	Expected Region	Actual Region	Status
Route53	eusc-de-east-1	eusc-de-east-1	✅ PASS
Tagging API	eusc-de-east-1	eusc-de-east-1	✅ PASS
ELB	eusc-de-east-1	eusc-de-east-1	✅ PASS
ELB v2	eusc-de-east-1	eusc-de-east-1	✅ PASS

Critical:

✅ Using eusc-de-east-1 region (NOT us-east-1)
✅ Using .amazonaws.eu domain (NOT .amazonaws.com)
✅ ELB and ELB v2 share the same custom endpoint

✅ 3. Network Load Balancer (NLB) Configuration

NLB Hostname:

a2028d926bfdf4d1c86ac8a24131ffc3-2020146581.eusc-de-east-1.elb.amazonaws.eu

Verification:

✅ Hostname contains eusc-de-east-1
✅ Uses .amazonaws.eu domain (EUSC-specific)
✅ NLB successfully provisioned and active

Comparison with Standard AWS:

Standard AWS: *.elb.us-east-1.amazonaws.com
EUSC: *.elb.eusc-de-east-1.amazonaws.eu ✅

✅ 4. IngressController Status

status:
  domain: apps.weli-eusc.ci-eusc.devcluster.openshift.com
  endpointPublishingStrategy:
    type: LoadBalancerService
  conditions:
  - type: Available
    status: "True"
  - type: Progressing
    status: "False"
  - type: Degraded
    status: "False"
  - type: LoadBalancerReady
    status: "True"
    message: The LoadBalancer service is provisioned
  - type: DNSReady
    status: "True"
    message: The record is provisioned in all reported zones

Router Pods:

NAME                              READY   STATUS    RESTARTS   AGE
router-default-6f9ddc44ff-m6ktv   1/1     Running   1          27m
router-default-6f9ddc44ff-p2k2h   1/1     Running   2          27m

✅ 5. Route Functionality Test

Test Route: OpenShift Web Console

URL: https://console-openshift-console.apps.weli-eusc.ci-eusc.devcluster.openshift.com
HTTP Status: 200 OK ✅
Result: Route successfully accessible

Verification Points:

✅ DNS resolution working
✅ NLB routing working
✅ HTTPS certificate working
✅ Application responding correctly

✅ 6. EUSC-Specific Configuration Verification

Route53 Region Configuration

Expected: eusc-de-east-1
Actual: eusc-de-east-1 ✅
Evidence: Operator logs show using region from operator config {"region name": "eusc-de-east-1"}

Tagging API Region Configuration

Expected: eusc-de-east-1
Actual: eusc-de-east-1 ✅
Evidence: Created tags client {"endpoint": "https://tagging.eusc-de-east-1.amazonaws.eu"}

ELB Endpoint Sharing

ELB: https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu
ELB v2: https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu
Status: ✅ Both using the same endpoint (as designed in PR Add a fast forward script for release branches #1360)

Verification Checklist

Basic Functionality

Ingress Operator pods running normally
Ingress Operator logs show EUSC custom endpoint configuration
IngressController resource status normal
Router pods running normally

NLB Configuration

NLB successfully created
NLB hostname uses .amazonaws.eu domain
Target Groups configured correctly
NLB in active state

DNS Configuration

Route53 has wildcard DNS record for *.apps.weli-eusc.ci-eusc.devcluster.openshift.com
DNS record correctly points to NLB
DNS resolution working

Ingress Functionality

Test route accessible (Console route: HTTP 200)
HTTPS certificate working
Routing functionality working

EUSC-Specific Configuration

Route53 using eusc-de-east-1 region (NOT us-east-1)
Tagging API using eusc-de-east-1 region (NOT us-east-1)
ELB and ELB v2 share same custom endpoint
All endpoints using .amazonaws.eu domain

Issues Found

None - No critical issues identified.

Note: Operator logs contain "unable to determine partition from region" message, which is expected due to AWS SDK v1 limitations. This does not affect functionality as custom endpoints are explicitly configured.

Conclusions

✅ PR #1360 Verification: PASSED

Successfully Verified:

✅ Ingress Operator correctly identifies EUSC region
✅ All AWS API clients use correct EUSC custom endpoints
✅ Route53 uses eusc-de-east-1 region (NOT us-east-1)
✅ Tagging API uses eusc-de-east-1 region (NOT us-east-1)
✅ NLB uses correct EUSC domain (.amazonaws.eu)
✅ DNS and routing functionality fully operational

Tag Domain Name Issue: NOT Present

The "tag domain name" issue mentioned in documentation does not occur:

✅ Route53 correctly uses eusc-de-east-1 region
✅ Tagging API correctly uses eusc-de-east-1 region
✅ No incorrect usage of us-east-1 region detected

Release Confirmation

The release image used (4.22.0-0-2026-03-21-085233-test-ci-ln-wc39d1t-latest) includes PR #1360 fixes and works correctly in EUSC environment.

Recommendation

✅ PR #1360 is ready for merge

All EUSC-specific functionality has been verified and is working as designed. The Ingress Operator correctly handles:

EUSC region identification
Custom endpoint configuration for ELB, Route53, and Tagging API
NLB provisioning with EUSC-specific domain
DNS record management in EUSC Route53

Test Artifacts

Cluster Installation Time: ~43 minutes
Test Date: 2026-03-21
Tester: OpenShift QE Team
Installation Method: IPI with custom RHCOS AMI (120GB root volume)
Network Configuration: Public cluster with external DNS

Related PRs

CORS-4336: Support for AWS European Sovereign Cloud installer#10303 - Installer EUSC support
CORS-4337: allow AWS Europe Sovereign Cloud partition api#2708 - API definition updates
CORS-4335: Add support for AWS European Sovereign Cloud cluster-ingress-operator#1360 - Ingress Operator EUSC support (this PR)

Report Version: 1.0
Generated: 2026-03-21 20:30 UTC

liweinan · 2026-03-21T20:37:54Z

cluster destroyed:

vagrant@10:~/works$ ./openshift-install destroy cluster
INFO Credentials loaded from the AWS config using "SharedConfigCredentials: /home/vagrant/.aws/credentials" provider
INFO elbv2 endpoint is empty, using elb endpoint: https://elasticloadbalancing.eusc-de-east-1.amazonaws.eu
INFO Terminated                                    instance=i-0027fb4f664a14756
INFO Deleted                                       id=net/weli-eusc-s92nv-int/e62ff1a531385316/48eb2b4b76fb6b5b resourceType=listener
INFO Deleted                                       id=apiserver-target-dvsq7/fc5408d474c9cb57 resourceType=targetgroup
INFO Deleted                                       id=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln policy=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln-policy
INFO Deleted                                       id=weli-eusc-s92nv-cloud-credential-operator-iam-ro-28fln
INFO Disassociated                                 id=weli-eusc-s92nv-worker-profile name=weli-eusc-s92nv-worker-profile role=weli-eusc-s92nv-worker-role
INFO Deleted                                       InstanceProfileName=weli-eusc-s92nv-worker-profile arn=arn:aws-eusc:iam::082250599274:instance-profile/weli-eusc-s92nv-worker-profile id=weli-eusc-s92nv-worker-profile
INFO Deleted                                       id=weli-eusc-s92nv-openshift-machine-api-aws-7fg92 policy=weli-eusc-s92nv-openshift-machine-api-aws-7fg92-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-machine-api-aws-7fg92
INFO Deleted                                       id=net/weli-eusc-s92nv-int/e62ff1a531385316 resourceType=loadbalancer
INFO Deleted                                       id=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw policy=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-cloud-network-config-contro-l85qw
INFO Deleted                                       id=net/weli-eusc-s92nv-ext/e23a4a340de1c8b0 resourceType=loadbalancer
INFO Not found or already deleted                  id=net/weli-eusc-s92nv-ext/e23a4a340de1c8b0/d467a0b1f4d6fd63 resourceType=listener
INFO Deleted                                       id=nat-0ab41a56f2a9286a9 resourceType=natgateway
INFO Deleted                                       id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role policy=weli-eusc-s92nv-master-policy
INFO Disassociated                                 id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-profile role=weli-eusc-s92nv-master-role
INFO Deleted                                       InstanceProfileName=weli-eusc-s92nv-master-profile arn=arn:aws-eusc:iam::082250599274:instance-profile/weli-eusc-s92nv-master-profile id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role
INFO Deleted                                       id=weli-eusc-s92nv-master-role name=weli-eusc-s92nv-master-role
INFO Deleted                                       id=weli-eusc-s92nv-openshift-image-registry-bbqng policy=weli-eusc-s92nv-openshift-image-registry-bbqng-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-image-registry-bbqng
INFO Disassociated                                 id=rtbassoc-09d02f92b6d4172e7 resourceType=route-table
INFO Deleted                                       id=rtb-0fcb44fa352346c64 resourceType=route-table
INFO Deleted                                       id=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l policy=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l-policy
INFO Deleted                                       id=weli-eusc-s92nv-aws-ebs-csi-driver-operator-wfj2l
INFO Disassociated                                 id=rtbassoc-0d9990447edbb515d resourceType=route-table
INFO Deleted                                       id=rtb-029c5de9d3be5cd24 resourceType=route-table
INFO Not found or already deleted                  id=net/weli-eusc-s92nv-int/e62ff1a531385316/a936d3a16b432d3e resourceType=listener
INFO Deleted                                       id=sg-0b3c4db8b7c06fa4d resourceType=security-group
INFO Deleted                                       id=apiserver-target-r9flz/417b4a988db3dce4 resourceType=targetgroup
INFO Deleted                                       id=weli-eusc-s92nv-openshift-ingress-vbxnn policy=weli-eusc-s92nv-openshift-ingress-vbxnn-policy
INFO Deleted                                       id=weli-eusc-s92nv-openshift-ingress-vbxnn
INFO Deleted                                       id=subnet-0016e00f58695ffac resourceType=subnet
INFO Deleted                                       id=weli-eusc-s92nv-worker-role name=weli-eusc-s92nv-worker-role policy=weli-eusc-s92nv-worker-policy
INFO Deleted                                       id=weli-eusc-s92nv-worker-role name=weli-eusc-s92nv-worker-role
INFO Deleted
INFO Deleted                                       classic load balancer=a2028d926bfdf4d1c86ac8a24131ffc3 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=sg-076c19d61a09ab930 resourceType=security-group
INFO Deleted                                       id=rtb-0e40882cab21e13f6 resourceType=route-table
INFO Deleted                                       id=subnet-0d46147c3e680416a resourceType=subnet
INFO Deleted                                       id=nat-0187425548704793e resourceType=natgateway
INFO Deleted                                       id=additional-listener-v9699/76c7b6bd09fb46d3 resourceType=targetgroup
INFO Disassociated                                 id=rtbassoc-0326cc88c13d95250 resourceType=route-table
INFO Deleted                                       id=rtb-057870fbf836380d9 resourceType=route-table
INFO Deleted                                       id=a2028d926bfdf4d1c86ac8a24131ffc3 resourceType=loadbalancer
INFO Deleted                                       id=vpce-02f160f39bedbac0e resourceType=vpc-endpoint
WARNING could not determine whether hosted zone is private  hosted zone=weli-eusc.ci-eusc.devcluster.openshift.com. id=Z06024201CZSGSQHJJ3CY
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A api-int.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY public zone=/hostedzone/Z04057337TKVHDWNA7XB record set=A api.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY public zone=/hostedzone/Z04057337TKVHDWNA7XB record set=A \052.apps.weli-eusc.ci-eusc.devcluster.openshift.com.
WARNING could not determine whether hosted zone is private  hosted zone=weli-eusc.ci-eusc.devcluster.openshift.com. id=Z06024201CZSGSQHJJ3CY
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A api.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY record set=A \052.apps.weli-eusc.ci-eusc.devcluster.openshift.com.
INFO Deleted                                       id=Z06024201CZSGSQHJJ3CY
INFO Released                                      id=eipalloc-0ddd9a3d2e94b9df4 resourceType=elastic-ip
INFO Deleted                                       id=subnet-0b806b5c7b9e2def9 resourceType=subnet
INFO Deleted                                       id=sg-02738de8acbe8ca39 resourceType=security-group
INFO Deleted                                       id=sg-0031209cdc8d80186 resourceType=security-group
INFO Deleted                                       id=sg-0847162268706887c resourceType=security-group
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=vpc-05f0a100aa18a5abe resourceType=vpc subnet=subnet-0b0d11cbc3ee3e7f8
INFO Released                                      id=eipalloc-0fc6a065c3622a91d resourceType=elastic-ip
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=igw-0ee6005c1878801f2 resourceType=internet-gateway
INFO Deleted                                       NAT gateway=nat-0ab41a56f2a9286a9 id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       NAT gateway=nat-0187425548704793e id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Deleted                                       id=vpc-05f0a100aa18a5abe resourceType=vpc
INFO Time elapsed: 5m46s
INFO Uninstallation complete!
vagrant@10:~/works$

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 2, 2026

openshift-ci bot requested review from neisw and xingxingxia March 2, 2026 18:05