OCPBUGS-23746: apiservice: Add retry to API services checks

[tchap]

checkDiscoveryForByAPIServices now performs 3 attempts to pass the check, each with 10-second timeout, 5-second backoff in between.

[openshift-ci-robot]

@tchap: This pull request references Jira Issue OCPBUGS-23746, which is invalid:

• expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "4.21.z" instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

checkDiscoveryForByAPIServices now performs 3 attempts to pass the check, each with 10-second timeout, 5-second backoff in between.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: tchap
Once this PR has been reviewed and has the lgtm label, please assign p0lyn0mial for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• pkg/operator/apiserver/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[sdodson]

Even when a node is rebooting this problem should really just be a single request that fails while existing connections are reset and new connections are made through the load balancer? Or would this condition extend for the duration in which a given member is down?

[tchap]

Even when a node is rebooting this problem should really just be a single request that fails while existing connections are reset and new connections are made through the load balancer? Or would this condition extend for the duration in which a given member is down?

@sdodson I started with checking this job. I thought it stayed unavailable for 7 seconds, but now I can see it actually blipped twice. Once very short for like 150 ms with 503, the other for 1.5 seconds or so with context deadline exceeded. Hmm.