CLDSRV-805: Fix flaky GCP tests due to rate limit by BourgoisMickael · Pull Request #6111 · scality/cloudserver

BourgoisMickael · 2026-03-16T18:57:02Z

SlowDown: The project exceeded the rate limit for bucket operations
(bucket creating, updating and deleting). Reduce your request rate.

Fix GCP bucket rate related issues by:

retrying more
encapsulating some functions in retry
deduplicating some tests (HeadBucket & Bucket versioning)
regrouping some tests in the same bucket to reduce gcp bucket quota pressure
- in bucket: head.js, get.js -> bucket.js
- inbucket: getVersioning.js, putVersioning.js -> versioning.js
- in object: head.js, get.js, put.js, delete.js, copy.js -> object.js
- in object: putTagging.js, getTagging.js, deleteTagging.js -> tagging.js

This reduces the number of buckets created across a full GCP test run from 21 to 12 (-9 buckets, -43%)

bert-e · 2026-03-16T18:57:06Z

Hello bourgoismickael,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

claude · 2026-03-16T18:59:49Z

tests/functional/raw-node/test/GCP/bucket/bucket.js

+            async.mapLimit(
+                createdObjects,
+                10,
+                async object => gcpClient.send(new PutObjectCommand({


The populateBucket and removeObjects helpers call gcpClient.send() directly without gcpRetry. If a PutObjectCommand or DeleteObjectCommand hits a SlowDown/429 during setup/teardown, the test will still fail. Consider wrapping these calls with gcpRetry to match the PR goal of fixing rate-limit flakiness.

--- Claude Code

This is a copy of existing code to another file

claude · 2026-03-16T19:00:04Z

bucket.js populateBucket (line 74) and removeObjects (line 93) call gcpClient.send() directly without gcpRetry. These setup and teardown helpers are still vulnerable to SlowDown and 429 errors. Wrap with gcpRetry inside these helpers. Review by Claude Code

codecov · 2026-03-16T19:00:14Z

❌ 2 Tests Failed:

Tests completed	Failed	Passed	Skipped
8225	2	8223	0

View the full list of 2 ❄️ flaky test(s)

"after each" hook for "should put an object and set the acl via query param"::PUT object With v4 signature "after each" hook for "should put an object and set the acl via query param"

Flake rate in main: 100.00% (Passed 0 times, Failed 27 times)

Stack Traces | 0.025s run time

Expected values to be strictly equal:
+ actual - expected

+ '404 NOT FOUND'
- '200 OK'

"after each" hook for "should put an object and set the acl via query param"::PUT object With v4 signature "after each" hook for "should put an object and set the acl via query param"

Flake rate in main: 100.00% (Passed 0 times, Failed 27 times)

Stack Traces | 0.025s run time

done() called multiple times in hook <PUT object With v4 signature "after each" hook for "should put an object and set the acl via query param"> of file .../test/object/put.js

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}

claude · 2026-03-17T14:30:28Z

LGTM

Clean consolidation of GCP test files to reduce bucket create/delete operations and avoid rate limiting. Test logic is faithfully preserved, retry defaults are reasonable (6 attempts with exponential backoff up to ~63s), and timeouts are appropriately increased.

Review by Claude Code

bert-e · 2026-03-17T14:36:21Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

claude · 2026-03-17T14:38:30Z

tests/functional/raw-node/test/GCP/object/object.js

+                        Key: badObjectKey,
+                    }, err => {
+                        assert(err);
+                        assert.strictEqual(err.$metadata.httpStatusCode, 404);


Missing optional chaining on err.$metadata. If $metadata is undefined this will throw a TypeError. Use err.$metadata?.httpStatusCode like line 123 does.

Same issue on lines 225 and 244.

— Claude Code

Copied from existing code as is

claude · 2026-03-17T14:38:40Z

tests/functional/raw-node/test/GCP/object/object.js

+                    Bucket: bucketName,
+                    Key: this.test.key,
+                }, (err, res) => {
+                    assert.notStrictEqual(res.VersionId, this.test.uploadId);


Missing error check. If putObject fails, res will be undefined and res.VersionId will throw a TypeError. Add an assert.equal(err, null) check before accessing res.

— Claude Code

Copied from existing code as is

claude · 2026-03-17T14:38:42Z

tests/functional/raw-node/test/GCP/object/object.js

+                        assert.strictEqual(err.$metadata.httpStatusCode, 404);
+                        resolve();
+                    });
+                });


If the assertion fails, the Promise is never settled (no reject call), so the test hangs until timeout instead of reporting the assertion error. Either wrap assertions in try/catch with reject, or use the done callback pattern like the GET Object 404 test at line 116.

— Claude Code

Copied from existing code as is

claude · 2026-03-17T14:39:11Z

object.js:87, 225, 244: Missing optional chaining on err.$metadata.httpStatusCode. If $metadata is undefined, this throws a TypeError. Line 123 correctly uses ?. but these three lines do not.
- Use err.$metadata?.httpStatusCode consistently
object.js:90: Promise wrapping a callback without a reject path. If assertions fail inside the callback, the promise is never settled and the test hangs until timeout.
- Wrap assertions in try/catch with reject, or use the done callback pattern like line 116
object.js:146: Missing error check in putObject callback before accessing res.VersionId. If the call fails, res is undefined and this throws a TypeError.
- Add assert.equal(err, null) before accessing res

Review by Claude Code

Because default retries can go up to 63s total wait time

Head tests where twice in head and get file. Regroup in 1 file for 1 bucket for both tests kind to reduce GCP quota pressure

Exact same test in 2 different files regrouped. Using 1 bucket on test suite to reduce gcp quota pressure

Regroup copy, delete, get, head, put into an object.js file Using 1 bucket on test suite to reduce gcp quota pressure

Using 1 bucket on test suite to reduce gcp quota pressure

claude · 2026-03-17T15:58:07Z

LGTM

Clean test consolidation that reduces GCP bucket creation from 21 to 12 buckets. The retry increase from 3 to 6 attempts with exponential backoff is reasonable for rate-limited GCP operations. Test logic is faithfully preserved across the consolidation.

Review by Claude Code

BourgoisMickael · 2026-03-18T14:39:37Z

/approve

bert-e · 2026-03-18T14:39:51Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge improvement/CLDSRV-805-gcp-rate-limit into
development/9.3
w/9.4/improvement/CLDSRV-805-gcp-rate-limit will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: approve

bert-e · 2026-03-18T15:02:42Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.3
✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

Please check the status of the associated issue CLDSRV-805.

Goodbye bourgoismickael.

The following options are set: approve

claude bot reviewed Mar 16, 2026

View reviewed changes

BourgoisMickael force-pushed the improvement/CLDSRV-805-gcp-rate-limit branch 2 times, most recently from af90065 to d4596ba Compare March 17, 2026 14:34

claude bot reviewed Mar 17, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 17, 2026

BourgoisMickael added 8 commits March 17, 2026 16:47

CLDSRV-805: Increase GCP default retry attempts

0766a1f

CLDSRV-805: Encapsulate GCP command in retry

47ffd1f

CLDSRV-805: Increase GCP test timeout to 120s

2dd90ff

Because default retries can go up to 63s total wait time

CLDSRV-805: Regroup and dedup bucket Head / Get

d1b760d

Head tests where twice in head and get file. Regroup in 1 file for 1 bucket for both tests kind to reduce GCP quota pressure

CLDSRV-805: Regroup and dedup versioning Get/Put

ed70012

Exact same test in 2 different files regrouped. Using 1 bucket on test suite to reduce gcp quota pressure

CLDSRV-805: Regroup object related tests

4b83110

Regroup copy, delete, get, head, put into an object.js file Using 1 bucket on test suite to reduce gcp quota pressure

CLDSRV-805: Factorize setup and cleanup function

7279058

CLDSRV-805: Regroup object tagging into 1 test

5f1edfe

Using 1 bucket on test suite to reduce gcp quota pressure

BourgoisMickael force-pushed the improvement/CLDSRV-805-gcp-rate-limit branch from d4596ba to 5f1edfe Compare March 17, 2026 15:56

BourgoisMickael requested review from a team, anurag4DSB, fredmnl and jonathan-gramain and removed request for a team March 17, 2026 15:59

tcarmet approved these changes Mar 17, 2026

View reviewed changes

dvasilas approved these changes Mar 18, 2026

View reviewed changes

bert-e merged commit 5f1edfe into development/9.3 Mar 18, 2026
51 of 54 checks passed

bert-e deleted the improvement/CLDSRV-805-gcp-rate-limit branch March 18, 2026 15:02

Conversation

BourgoisMickael commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

This reduces the number of buckets created across a full GCP test run from 21 to 12 (-9 buckets, -43%)

Uh oh!

bert-e commented Mar 16, 2026

Hello bourgoismickael,

Uh oh!

claude bot Mar 16, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 16, 2026

Uh oh!

codecov bot commented Mar 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

❌ 2 Tests Failed:

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

bert-e commented Mar 17, 2026

Request integration branches

Uh oh!

claude bot Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 17, 2026

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

claude bot commented Mar 17, 2026

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Integration data created

Uh oh!

bert-e commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

BourgoisMickael commented Mar 16, 2026 •

edited

Loading

codecov bot commented Mar 16, 2026 •

edited

Loading

BourgoisMickael Mar 17, 2026 •

edited

Loading