CLDSRV-835: Retry tcp flakiness with concurrency by BourgoisMickael · Pull Request #6116 · scality/cloudserver

BourgoisMickael · 2026-03-18T01:07:15Z

Multi-Object Versioning Delete Success / With default signature
  "before each" hook for "should batch delete 1000 objects quietly":
Error: read ECONNRESET
  at TCP.onStreamRead (node:internal/stream_base_commons:216:20)

The other option would be to change the http agent on the s3 client to use agentkeepalive lib and define freeSocketTimeout to be less than cloudserver keep alive (5s).

``` Multi-Object Versioning Delete Success / With default signature "before each" hook for "should batch delete 1000 objects quietly": Error: read ECONNRESET at TCP.onStreamRead (node:internal/stream_base_commons:216:20) ``` The other option would be to change the http agent on the s3 client to use agentkeepalive lib and define freeSocketTimeout to be less than cloudserver keep alive (5s).

bert-e · 2026-03-18T01:07:20Z

Hello bourgoismickael,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-18T01:07:26Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

claude · 2026-03-18T01:08:08Z

LGTM

The retry logic is clean: bounded retries (3 attempts), proper error propagation on exhaustion, and the comment clearly explains the root cause (idle sockets closed by keepAliveTimeout racing with Promise.all batch waits).

Review by Claude Code

BourgoisMickael · 2026-03-18T01:09:19Z

/create_integration_branches

bert-e · 2026-03-18T01:09:30Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-835-mongo-econreset into
development/9.3
w/9.4/bugfix/CLDSRV-835-mongo-econreset will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-03-18T01:09:32Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

codecov · 2026-03-18T01:11:36Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.26%. Comparing base (24831d6) to head (5ac52b8).
✅ All tests successful. No failed tests found.

Additional details and impacted files

see 2 files with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.3    #6116      +/-   ##
===================================================
+ Coverage            84.25%   84.26%   +0.01%     
===================================================
  Files                  206      206              
  Lines                13251    13251              
===================================================
+ Hits                 11164    11166       +2     
+ Misses                2087     2085       -2

Flag	Coverage Δ
file-ft-tests	`67.90% <ø> (ø)`
kmip-ft-tests	`28.08% <ø> (ø)`
mongo-v0-ft-tests	`69.13% <ø> (-0.08%)`	⬇️
mongo-v1-ft-tests	`69.22% <ø> (-0.01%)`	⬇️
multiple-backend	`35.10% <ø> (ø)`
sur-tests	`36.43% <ø> (-0.04%)`	⬇️
sur-tests-inflights	`37.49% <ø> (-0.04%)`	⬇️
unit	`69.90% <ø> (ø)`
utapi-v2-tests	`34.36% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

dvasilas · 2026-03-18T16:26:37Z

tests/functional/aws-node-sdk/test/versioning/multiObjectDelete.js

+            const putWithRetry = async (params, attempt = 0) => {
+                try {
+                    return await s3.send(new PutObjectCommand(params));
+                } catch (err) {


nit: We could filter and retry only on ECONNRESET.

There could be multiple network error arising depending on the race case, for example ECONNRESET or TimeoutError: socket hang up.

We should not have any case where we have a permanent error in here, so I'd say it's not that bad if we retry 3 times any kind of error

BourgoisMickael · 2026-03-18T17:14:42Z

/approve

bert-e · 2026-03-18T17:14:59Z

Build failed

The build for commit did not succeed in branch w/9.4/bugfix/CLDSRV-835-mongo-econreset

The following options are set: approve, create_integration_branches

bert-e · 2026-03-18T19:13:17Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/9.3
✔️ development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2026-03-18T19:40:40Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.3
✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

Please check the status of the associated issue CLDSRV-835.

Goodbye bourgoismickael.

BourgoisMickael requested review from a team, anurag4DSB, fredmnl and tcarmet and removed request for a team March 18, 2026 15:46

dvasilas approved these changes Mar 18, 2026

View reviewed changes

tmacro approved these changes Mar 18, 2026

View reviewed changes

bert-e merged commit b3e9b26 into development/9.3 Mar 18, 2026
31 checks passed

bert-e deleted the bugfix/CLDSRV-835-mongo-econreset branch March 18, 2026 19:40

Conversation

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Hello bourgoismickael,

Uh oh!

bert-e commented Mar 18, 2026

Request integration branches

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Integration data created

Uh oh!

bert-e commented Mar 18, 2026

Waiting for approval

Uh oh!

codecov bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

dvasilas Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Build failed

Uh oh!

bert-e commented Mar 18, 2026

In the queue

Uh oh!

bert-e commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 18, 2026 •

edited

Loading