CLDSRV-835: Fix flakiness tcp socket hang up by BourgoisMickael · Pull Request #6115 · scality/cloudserver

BourgoisMickael · 2026-03-18T00:32:14Z

This multiple concurrent uploads can help trigger that flakiness case

Object Part Copy / With v4 signature
  When copy source was put by MPU
    "before each" hook for "should copy a part from a source bucket to a different destination bucket":
TimeoutError: socket hang up
  at Socket.socketOnEnd (node:_http_client:598:25)
  at Socket.emit (node:events:531:35)
  at endReadableNT (node:internal/streams/readable:1698:12)
  at process.processTicksAndRejections (node:internal/process/task_queues:90:21)

This multiple concurrent uploads can help trigger that flakiness case ``` Object Part Copy / With v4 signature When copy source was put by MPU "before each" hook for "should copy a part from a source bucket to a different destination bucket": TimeoutError: socket hang up at Socket.socketOnEnd (node:_http_client:598:25) at Socket.emit (node:events:531:35) at endReadableNT (node:internal/streams/readable:1698:12) at process.processTicksAndRejections (node:internal/process/task_queues:90:21) ```

bert-e · 2026-03-18T00:32:18Z

Hello bourgoismickael,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

bert-e · 2026-03-18T00:32:24Z

Request integration branches

Waiting for integration branch creation to be requested by the user.

To request integration branches, please comment on this pull request with the following command:

/create_integration_branches

Alternatively, the /approve and /create_pull_requests commands will automatically
create the integration branches.

BourgoisMickael · 2026-03-18T00:32:33Z

/create_integration_branches

codecov · 2026-03-18T00:32:54Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.24%. Comparing base (24831d6) to head (c4c44f0).
⚠️ Report is 16 commits behind head on development/9.3.
✅ All tests successful. No failed tests found.

Additional details and impacted files

see 1 file with indirect coverage changes

@@                 Coverage Diff                 @@
##           development/9.3    #6115      +/-   ##
===================================================
- Coverage            84.25%   84.24%   -0.01%     
===================================================
  Files                  206      206              
  Lines                13251    13251              
===================================================
- Hits                 11164    11163       -1     
- Misses                2087     2088       +1

Flag	Coverage Δ
file-ft-tests	`67.90% <ø> (ø)`
kmip-ft-tests	`28.08% <ø> (ø)`
mongo-v0-ft-tests	`69.11% <ø> (-0.10%)`	⬇️
mongo-v1-ft-tests	`69.22% <ø> (-0.01%)`	⬇️
multiple-backend	`35.10% <ø> (ø)`
sur-tests	`36.46% <ø> (ø)`
sur-tests-inflights	`37.49% <ø> (-0.03%)`	⬇️
unit	`69.90% <ø> (ø)`
utapi-v2-tests	`34.36% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

claude · 2026-03-18T00:34:22Z

Lines 310-311: same template-literal string concatenation bug that was fixed on line 345 still exists here. The quote-plus-quote inside backticks is literal text, producing garbled output. Collapse into a single string.

Review by Claude Code

claude · 2026-03-18T00:41:42Z

tests/functional/aws-node-sdk/test/object/copyPart.js

+                    // an idle connection just as the client picks it from the pool.
+                    const uploadWithRetry = (params, attempt = 0) =>
+                        s3.send(new UploadPartCommand(params)).catch(err => {
+                            if (attempt < 3) {


The retry catches all errors, not just the transient socket hang up this PR targets. Consider narrowing the condition to only retry on TimeoutError or connection-related errors, so genuine failures (e.g. NoSuchUpload, auth errors) fail fast instead of retrying 3 times.

— Claude Code

It's ok like that, other errors should not happen, 3 retries for non transient errors is not that bad.

claude · 2026-03-18T00:41:55Z

The retry logic in uploadWithRetry catches all errors indiscriminately. Consider narrowing retries to only transient/connection errors (e.g. checking for TimeoutError or ECONNRESET) so that non-transient failures fail fast.

The rest of the changes (adding missing newlines to process.stdout.write, fixing broken template literal concatenation) look good.

Review by Claude Code

BourgoisMickael · 2026-03-18T01:02:38Z

@bert-e reset

bert-e · 2026-03-18T01:02:47Z

Reset complete

I have successfully deleted this pull request's integration branches.

The following options are set: create_integration_branches

bert-e · 2026-03-18T01:03:00Z

Integration data created

I have created the integration data for the additional destination branches.

this pull request will merge bugfix/CLDSRV-835-retry-tcp-flakiness into
development/9.3
w/9.4/bugfix/CLDSRV-835-retry-tcp-flakiness will be merged into development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

You can set option create_pull_requests if you need me to create
integration pull requests in addition to integration branches, with:

@bert-e create_pull_requests

The following options are set: create_integration_branches

bert-e · 2026-03-18T01:03:01Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

The following options are set: create_integration_branches

BourgoisMickael · 2026-03-18T17:14:21Z

/approve

bert-e · 2026-03-18T17:14:37Z

In the queue

The changeset has received all authorizations and has been added to the
relevant queue(s). The queue(s) will be merged in the target development
branch(es) as soon as builds have passed.

The changeset will be merged in:

✔️ development/9.3
✔️ development/9.4

The following branches will NOT be impacted:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

There is no action required on your side. You will be notified here once
the changeset has been merged. In the unlikely event that the changeset
fails permanently on the queue, a member of the admin team will
contact you to help resolve the matter.

IMPORTANT

Please do not attempt to modify this pull request.

Any commit you add on the source branch will trigger a new cycle after the
current queue is merged.
Any commit you add on one of the integration branches will be lost.

If you need this pull request to be removed from the queue, please contact a
member of the admin team now.

The following options are set: approve, create_integration_branches

bert-e · 2026-03-18T17:32:02Z

I have successfully merged the changeset of this pull request
into targetted development branches:

✔️ development/9.3
✔️ development/9.4

The following branches have NOT changed:

development/7.10
development/7.4
development/7.70
development/8.8
development/9.0
development/9.1
development/9.2

Please check the status of the associated issue CLDSRV-835.

Goodbye bourgoismickael.

CLDSRV-835: Add newlines to log lines

c4c44f0

BourgoisMickael force-pushed the bugfix/CLDSRV-835-retry-tcp-flakiness branch from 1135239 to c4c44f0 Compare March 18, 2026 00:40

claude bot reviewed Mar 18, 2026

View reviewed changes

scality deleted a comment from bert-e Mar 18, 2026

BourgoisMickael requested review from a team, anurag4DSB, fredmnl and leif-scality and removed request for a team March 18, 2026 15:47

tmacro approved these changes Mar 18, 2026

View reviewed changes

leif-scality approved these changes Mar 18, 2026

View reviewed changes

bert-e merged commit bf92ae2 into development/9.3 Mar 18, 2026
53 of 54 checks passed

bert-e deleted the bugfix/CLDSRV-835-retry-tcp-flakiness branch March 18, 2026 17:32

Conversation

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Hello bourgoismickael,

Uh oh!

bert-e commented Mar 18, 2026

Request integration branches

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

codecov bot commented Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

claude bot Mar 18, 2026

Choose a reason for hiding this comment

Uh oh!

BourgoisMickael Mar 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

claude bot commented Mar 18, 2026

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Reset complete

Uh oh!

bert-e commented Mar 18, 2026

Integration data created

Uh oh!

bert-e commented Mar 18, 2026

Waiting for approval

Uh oh!

BourgoisMickael commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

In the queue

Uh oh!

bert-e commented Mar 18, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov bot commented Mar 18, 2026 •

edited

Loading

BourgoisMickael Mar 18, 2026 •

edited

Loading