Bump Cli-testing to 1.3.0, update kafka libraries by SylvainSenechal · Pull Request #2341 · scality/Zenko

SylvainSenechal · 2026-03-02T16:23:10Z

Bump cli testing to 1.3.0 => Breaking changes around the keycloak seeder, so updated a few things related to that
Updated node runner to 24
Updated cucumber version from 10.x to 12.x
Updated kubernetes client from 0.2 to 1.4...
Kafka libraries : align all kafka interactions on the new JS native kafka library : platformatic/kafka => drop kafkajs, drop node-rdkafka (and drop the annoying node-gyp that was needed for node-rdkafka, which was always causing weird python warnings when installing). => This new kafka library is compatible with Bunm unlike the previous ones, so later we can investigate if using Bun is worth it (I believe it is 🌚 ), before that we could'nt even try because of this lib

Ended up adding some extra stuff as I was testing my changes in Codespace and found a few stuff bothering me. I think the pr is still not too hard to review.

Can review commit by commit too

bert-e · 2026-03-02T16:23:14Z

Hello sylvainsenechal,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
`/after_pull_request`	Wait for the given pull request id to be merged before continuing with the current one.
`/bypass_author_approval`	Bypass the pull request author's approval	⭐
`/bypass_build_status`	Bypass the build and test status	⭐
`/bypass_commit_size`	Bypass the check on the size of the changeset `TBA`	⭐
`/bypass_incompatible_branch`	Bypass the check on the source branch prefix	⭐
`/bypass_jira_check`	Bypass the Jira issue check	⭐
`/bypass_peer_approval`	Bypass the pull request peers' approval	⭐
`/bypass_leader_approval`	Bypass the pull request leaders' approval	⭐
`/approve`	Instruct Bert-E that the author has approved the pull request.		✍️
`/create_pull_requests`	Allow the creation of integration pull requests.
`/create_integration_branches`	Allow the creation of integration branches.
`/no_octopus`	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
`/unanimity`	Change review acceptance criteria from `one reviewer at least` to `all reviewers`
`/wait`	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
`/help`	Print Bert-E's manual in the pull request.
`/status`	Print Bert-E's current status in the pull request `TBA`
`/clear`	Remove all comments from Bert-E from the history `TBA`
`/retry`	Re-start a fresh build `TBA`
`/build`	Re-start a fresh build `TBA`
`/force_reset`	Delete integration branches & pull requests, and restart merge process from the beginning.
`/reset`	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

SylvainSenechal · 2026-03-11T15:12:21Z

tests/ctst/package.json

    "build": "tsc --build tsconfig.json",
    "lint": "eslint ."
+  },
+  "resolutions": {


related to issues newer azure libraries : https://scality.atlassian.net/browse/ZENKO-5213

SylvainSenechal · 2026-03-11T15:15:02Z

tests/ctst/steps/utils/utils.ts

This was an infinite loop, although catched by global test timeout, still nicer to properly control while loops

SylvainSenechal · 2026-03-11T15:15:46Z

tests/ctst/steps/utils/kubernetes.ts

These are all small syntax changes because I did a huge bump of the kubernetes client from version 0.x to 1.4.0

SylvainSenechal · 2026-03-11T15:18:04Z

tests/ctst/steps/utils/kafka.ts

+ * new `putBucketNotificationConfiguration` to the connector before the
+ * test proceeds to trigger events.
+ */ 
+export async function waitForBucketInConnectorPipeline(


Used in bucket notification : Before, we were doing

putBucketNotificationConfiguration

sleep for 10sec and pray the oplog has picked up the change and updated kafka connector.

It was risky, because triggering the notification event before kafka connector was setup would imply the notification would never be sent

Now we detect the kafka connector is really setup.

I'm kinda sad though : I thought this would fix bucket notif flaky, but it doesn't, so there are other things to check

SylvainSenechal · 2026-03-11T15:19:18Z

tests/ctst/steps/bucket-policies/common.ts

            assert.strictEqual(this.getResult().err?.includes('AccessDenied') ||
-                this.getResult().err?.includes('403'), true);
+                this.getResult().err?.includes('403')||
+                this.getResult().statusCode === 403, true);


For some reason, I got issues with this part not catching the 403 anymore.
Not fully sure why, but with the bump of the cli testing, the version of the sdk has increased quite a lot, from ~3.900 to ~3.100, maybe something changed I don't know

SylvainSenechal · 2026-03-11T15:20:41Z

tests/ctst/features/azureArchive.feature

-    Given that lifecycle is "resumed" for the "e2e-azure-archive" location
-    Then object "obj-1" should be "transitioning" and have the storage class "e2e-azure-archive"
-    And object "obj-2" should be "transitioning" and have the storage class "e2e-azure-archive"
+# This test is flaky, and doesn't make much sense as it is :


@francoisferrand Let's discuss this. I only commented for now, maybe we wanna keep it ?

But looking at the test it doesnt make much sense (cf the comment I added in the code).
But at the same time I feel like I'm missing something, there must be a reason why this test was created

Don't forget to resolve this before merging 😉

SylvainSenechal · 2026-03-11T15:21:40Z

tests/ctst/common/hooks.ts

    cleanupAccount,
 } from './utils';

+import 'cli-testing/hooks/KeycloakSetup';


We have BeforeAll in Cli-testing, no import => they are not run
Imo should be in this repo but its a story for another day

SylvainSenechal · 2026-03-11T15:22:13Z

.github/scripts/end2end/run-e2e-ctst.sh

 POD_NAME="${ZENKO_NAME}-ctst-tests"
 CTST_VERSION=$(sed 's/.*"cli-testing": ".*#\(.*\)".*/\1/;t;d' ../../../tests/ctst/package.json)

-# Configure keycloak


Seed keycloak is now done in a before all from Cli-testing

SylvainSenechal · 2026-03-11T15:25:23Z

tests/ctst/common/common.ts

 import { safeJsonParse } from './utils';
 import assert from 'assert';
-import { Admin, Kafka } from 'kafkajs';
+import { Admin } from '@platformatic/kafka';


Switched the kafka library
Before we were using kafkajs, and node-rdkafka.

These internally relied on librdkafka, a C library. Because of that, we needed to setup quite a few extra stuff to make them work : node-gyp (which was throwing weird python errors when doing yarn install in the Codespace), and also need a bunch of things to be installed in the dockerfile (you'll see later in the pr, librd-XXX).
Now this library is pure js, it removes some dependency, and also it will be compatible with Bun if we wanna use it later

notes:

such comments would be better in the PR comment, so reviewers know before even reading the code

would be good to keep such change in a separate PR (even if it means stacking multiple PRs) - not saying to change it now (too late), but as a goal for next changes

Thanks for the kafka change!

SylvainSenechal · 2026-03-11T15:26:42Z

tests/ctst/steps/azureArchive.ts

        const tarName = await isObjectRehydrated(this, objectName);
        assert(tarName);
-        await AzureHelper.sendBlobCreatedEventToQueue(
+        const sent = await AzureHelper.sendBlobCreatedEventToQueue(


Lost crazy amount of time on this :
sendBlobCreatedEventToQueue was erroring, the error wasn't thrown but instead the function return a boolean that was never checked -_-

francoisferrand · 2026-03-13T10:11:04Z

tests/ctst/common/common.ts

+            const low = earliestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);
+            const high = latestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);


partitions are lists, right? So searching is not efficient...

are partitions "ordered" predictably?

do earliest and latest have the same order?

...could we not just do a "zip" merge, something like:

for (let i = 0; i < partitionCount; i++) { partitions.push({ low: earliestResult[0]?.partitions[I], high: latestResult[0]?.partitions[I] });

I think we don't have guarantee about the order, but the list should be small enough that in practice this find is not too impactfull

bert-e · 2026-03-13T12:47:01Z

Waiting for approval

The following approvals are needed before I can proceed with the merge:

the author
2 peers

maeldonn

LGTM

maeldonn · 2026-03-13T14:30:01Z

tests/ctst/Dockerfile

 FROM ghcr.io/scality/zenko-drctl:$DRCTL_TAG AS drctl

-FROM node:22.19.0-bookworm-slim
+FROM node:24-bookworm-slim


Shouldn't we pin the version like before ?

yeah gonna use 24.14

maeldonn · 2026-03-13T14:31:38Z

tests/ctst/features/azureArchive.feature

-    Given that lifecycle is "resumed" for the "e2e-azure-archive" location
-    Then object "obj-1" should be "transitioning" and have the storage class "e2e-azure-archive"
-    And object "obj-2" should be "transitioning" and have the storage class "e2e-azure-archive"
+# This test is flaky, and doesn't make much sense as it is :


Don't forget to resolve this before merging 😉

maeldonn · 2026-03-13T14:42:00Z

tests/ctst/steps/bucket-policies/common.ts

-            // S3 is correct.
            assert.strictEqual(this.getResult().err?.includes('AccessDenied') ||
-                this.getResult().err?.includes('403'), true);
+                this.getResult().err?.includes('403')||


Do we really need to keep this condition ?

I am not 100% sure anymore tbh, might have been something that failed while i was working, but probably only need one of the two now.
But a/b testing by launching the ci again with and without i too long to just fix this

francoisferrand · 2026-03-18T08:31:54Z

/after_pull_request=2357

bert-e · 2026-03-18T08:32:00Z

Waiting for other pull request(s)

The current pull request is locked by the after_pull_request option.

In order for me to merge this pull request, run the following actions first:

➡️ Merge the OPEN pull request:

pull request Release 2.13.9 #2357

Alternatively, delete all the after_pull_request comments from this pull request.

The following options are set: after_pull_request

Issue: Zenko-5203

Issue: ZENKO-5203

…file Issue: ZENKO-5203

Issue: ZENKO-5203

…2.7.0 Issue: ZENKO-5203

Issue: ZENKO-5203

…ibraries Issue: ZENKO-5203

…he DLQ was ~250 in the tests, stopping sorbetclt from restoring objects Issue: ZENKO-5203

…rs instead of sleep Issue: ZENKO-5203

Issue: ZENKO-5203

…prove debugging Issue: ZENKO-5203

Issue: ZENKO-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch 7 times, most recently from 50ff85e to b441c6e Compare March 4, 2026 09:49

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch 2 times, most recently from b0358e1 to e39e7a2 Compare March 11, 2026 15:10

SylvainSenechal commented Mar 11, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from e39e7a2 to dedf6c3 Compare March 12, 2026 13:05

SylvainSenechal marked this pull request as ready for review March 12, 2026 13:08

SylvainSenechal requested review from a team, benzekrimaha, francoisferrand and maeldonn and removed request for benzekrimaha March 12, 2026 13:08

delthas mentioned this pull request Mar 12, 2026

Add CTST tests for storage usage reporting API #2340

Open

francoisferrand reviewed Mar 13, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from dedf6c3 to 4a27e41 Compare March 13, 2026 10:15

scality deleted a comment from bert-e Mar 13, 2026

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 4a27e41 to 0c6fedd Compare March 13, 2026 13:24

maeldonn approved these changes Mar 13, 2026

View reviewed changes

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 0c6fedd to 879ebee Compare March 13, 2026 15:46

SylvainSenechal added 20 commits March 18, 2026 13:03

use node 24 for ctst linting step

a3b9307

Issue: Zenko-5203

use new platformatic kafka lib for ctst

fe2e6ca

Issue: ZENKO-5203

replace kafkajs library with platformatic kafka

629ceae

Issue: ZENKO-5203

remove node gyp following removal of node rd kafka

452509a

Issue: ZENKO-5203

remove useless tools previously used for node rd kafka in ctst docker…

40a8e26

…file Issue: ZENKO-5203

bump cucumber to 12.7

d266ac3

Issue: ZENKO-5203

bump node image to 24 in ctst dockerfile following cucumber bump to 1…

2bbf35f

…2.7.0 Issue: ZENKO-5203

bump kubernetes client to 1.4.0

8448842

Issue: ZENKO-5203

bump cli-testing to 1.3.0

a056c2d

Issue: ZENKO-5203

seed keycloak from cli testing before all following 1.3.0 bump

d306ee1

Issue: ZENKO-5203

update cucumber formatter

c3cc3aa

Issue: ZENKO-5203

fix infinite loop on verify object location

13c8761

Issue: ZENKO-5203

fix 403 auth check

4d3c957

Issue: ZENKO-5203

add missing cold storage tag to test

314b41b

Issue: ZENKO-5203

pin azure core client following issues with mismatched azure client l…

40e4449

…ibraries Issue: ZENKO-5203

increase sorbetclt limit to 10000 from default 100, as it was found t…

56a899d

…he DLQ was ~250 in the tests, stopping sorbetclt from restoring objects Issue: ZENKO-5203

increase notification test reliability by checking for kafka connecto…

fcefce9

…rs instead of sleep Issue: ZENKO-5203

remove useless azure archive test

0f02383

Issue: ZENKO-5203

improve uniqueness of object names in tests to avoid collision and im…

7e05fe0

…prove debugging Issue: ZENKO-5203

add cucumber extension to codespace

e493b0f

Issue: ZENKO-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 879ebee to 5ab91e4 Compare March 18, 2026 14:15

remove after hook for azure archive

930707a

Issue: ZENKO-5203

SylvainSenechal force-pushed the improvement/ZENKO-5203 branch from 5ab91e4 to 930707a Compare March 18, 2026 14:19

		const low = earliestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);
		const high = latestResult[0]?.partitions.find(p => p.partitionIndex === i)?.offset ?? BigInt(0);

Conversation

SylvainSenechal commented Mar 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bert-e commented Mar 2, 2026

Hello sylvainsenechal,

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand Mar 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bert-e commented Mar 13, 2026

Waiting for approval

Uh oh!

maeldonn left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

francoisferrand commented Mar 18, 2026

Uh oh!

bert-e commented Mar 18, 2026

Waiting for other pull request(s)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

SylvainSenechal commented Mar 2, 2026 •

edited

Loading

francoisferrand Mar 13, 2026 •

edited

Loading