From v0.66.0 - %SHARD_NUM% placeholder not rendered for shardCount: 1

[rafaip]

Description

In v0.66.0, the defaultPlaceholders fallback for rendering %SHARD_NUM% in single-shard VMAgent deployments was removed during a refactor. This causes StatefulSet creation to fail when the VMAgent CR uses %SHARD_NUM% in affinity/labels with shardCount: 1.

Steps to reproduce

1. Create a VMAgent CR with shardCount: 1 (or unset) and %SHARD_NUM% in the affinity:

  apiVersion: operator.victoriametrics.com/v1beta1
  kind: VMAgent
  metadata:
    name: main
  spec:
    shardCount: 1
    statefulMode: true
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchLabels:
                shard-num: '%SHARD_NUM%'
            topologyKey: kubernetes.io/hostname

2. Deploy with operator v0.66.0+

Expected behavior

%SHARD_NUM% is replaced with "0" (as it was in v0.65.0), and the StatefulSet is created successfully.

Actual behavior

The literal string %SHARD_NUM% is passed to the Kubernetes API, which rejects it:

Warning  ReconciliationError  <unknown>  victoria-metrics-operator  cannot reconcile *v1.StatefulSet for vmagent(0): cannot create new sts vmagent-main under namespace victoria-metrics: StatefulSet.apps "vmagent-main" is invalid: spec.template.spec.affinity.podAntiAffinity.requiredDuringSchedulingIgnoredDuringExecution[0].labelSelector.matchLabels: Invalid value: "%SHARD_NUM%": a valid label must be an empty string or consist of alphanumeric characters, '-', '_' or '.', and must start and end with an alphanumeric character (e.g. 'MyValue',  or 'my_value',  or '12345', regex used for validation is '(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?')

Additionally, the operator deletes the existing StatefulSet before attempting to create the new one, causing a full outage of the VMAgent.

Root cause

PR 1602 (Nov 7, 2025)
In v0.65.0, internal/controller/operator/factory/vmagent/vmagent.go had:

  // To save compatibility in the single-shard version still need to fill in %SHARD_NUM% placeholder           
  var defaultPlaceholders = map[string]string{shardNumPlaceholder: "0"}

This was called in the non-sharded deploy path via:

  newDeploy, err = k8stools.RenderPlaceholders(newDeploy, defaultPlaceholders)                                 

In v0.66.0, this was removed. RenderShard is now only called inside if cr.IsSharded(), which returns false for shardCount <= 1. The placeholder is never rendered.

Impact

• Any VMAgent with shardCount: 1 (or unset) using %SHARD_NUM% in affinity, labels, or annotations will fail
  to reconcile
• The operator deletes the existing StatefulSet before the new one is validated, causing data loss / scrape
  gap during the failed upgrade

Environment

• Operator version: v0.66.1 (helm chart victoria-metrics-operator 0.57.1)
• CRDs version: v0.66.1 (helm chart victoria-metrics-operator-crds 0.6.1)
• Previous working version: v0.65.0 (helm chart victoria-metrics-operator 0.56.4)

[vrutkovs]

Does it make sense to create a VMAgent with shardCount: 1 set? Same for %SHARD_NUM% for unset shardCount - what is the practical usecase for this? Perhaps it needs to be stopped at validation time?

[AndrewChubatiuk]

HPA updates VMAgent shard count and due to decreased load it can scale down to 1. don't see any reason to restrict scaling down to 1 while using sharded mode

[vrutkovs]

Why would HPA change the number of shards instead of number of replicas? It's odd to decrease the number of shards during the VMAgent lifecycle, as you'll have the queue dropped too, and the scrape gap / data loss will be imminent.

[vrutkovs]

Discussed it with Andrew today:

Why would HPA change the number of shards instead of number of replicas?

This can happen on large number of targets - https://docs.victoriametrics.com/vmagent/#scraping-big-number-of-targets - so its a valid behaviour. Sorry for the noise

[AndrewChubatiuk]

fix will be available in upcoming 0.69.0 and 0.68.4

[AndrewChubatiuk]

fix is available in 0.68.4