[improve][build] Adopt Gradle best practices: convention plugins, build-logic composite build, and unconditional module inclusion

[lhotari]

Motivation

Align the Gradle build with Gradle's documented best practices to improve build performance, maintainability, and configuration cache effectiveness. The main issues addressed:

1. Monolithic subprojects{}/allprojects{} blocks — 260+ lines of shared config injected non-obviously, preventing Gradle's Isolated Projects optimization
2. Build logic in buildSrc — Gradle recommends composite builds over buildSrc because buildSrc changes invalidate the entire build, causing all tasks to lose build cache entries (gradle/gradle#6045). Migrated to build-logic/ composite build.
3. Conditional module inclusion in settings.gradle.kts — dynamic project structure breaks configuration cache and IDE integration
4. Eager task configuration — tasks.withType<> without .configureEach causes unnecessary configuration overhead
5. Configuration cache invalidation — generatePulsarVersion task's gitDirty/gitCommitId inputs caused cascading rebuilds across the entire project on every file change

Modifications

Convention plugins (build-logic composite build replacing buildSrc):

• Migrated buildSrc to build-logic/ composite build per Gradle best practice, so build logic changes only invalidate projects that depend on the changed code
• Created pulsar.java-conventions, pulsar.code-quality-conventions, pulsar.test-certs-conventions, pulsar.nar-conventions convention plugins
• Moved existing pulsar.shadow-conventions and pulsar.client-shade-conventions from buildSrc to build-logic
• Each subproject now explicitly opts in via plugins { id("pulsar.java-conventions") } instead of inheriting from subprojects{}
• Eliminated afterEvaluate + reflection for license plugin configuration
• Deleted buildSrc/ and gradle/code-quality.gradle.kts

Unconditional module inclusion:

• Removed all 4 conditional blocks from settings.gradle.kts (-PcoreModules, -Pdocker, -PintegrationTests, shade-test detection)
• All modules are always included; Docker builds only run when explicitly requested
• Removed assemble→dockerBuild dependency from Docker modules
• Made pulsar-bom project references unconditional

Configuration cache & build performance:

• All tasks.withType<> changed to tasks.withType<>().configureEach for lazy task configuration
• Removed gitDirty and gitCommitId from generatePulsarVersion task inputs to prevent cascading rebuilds
• Migrated pulsar-functions-runtime-all fat JAR from manual dependsOn(runtimeClasspath) + zipTree() to Shadow plugin for configuration-cache-compatible lazy resolution
• Used shadow { addShadowVariantIntoJavaComponent.set(false) } in shadow conventions
• Disabled microbench shadowJar from assemble lifecycle via addShadowJarToAssembleLifecycle

CI & documentation:

• Removed -PcoreModules and -Pdocker flags from GitHub Actions workflows and docker/build.sh
• Updated README files to reflect new build commands
• Added version-catalog-update and versions plugins for dependency management

Verifying this change

• Make sure that the change passes the CI checks.

This change is already covered by existing tests. Build verification:

• ./gradlew help — configuration phase succeeds
• ./gradlew assemble — full build produces identical artifacts
• ./gradlew assemble (second run) — "Reusing configuration cache", all tasks UP-TO-DATE
• ./gradlew clean && ./gradlew assemble — all compileJava/shadowJar/nar tasks served FROM-CACHE
• ./gradlew :pulsar-broker:test --dry-run — test task graph unchanged
• ./gradlew :pulsar-io:pulsar-io-data-generator:nar --dry-run — NAR packaging works

Does this pull request potentially affect one of the following parts:

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc-not-needed