Package Manager Benchmarks

This repo contains a suite of fixtures & tools to track the performance of package managers. We benchmark various Node.js package managers (npm, yarn, pnpm, berry, deno, bun, vlt, nx, turbo) across different project types and scenarios.

Environment

We currently only test the latest linux runner which is the most common GitHub Action environment. The standard GitHub-hosted public runner environment specs are below:

• VM: Linux
• Processor (CPU): 4
• Memory (RAM): 16 GB
• Storage (SSD): 14 GB
• Workflow label: ubuntu-latest

We may add Mac/Windows in the future but it will likely exponentially increase the already slow run time of the suite (~40min).

Overview

The benchmarks measure:

• Per-package installation times
• Total Project installation times
• Run script execution performance
• Standard deviation of results

Project Types (fixtures)

• Next.js
• Astro
• Svelte
• Vue
• Large
• Babylon.js (Massive 3D engine monorepo with 86+ packages)

Package Managers

• npm
• yarn
• pnpm
• Yarn Berry
• Deno
• Bun
• VLT
• NX
• Turbo
• Node.js

Configuration/Normalization

We do a best-effort job to configure each tool to behave as similar as possible to its peers but there's limitations to this standardization in many scenarios (as each tool makes decisions about its default support for security checks/validations/feature-set). As part of the normalization process, we count the number of packages - post-installation - & use that to determine the average speed relative to the number of packages installed. This strategy helps account for when there are significant discrepancies between the package manager's dependency graph resolution (you can read/see more here).

Example:

• Package Manager A installs 1,000 packages in 10s -> an avg. of ~10ms per-package
• Package Manager B installs 10 packages in 1s -> an avg. of ~100ms per-package

Testing Package Installation

The installation tests we run today mimic a matrix of different variations (cold cache, warm cache, cold cache with lockfile, etc) for a variety of test fixtures (ie. we install the packages of a next, vue, svelte & astro starter project).

Supported Tools

• vlt
• npm
• pnpm
• yarn
• yarn berry
• deno
• bun

Supported Registries

Registry benchmarking is available in the registry-clean and registry-lockfile variations.

• npm (https://registry.npmjs.org/)
• vlt (https://registry.vlt.io/npm/)
• aws (CodeArtifact benchmark registry)

Defaults:

• Local and CI run npm,vlt,aws by default for registry-* variations.
• You can limit registries with --registries (wrapper) or BENCH_INCLUDE_REGISTRY (script-level env).

Examples:

# Run only npm + vlt registry benchmarks
./bench run --variation=registry-clean --fixtures=next --registries=npm,vlt

# Run only aws registry benchmarks (requires token)
CODEARTIFACT_AUTH_TOKEN=<token> ./bench run --variation=registry-clean --fixtures=next --registries=aws

Auth notes:

• aws requires CODEARTIFACT_AUTH_TOKEN.

Testing Script Execution

This suite also tests the performance of basic script execution (ex. npm run foo). Notably, for any given build, test or deployment task the spawning of the process is a fraction of the overall execution time. That said, this is a commonly tracked workflow by various developer tools as it involves the common set of tasks: startup, filesystem read (package.json) & finally, spawning the process/command.

Supported Tools

• vlt
• npm
• pnpm
• yarn
• yarn berry
• deno
• bun
• node
• turborepo
• nx

Running Benchmarks

Local Development

• 1. Setup:
  • 1.1 Install jq using your OS package manager
  • 1.2 Install hyperfine, version >= 1.19.0 is required
  • 1.3 Install package managers and corepack:

    npm install -g npm@latest corepack@latest vlt@latest bun@latest deno@latest nx@latest turbo@latest

  • 1.4 Make a new results folder:

    mkdir -p results

• 2. Run benchmarks:

  # Install and benchmark a specific project
  ./bench run --fixtures=<fixture> --variation=<variation>
  
  # Example: Benchmark Next.js with cold cache
  ./bench run --fixtures=next --variation=clean

• 3. Local bench wrapper options:

  # Defaults: next + clean (uses script defaults for runs/warmup)
  ./bench run
  
  # Pick fixtures, package managers, and runs
  ./bench run --fixtures=next,astro --pms=vlt,pnpm --runs=3
  
  # Run all fixtures (except "run")
  ./bench run --fixtures=all --variation=clean
  
  # Run multiple variations in one command
  ./bench run --fixtures=svelte --variation=lockfile,registry-lockfile,run --registries=npm,vlt
  
  # Script-execution benchmark
  ./bench run --variation=run --pms=vlt
  
  # Clean failed runs immediately during run/chart (optional)
  ./bench run --fixtures=next --variation=clean --clean
  
  # Generate chart data (and copy to app/latest) after a run
  ./bench run --fixtures=next --runs=3 --chart
  
  # Run benchmarks then process outputs in one command
  ./bench run --fixtures=next --runs=3 --process
  
  # Process existing results into chart data
  ./bench chart --fixtures=next --variation=clean

  By default, cleaning happens during ./bench process. Use --clean on run/chart for immediate cleanup.

• 4. Process benchmark output:

  # Filter failed runs and generate dated + latest outputs and chart data
  ./bench process
  # Or pass --process directly when running
  ./bench run --fixtures=next --runs=3 --process

GitHub Actions

The benchmarks run automatically on:

• Push to main branch
• Manual workflow trigger

The workflow:

1. Sets up the environment with all package managers
2. Runs benchmarks for each project type (cold and warm cache)
3. Executes task performance benchmarks
4. Processes results and generates visualizations
5. Deploys results to GitHub Pages (main branch only)

Results

Console Output

Each benchmark run provides a summary in the console:

=== Project Name (cache type) ===
package-manager: X.XXs (stddev: X.XXs)
...

Generated Results

Results are organized by date in the YYYY-MM-DD/ directory:

• <fixture>-<variation>.json: Cold cache installation results
• <fixture>-<variation>-package-count.json: The count of packages installed for each package manager for a given fixture and variation
• index.html: Interactive visualization

Visualization

The generated charts show:

• Per-package installation times for each fixture and variation combination
• Total installation times for each different combination
• Run scripts execution performance
• Standard deviation in tooltips
• Summary table with total installation times and package counts

View Results Online

Results are automatically deployed to GitHub Pages when running on the main branch:

https://vltpkg.github.io/benchmarks/

Each run creates a new dated html file with its results, making it easy to track performance over time.

Contributing

1. Fork the repository
2. Create your feature branch
3. Commit your changes
4. Push to the branch
5. Create a Pull Request

Local debugging

You can debug the result processing scripts and the web app locally using data from previous GitHub Action runs of the Package Manager Benchmarks workflow.

Preferred workflow (matches CI):

1. Install and authenticate GitHub CLI (gh auth login).

2. Download artifacts from the latest successful run on main:

   ./scripts/download-latest-artifacts.sh

   This keeps artifact-style directories such as results/results-<fixture>-<variation>/ and versions-temp/.

3. Process downloaded artifacts:

   ./bench process

   This cleans benchmark outputs, builds dated and latest result folders, and generates chart data (results/<date>/chart-data.json and results/latest/chart-data.json).

4. Copy the latest chart data into the app:

   mkdir -p app/latest
   cp results/latest/chart-data.json app/latest/chart-data.json

5. Run the app:

   cd app
   vlt install || npm install
   vlr dev

License

This project is licensed under the BSD-2-Clause-Patent License - see the LICENSE file for details.