Editable in manifest mode

[Ikar-Rahl]

Proposed Workflow ("Editable Mode")

I propose a formal "Editable" workflow integrated into Manifest Mode.

This is an ongoing draft of my initial experimentations and is not (yet) a functional proposal.

1. Setup

I am working on MyProj defined by vcpkg.json.

2. Materialization

I signal to vcpkg that zlib should be editable (via a command, environment variable, or configuration).

Expected outcome:

• First, vcpkg automatically resolves the correct version of zlib based on the current manifest, fetches the source (preferably via git clone to preserve history, or source extraction), and places it in a local user-writable directory (e.g., ./vcpkg_edits/zlib).
• Then, vcpkg automatically treats this new local directory as the source for the zlib port, effectively creating a temporary local overlay.

3. Iteration

1. I modify the source in ./vcpkg_edits/zlib.
2. I run the standard build command for MyProj (e.g., cmake --build .).

Crucial Change (The "Dirty Build"):

• For the Port: Vcpkg detects this port is currently "Editable". Instead of the standard behavior (hash check -> clean -> rebuild -> repackage), it performs a Dirty Build (Incremental). It invokes the underlying build system (CMake, Ninja, Make) directly on the existing build artifacts in the editable directory.
• For the Consumer: Simultaneously, the ABI hash of the editable port is updated (via a "dirty bit" or salted hash). This signals to the consumer (MyProj) that a dependency has changed, triggering a non-standard incremental build/re-link of the consumer project against the updated binary.

4. Completion

I validate my changes. I can then commit the changes (if git history was preserved or added) or generate a patch. I disable the editable state, and vcpkg returns to strict manifest compliance.

The Golden Rule: "Whatever happens in editable mode, stays in editable mode." No impact on CI, registry, shared code. Local only.

[Ikar-Rahl]

@microsoft-github-policy-service agree company="Adobe"

[Ikar-Rahl]

Working Proof of Concept (POC)

Here is a working POC demonstrating the proposed workflow.
Sister PR in vcpkg repo: microsoft/vcpkg#49403

By adding the following to vcpkg-configuration.json:

"editable-ports": {
  "path": "editable-ports",
  "ports": [
    "libpng"
  ]
}

You get a directory in editable-ports/libpng that contains 3 subfolders:

• The port itself: Used exactly like an overlay port.
• Src dir: Currently supports vcpkg_from_git, vcpkg_from_github, and vcpkg_from_gitlab (restricted to ports with exactly one call to these functions).
• Build dir: Similar structure to the one in buildtrees.

Internal Logic

We defined 3 new internal variables:

• Editable: Is the current port being built editable?
• Editable-Mode: Is the current vcpkg install process, globally, running in an editable mode? (This allows incremental build and install of downstream dependencies).
• IncrementalInstall: Allows installing only updated files and removing deleted ones.

This is a POC to drive future discussions and enhancements, but it proves the feasibility of this approach and outlines what is needed.

Capabilities

• Source Editing: Direct editing of port sources.
• Content-Based ABI: ABI Hash is calculated based on the content of the src folder (disables cache uploading when in editable mode).
• Incremental Builds: Skips configuration step (relying on the underlying build system to retrigger if needed) and performs incremental build/link.
• Clean Package Dir: The internal package directory is fully cleaned before install (as some ports rely on it being empty).
• Incremental Install: Updates vcpkg_installed based on file content changes.
• Downstream propagation: Incremental build/link/install for all ports depending on the editable port.

Current Limitations

• Dirty Install State: Building a port while having installed files from its previous version in the vcpkg_installed folder may cause issues with legacy CMake (2.x) or exotic generators. FIXED
• Fetcher Support: Only vcpkg_from_git/github/gitlab are currently supported.
• Single Source: Only 1 vcpkg_from_* call is supported; an error is thrown if a second is encountered. FIXED
• Build System: Only vcpkg_cmake_configure is supported (no support for meson, etc., yet).
• Source Updates: Modifications to the vcpkg_from_* arguments after the initial clone are ignored until the user deletes the src folder manually. (This behavior might be desirable).
• Global Flag: Editable-Mode is currently a global flag that applies to all ports, not just those dependent on an editable port. FIXED
• Testing: Not thoroughly tested; edge cases likely exist.

All of these cases can be enhanced, but the priority was to demonstrate that this is a viable technical direction.

[Ikar-Rahl]

Updates

• Wildcard Support: Added wildcard support for the editable-port field. Patterns like "boost-*" or "*" now work as expected.
• Updated Schema: Reflects the new configuration options.
• Staged Install Mechanism: Switched from keeping files in the vcpkg_installed folder to staging them in a temporary directory and restoring them with the correct timestamp if not updated. This allows a port to build with its files properly removed and unavailable during the build process, providing a much more robust approach.
• Editable Subtree Scope: Switched from a global "Editable Mode" flag (which applied to all ports) to an "Editable Subtree" logic. The mode now applies only if a port is explicitly editable or has a direct/indirect dependency that is editable. This correctly limits the incremental build behavior to the affected downstream dependencies.
• ABI Salting: The ABI hash is now salted on the initial editable run to ensure it does not reuse the previously installed non-editable port, forcing a proper initial rebuild.
• General Cleanup: Multiple cleanups from previous iterations (though some artifacts may still remain).

This fixes the Dirty Install State and Global Flag point described before.

[Ikar-Rahl]

Updates

• Added support for multi-source ports: sources/src1, sources/src2 etc. (Single Source limitation fixed)
• Redirected package folder to the editable port (now it contain: port, sources, build and package).

Limitations left:

• Fetcher Support: Only vcpkg_from_git/github/gitlab are currently supported. The rest still build though.
• Build System: Only vcpkg_cmake_configure is supported (no support for meson, etc., yet). They other should still build though.
• Source Updates: Modifications to the vcpkg_from_* arguments after the initial clone are ignored until the user deletes the src folder manually. This will be tackled next.
• Testing: Not thoroughly tested; edge cases likely exist.

[Ikar-Rahl]

Pushed a cleaned up version. Mostly tweaks and fixes.
This seems to us as a functional and stable version, tested on hundred of ports, but mostly on cmake and some pre-build binaries.

[BillyONeal]

Somewhat repeating what I said over email:

The big problem with this feature as presented is that we effectively are forced to rebuild the universe of dependencies with this, because things that are --editable don't have ABI hashes, and the ABI hash is how manifest mode avoids constant rebuilds of things.

In particular, just not blowing away the sources is insufficient to achieve the outcome you're going for here, due to the interaction between ports and the installed tree. The installed tree is a shared resource that is both an input and output of the build and install process. Ports don't expect to have the bits they install already be installed, and ports do not usually have dependencies wired up correctly for incremental builds. That is, given port A depends on B depends on C, which install include/A.h, include/B.h, and include/C.h, respectively, we can't safely build B under our current model without deleting installed/<triplet>/include/A.h and installed/<triplet>/include/B.h first, which breaks any ability to incrementally build A.

As we described over email if the feature as proposed can be made to work we would be happy with it in principle but we think a writeup of how that end to end scenario is supposed to work needs to happen before we should merge this part in isolation.

[Osyotr]

If I understand correctly, the main goal of this PR is to be able to create and test patches without triggering a world rebuild.
I think it's achievable now for most cmake-based and probably many other ports by building without cleaning buildtrees and running ninja/make/etc. directly from the build dir and manually deploying built artifacts.
For reliability, vcpkg could provide some scripts to recreate the environment it was running the build in.

[BillyONeal]

I think it's achievable now for most cmake-based and probably many other ports by building without cleaning buildtrees and running ninja/make/etc. directly from the build dir and manually deploying built artifacts.

The problem is that most builds are not prepared for a previous version of themselves to already be installed. That is, when building zlib, the build system has to deal with "/usr/include/zlib.h" (being the one from the installed tree) and mylocalbuild/src/inc/zlib.h both being reachable. If you're lucky, #include <zlib.h> goes to the second one, but depending on how their build is configured, it might go to the first one making local --editable edits to mylocalbuild/src/inc/zlib.h do nothing.

For reliability, vcpkg could provide some scripts to recreate the environment it was running the build in.

We already have vcpkg env but I don't think that achieves the "editing my dependencies feels the same as editing my top level consumer" scenario that @Ikar-Rahl is trying to achieve here.

[Ikar-Rahl]

Hello

The big problem with this feature as presented is that we effectively are forced to rebuild the universe of dependencies with this, because things that are --editable don't have ABI hashes, and the ABI hash is how manifest mode avoids constant rebuilds of things.

This PR does not rebuild the universe. Editable ports have an ABI.

just not blowing away the sources is insufficient to achieve the outcome you're going for here

The outcome is achieved, this PR is used by our teams without major issue so far and it does everything we wanted it to do.

Ports don't expect to have the bits they install already be installed

That's why we stash the install tree of the port before installing it, no issue there. It does not break incremental ability as the timestamps are restored. We don't do it for it dependencies though, unsure how needed it is but can be added.

I understand that you may not have time to take a serious look at it, but most of our concerns were already covered, the implementation ... just works. There may be some caveats we did not think through, but we are already benefiting from the enhanced workflow in our daily work.

[dg0yt]

This PR does not rebuild the universe. Editable ports have an ABI.

AFAICT the actual topic is the universe of reverse dependencies, and editable ports not having an ABI hash.

• Reverse dependencies are rebuilt when a package is changed. In vcpkg, changes in packages are tracled via ABI hashes which reflect the input files, triplet, and environment.
  https://learn.microsoft.com/en-us/vcpkg/reference/binarycaching#abi-hash
• Artifacts from editable ports change through direct, untracked modificiations to the sources. These modifications cannot be reflected in the ABI hash. That's why editable mode is incompatible with "ABI" tracking in vcpkg, and an ABI hash cannot be assigned to the installation of an editable port.
• Without a corresponding ABI hash, it is not possible to cache binary artifacts, and to determine ABI hashes for artifacts of reverse dependencies.
  Every "editable" change to e.g. zlib will have to rebuild everything which depends on it (e.g. down to Qt, OpenCV, ...), with binary caching disabled.

[Ikar-Rahl]

Artifacts from editable ports change through direct, untracked modificiations to the sources.

They are tracked. Commands.build.cpp 1532 : abi_tag_entries.emplace_back("editable_sources", "0");
Would be useless if it did not detect changes if the sources.

Without a corresponding ABI hash, it is not possible to cache binary artifacts, and to determine ABI hashes for artifacts of reverse dependencies.

We specifically do not want to have binary artifact for editable ports, as those would pollute the cache. So the caching is disabled on purpose.

Every "editable" change to e.g. zlib will have to rebuild everything which depends on it (e.g. down to Qt, OpenCV, ...), with binary caching disabled.

This is actually something wanted, if you edit zlib, you want to rebuild all zlib dependencies. And those are being correctly rebuild, incrementally, and also incrementally installed, preserving timestamps.

As far as we have seen, there is no unnecessary neither missing operations that is happening during the whole workflow.

[dg0yt]

Artifacts from editable ports change through direct, untracked modificiations to the sources.

They are tracked. Commands.build.cpp 1532 : abi_tag_entries.emplace_back("editable_sources", "0"); Would be useless if it did not detect changes if the sources.

The problem is that the ABI hash would need needs to consider the full set of sources to track modifications in editable mode.
This is not solved by putting an "editable" marker into the ABI hash.

[dg0yt]

So IIUC the new code block walks through the source dir (editable_sources_path / "sources") to track modifications to the sources and put that into the hash. This is significantly different from the the current approach in classic mode where vcpkg tool has little knowledge about actual source dirs.

[Ikar-Rahl]

Yes.