Build fails for t420-maximized/t430-maximized : microcode update file creation race condition (empty when trying to inject in CBFS in final rom stitching step)

[haxalicious]

Context of the Build

1. What board are you trying to build?
t420-maximized

2. What repository:branch are you using to build from?

• Heads:Master
• Other (please specify)

3. What version of coreboot are you trying to build
v24.12

4. In building the rom where did you get the blobs?

• No blobs required
• Provided by the company that installed Heads on the device
• Extracted from a backup rom taken from this device
• Extracted from another backup rom taken from another device (please identify the board model)
• Extracted from the online bios using the automated tools provided in Heads
• I don't know

5. If using the automated tools to get the blobs did you run the relevant scripts in the blobs directory

• Yes
• No

6. What operating system are you using

Arch Linux, however this occurs on Debian 12 as well

Please describe the problem

Describe the bug
Fails building due to coreboot 3rdparty/intel-microcode directory being empty.

To Reproduce
Steps to reproduce the behavior:

1. git clone https://github.com/linuxboot/heads
2. cd heads
3. ./docker_repro.sh make BOARD=t420-maximized

Expected behavior
Build succeeds

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
build.log
coreboot-24.12.log

[tlaurion]

@haxalicious please include `/home/haxalicious/heads/build/x86/log/coreboot-24.12.log.

As you gave in your log:
./docker_repro.sh make BOARD=t420-maximized doesn't produce full logs.
As instructed in matrix chatroom
./docker_repro.sh make BOARD=t420-maximized V=1 CPUS=1 would produce full build log. This only tells you where the error happened.

Your output only shows the end of the build, and is not enough to understand what is happening.

2025-03-26 18:17:25+00:00 MAKE coreboot-24.12
tail /home/haxalicious/heads/build/x86/log/coreboot-24.12.log
-----
    CC         bootblock/southbridge/intel/common/usb_debug.o
    AR         bootblock/external/vboot_reference-bootblock/vboot_fw.a
    CC         bootblock/mainboard/lenovo/t420/static.o
    HOSTCC     cbfstool/ifwitool (link)
    HOSTCC     cbfstool/cse_serger (link)
    LINK       cbfs/fallback/romstage.debug
    OBJCOPY    cbfs/fallback/romstage.elf
    LINK       cbfs/fallback/bootblock.debug
    OBJCOPY    cbfs/fallback/bootblock.elf
    OBJCOPY    bootblock.raw.elf
    OBJCOPY    bootblock.raw.bin
Created CBFS (capacity = 8191460 bytes)
    BOOTBLOCK
    CBFS       cbfs_master_header
    CBFS       cpu_microcode_blob.bin
E: File 'cpu_microcode_blob.bin' is empty?
E: Failed while operating on 'COREBOOT' region!
E: The image will be left unmodified.
make[1]: *** [Makefile.mk:1286: t420-maximized/coreboot.pre] Error 1
make[1]: Leaving directory '/home/haxalicious/heads/build/x86/coreboot-24.12'
make: *** [Makefile:570: /home/haxalicious/heads/build/x86/coreboot-24.12/t420-maximized/.build] Error 1
```

[haxalicious]

@haxalicious please include `/home/haxalicious/heads/build/x86/log/coreboot-24.12.log.

As you gave in your log: ./docker_repro.sh make BOARD=t420-maximized doesn't produce full logs. As instructed in matrix chatroom ./docker_repro.sh make BOARD=t420-maximized V=1 CPUS=1 would produce full build log. This only tells you where the error happened.

Your output only shows the end of the build, and is not enough to understand what is happening.

2025-03-26 18:17:25+00:00 MAKE coreboot-24.12
tail /home/haxalicious/heads/build/x86/log/coreboot-24.12.log
-----
    CC         bootblock/southbridge/intel/common/usb_debug.o
    AR         bootblock/external/vboot_reference-bootblock/vboot_fw.a
    CC         bootblock/mainboard/lenovo/t420/static.o
    HOSTCC     cbfstool/ifwitool (link)
    HOSTCC     cbfstool/cse_serger (link)
    LINK       cbfs/fallback/romstage.debug
    OBJCOPY    cbfs/fallback/romstage.elf
    LINK       cbfs/fallback/bootblock.debug
    OBJCOPY    cbfs/fallback/bootblock.elf
    OBJCOPY    bootblock.raw.elf
    OBJCOPY    bootblock.raw.bin
Created CBFS (capacity = 8191460 bytes)
    BOOTBLOCK
    CBFS       cbfs_master_header
    CBFS       cpu_microcode_blob.bin
E: File 'cpu_microcode_blob.bin' is empty?
E: Failed while operating on 'COREBOOT' region!
E: The image will be left unmodified.
make[1]: *** [Makefile.mk:1286: t420-maximized/coreboot.pre] Error 1
make[1]: Leaving directory '/home/haxalicious/heads/build/x86/coreboot-24.12'
make: *** [Makefile:570: /home/haxalicious/heads/build/x86/coreboot-24.12/t420-maximized/.build] Error 1

Added requested file

[tlaurion]

Can't replicate here and nothing in log helps.
It needs to be a problem with git submodules sync @haxalicious since CircleCI doesn't replicate your issue.

Do
rm -rf build/x86/coreboot-24.12

And restart build with (3rd time asked):
./docker_repro.sh make BOARD=t420-maximized V=1 CPUS=1

No logs, no help :/

./docker_repro.sh make BOARD=t420-maximized V=1

Bare minimal here is V=1

[haxalicious]

build.log

[Oessel]

I just wanted to open the same issue for t430-hotp-maximized, when i saw this...

[tlaurion]

Trying to replicate, removing build artifacts in platform related build dirs, patch created files, cleaning coreboot fork+repatch+build:

sudo rm -rf build/x86/coreboot-24.12/src/mainboard/lenovo/sklkbl_thinkpad/variants/t480*
sudo rm -rf build/x86/coreboot-24.12/t*maximized
sudo rm -rf build/x86/t*-maximized
echo "bogus" |sudo tee build/x86/coreboot-24.12/.canary > /dev/null
./docker_repro.sh make BOARD=t430-hotp-maximized V=1

Still doesn't replicate the race condition.

[tlaurion]

@haxalicious @Oessel : tlaurion@a21d294 should fix this even if I cannot replicate?

Bug+fix reported for discussion under matrix coreboot channel post

[Oessel]

Still the same Error.

[tlaurion]

Still the same Error.

Trying to replicate, removing build artifacts in platform related build dirs, patch created files, cleaning coreboot fork+repatch+build:

sudo rm -rf build/x86/coreboot-24.12/src/mainboard/lenovo/sklkbl_thinkpad/variants/t480*
sudo rm -rf build/x86/coreboot-24.12/t*maximized
sudo rm -rf build/x86/t*-maximized
echo "bogus" |sudo tee build/x86/coreboot-24.12/.canary > /dev/null
./docker_repro.sh make BOARD=t430-hotp-maximized V=1

Still doesn't replicate the race condition.

@Oessel with the commit above cherry-picked and while having clean the artifacts of past build? If the microcode is empty under build dir, the build system di any rewrite it without either cleaning past build artifacts or rebuilding clean.

Would be nice if you could reply doing your repro traces, I'm not mind reader of what was done to arrive to same result and logs provided in this ticket alone don't explain the issue of coreboot build system as of 24.12 release. This is now a Heads issue for people wanting to locally build but neither I or Circleci can reproduce the issue. :/

If one of you wants to open an issue upstream under coreboot that would be nice otherwise I will try to find time doing that in the next days....

[tlaurion]

I was able to reproduce. PoC patch at tlaurion@a21d294 doesn't fix it, but shows more clearly the bug in Makefiles.

I can confirm that in the logs we now have

# Concatenate all microcode binaries into a temporary file,
cd t430-maximized/; /home/user/heads/build/x86/coreboot-24.12/util/crossgcc/xgcc/bin//iasl -vw 3150 -we -p dsdt.aml dsdt.asl
# then atomically move it into place so that CBFS only sees a complete file.
cat   > t430-maximized/cpu_microcode_blob.bin.tmp && mv t430-maximized/cpu_microcode_blob.bin.tmp t430-maximized/cpu_microcode_blob.bin

As can be seen here:

• cat > t430-maximized/cpu_microcode_blob.bin.tmp && mv t430-maximized/cpu_microcode_blob.bin.tmp t430-maximized/cpu_microcode_blob.bin
  • cat (outputs file content of nothing) into t430-maximized/cpu_microcode_blob.bin.tmp which is then moved to mv t430-maximized/cpu_microcode_blob.bin.tmp t430-maximized/cpu_microcode_blob.bin

ls -alh build/x86/coreboot-24.12/t430-maximized/cpu_microcode_blob.bin 
-rw-r--r-- 1 root root 0 Apr  1 12:48 build/x86/coreboot-24.12/t430-maximized/cpu_microcode_blob.bin

Will try to find time to open bug.... The fun with race conditions.

• Pinged coreboot matrix opened thread
• Pinged coreboot devs in Heads matrix channel

[miczyg1]

From what I see t430-maximized coreboot config has the following code options:

CONFIG_CPU_MICROCODE_CBFS_DEFAULT_BINS=y
# CONFIG_CPU_MICROCODE_CBFS_EXTERNAL_BINS is not set
# CONFIG_CPU_MICROCODE_CBFS_EXTERNAL_HEADER is not set
# CONFIG_CPU_MICROCODE_CBFS_NONE is not set

This causes the coreboot to take the microcode blobs defined in Makefiles in various places in the cpu_microcode_bins variable. In case of t430 it should take the blob pointed by <coreboot_root>/src/cpu/intel/model_206ax/Makefile.mk:

cpu_microcode_bins += $(wildcard 3rdparty/intel-microcode/intel-ucode/06-2a-*)
cpu_microcode_bins += $(wildcard 3rdparty/intel-microcode/intel-ucode/06-3a-*)

3rdparty/intel-microcode is a submodule, so if the submoule is not downloaded when the Makefile is being processed, then the wildcard will find no matches. That is the likely explanation of what happens.

I tried to build t430-maximized with ./docker_repro.sh make BOARD=t430-maximized V=1 and in the build directory I can clearly see that the <coreboot_root>/3rdparty/intel-microcode directory is empty. The proper fix would be for the heads build system to initialize the coreboot's submodules (intel-microcode at minimum here) before starting the build process of coreboot, because the upstream coreboot tarballs do not contain the microcode submodule content under aforementioned directory. Knowing heads, you would like to verify these submodules against hashes, right? So heads would have to download tarballs of the submodules from github and check their hashes. Not sure how to get the proper submodule revisions, because the coreboot tarball is stripped from .git directory.

Also maybe worth checking is the CONFIG_USE_BLOBS option. It causes coreboot build system to download the blobs and intel-microcode repo, but only if git is available:

ifneq ($(GIT),)
ifneq ($(UPDATED_SUBMODULES),1)
$(info Updating git submodules.)
# try to fetch non-optional submodules if the source is under git
forgetthis:=$(shell git submodule update --init $(quiet_errors))
# Checkout Cmocka repository
forgetthis:=$(shell git submodule update --init --checkout 3rdparty/cmocka $(quiet_errors))
ifeq ($(CONFIG_USE_BLOBS),y)
# These items are necessary because each has update=none in .gitmodules.  They are ignored
# until expressly requested and enabled with --checkout
forgetthis:=$(shell git submodule update --init --checkout 3rdparty/blobs $(quiet_errors))
forgetthis:=$(shell git submodule update --init --checkout 3rdparty/intel-microcode $(quiet_errors))

@tlaurion