PRNG: use xoshiro256++ in random mode, keep LCG in repeatable#60
Open
rlseaman wants to merge 1 commit intoSmithsonian:mainfrom
Open
PRNG: use xoshiro256++ in random mode, keep LCG in repeatable#60rlseaman wants to merge 1 commit intoSmithsonian:mainfrom
rlseaman wants to merge 1 commit intoSmithsonian:mainfrom
Conversation
repeatable mode preserves the historical LCG so output is bit-for-bit reproducible across digest2 versions (the reproducibility contract). random mode uses xoshiro256++ (Blackman & Vigna 2018) for its longer period, better seed decorrelation, and parallel-stream primitives. No user-facing config change: the PRNG follows the existing repeatable/random mode choice. See PRNG_TRADE_STUDY.md for the trade study and rationale. Verified on macOS: sorted output in repeatable mode is byte-identical to pre-patch upstream; random mode runs clean and produces scores within expected Monte Carlo noise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Author
|
Sorry for the Claude boilerplate description. It was a fairly interesting trade study and I've been planning to suggest improvements to the digest2 PRNG for a long time. The basic notion is to keep the current algorithm around for repeatable mode, while supporting a better, more modern PRNG for stochastic use cases. |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR upgrades digest2’s pseudo-random generation by using xoshiro256++ for random mode while preserving the legacy LCG for repeatable mode to maintain cross-version bit-for-bit reproducibility.
Changes:
- Add splitmix64 seeding + xoshiro256++ generator and route
tkRand()based onrepeatable. - Extend
trackletto carry xoshiro state and preserve it across tracklet recycling. - Update CLI/library seeding sites and document the PRNG-by-mode behavior.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
digest2/digest2/digest2.h |
Adds xoshiro state to tracklet and exposes tkSeed() API. |
digest2/digest2/digest2.c |
Uses tkSeed() for per-thread seeding and preserves the 4-word RNG state in resetInvalid(). |
digest2/digest2/d2math.c |
Implements splitmix64 + xoshiro256++ and switches tkRand() behavior by mode. |
digest2/digest2/d2lib.c |
Routes library seeding through tkSeed() for both repeatable and random modes. |
digest2/digest2/OPERATION.md |
Documents that repeatable uses LCG while random uses xoshiro256++. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+97
to
+98
| uint64_t rand64; // LCG random number (also state slot 0 for xoshiro) | ||
| uint64_t rng_s[4]; // xoshiro256++ state (used when prngChoice != PRNG_LCG) |
| tk->rand64 = 2 * (seed / 2) + 1; /* LCG: odd seed per NAG */ | ||
| uint64_t s = seed; | ||
| for (int i = 0; i < 4; i++) | ||
| tk->rng_s[i] = splitmix64(&s); /* xoshiro: expand via splitmix */ |
| digest2's scores are insensitive to that difference in practice. | ||
|
|
||
| Users need not (and cannot) choose the PRNG directly: the `repeatable` vs. | ||
| `random` decision selects it. See `PRNG_TRADE_STUDY.md` for detail. |
| void initGlobals(void); | ||
| void score(tracklet * tk); | ||
| double tkRand(tracklet * tk); | ||
| void tkSeed(tracklet * tk, uint64_t seed); |
| /* | ||
| * xoshiro256++ (Blackman & Vigna 2018) — opt-in alternative PRNG. | ||
| * Period 2^256-1, passes BigCrush, has jump() / long_jump() primitives | ||
| * for provably-disjoint parallel substreams. See PRNG_TRADE_STUDY.md. |
Member
|
Hi @rlseaman , Whenever any of us will have time to review it, we will work on it, but we are quite busy at the moment, so this might have to wait for some time. Thanks, |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
randommode.repeatablemode so output remains bit-for-bit reproducible across digest2 versions.repeatable/randomkeywords select the PRNG.Context
This work comes from the Catalina Sky Survey (CSS) — the MPC's most prolific tracklet-producing site and a heavy user of digest2 in production scoring pipelines. CSS has a long-running R&D interest in improvements for survey-scale parallel workloads, and this is the first of a small series of upstream-targeted PRs drawn from that work.
Motivation
The LCG (
a=13^13, m=2^59, 59-bit state, 2^57 period) works for digest2's current workload but has three structural limits that matter at scale:rand() = 1andrand() = 2both seed the LCG to state1after the2*(k/2)+1rounding — adjacent seeds produce identical starts.jump()primitive for provably-disjoint parallel substreams.None of these are urgent bugs today. They gate future HPC / parallel deployment.
Why couple PRNG to mode rather than expose a `prng` keyword
repeatablemode promises deterministic output. Pinning LCG there extends the promise across versions: a saved digest2 score from 2023 should replay bit-for-bit in 2026. This is the reproducibility-across-versions contract relied on by CI pipelines, bug reports, and cross-survey validation.randommode's benefits from a better PRNG — longer period, seed decorrelation, parallel jumps — all matter only inrandommode. Using xoshiro256++ where it helps, LCG where stability matters.Users don't choose the PRNG directly because they shouldn't have to: the
repeatablevsrandomdecision already captures the intent.What changes
d2math.ctkRandbranches onrepeatabledigest2.huint64_t rng_s[4]next torand64;tkSeedprototypedigest2.ctkSeed()at both seed sites;resetInvalidsaves the full 4-word stated2lib.ctkSeed()at the Python-binding seed sitesOPERATION.mdNet: 79+/11− across 5 changed files.
d2cli.cis untouched.What does NOT change
repeatablemode output is bit-for-bit identical to pre-patch. Verified on macOS: 0-line sorted diff against pre-patchmainon a 500-tracklet benchmark.repeatable/randomconfig keywords themselves are unchanged.rand(),d2lib.crand_r()) is preserved — both routed throughtkSeed().Verification
-O2) and RHEL 8.10 (x86_64, gcc 8.5), no warnings.Performance
Macro-benchmark on 3000 tracklets,
repeatablemode, best-of-3:This is not a speed PR. A full 6-way PRNG comparison confirms all modern PRNGs are within ~2% on real digest2 — the bottleneck isn't PRNG. This is a capability PR: period, parallel-stream primitives, statistical quality headroom.
Further detail
Full design rationale, statistical comparison (LCG vs xoroshiro128+ vs xoshiro256++ with χ², moments, autocorrelation, 2D uniformity, bit-balance, gap tests), speed benchmark (micro + macro, 6 PRNGs), and deployment considerations (HPC / GPU / embedded / web / SQL) are in the trade study document:
→ PRNG_TRADE_STUDY.md (on CSS R&D branch)
References:
Test plan
-O2)repeatablemode byte-identical to pre-patch on 500-tracklet benchmark (macOS)randommode runs cleanly on both platformsresetInvalidcorrectly preserves 4-word xoshiro state across tracklet recycling