Add digest2 NEO scoring and cross-night tracklet linking Investigated 4 candidate technologies for improving the pipeline; GPU reprojection and local plate-solving were rejected with real reasons (no GPU path in Reproject.jl/wcslib; multi-GB index files for marginal gain), documented in Design refinements. Built the other two: - digest2_score wraps the MPC's own external digest2 classifier (verified by actually cloning, building, and running it). Real verification against ZTF field 451 surfaced a genuinely useful, non-obvious finding: it correctly scored the field's 2 known Main Belt objects low on neo_score, while 131 of 133 tracklets scored 100 — bogus links of stationary stars, an artifact of that demo's own loose match_radius, not something digest2 got wrong. Written up in full on its own wiki page. - link_across_nights extends link_candidates' within-night linear linking across multiple observing nights (fit-and-extrapolate on sky coordinates, union-find grouping). Validated synthetically; a real multi-night same-object case was searched for (checked the existing 5-field IASC/PS1 dataset, then queried SkyBoT directly) but not found within reasonable effort — stated plainly as an open gap rather than skipped past. 188/188 tests pass (digest2_score's live test ran for real against a locally-built binary, not just skipped).

Add mpc80_report: legacy MPC 80-column format export IASC's coordinator confirmed (email reply) that reports must be in the 80-column format specifically, not ADES — which ades_psv's docstring had previously (and correctly, for MPC's own preference) assumed was the only format worth implementing. Column layout verified against the MPC's own published spec (OpticalObs.html, PackedDes.html), not guessed. Reuses the same observer-assigned temporary designation this pipeline's ades_psv trkSub already relies on for undesignated candidates (base-36 tracklet id), one character shorter (7, not 8) to fit the format's designation slot. Refactored julian_date_to_iso8601's Gregorian calendar math into a shared _jd_to_calendar, reused by the new MPC date field formatter.

Profile the pipeline for another build_reference-sized win; mostly not found Profiled detect_sources, _detect_all_frames, find_variable_sources, photometric_scale, and link_candidates directly on a real, unusually dense dataset (V1012 Mon field, ~200-580 detections/frame). Most stages are already fast (find_variable_sources 0.20s, photometric_scale 0.08s for 34 frames). link_candidates' real cost there (0.57s) is a known, already-mitigated design tradeoff (pairwise seed search scaling with detection density, already why threshold gets tuned up for dense fields), not a hidden bug. One real fix found: _detect_all_frames' detections_per_frame was an untyped Vector{Any}, even though detect_sources always returns the same concrete Table type (confirmed directly, not assumed). Typed it explicitly. Measured no significant wall-clock change from this alone — kept as a real, zero-risk type-stability improvement, not reported as a speedup it didn't produce. Full test suite passes unchanged (158/158).

Raise find_variable_sources' systematic_error_fraction default: 2% -> 3% The 2% default rested on one real field (451) and one real confirmed variable from a different field. Found two more real, independently confirmed variables (V1012 Mon, ASASSN-V J072906.85-090518.2) via a real VSX/IRSA search for bright short-period eclipsing binaries with dense real same-night ZTF coverage, then repeated the exact same floor sweep against each: 197 and 182 real matched stationary stars respectively. At 3% (already a measured point in the original field-451 sweep), all three confirmed variables stay above chi2_threshold=10.0 (reduced chi2 13.8/61.2/17.5), while the false-positive rate is the same or better than 2% in every field (1.3%/2.04%/0.55% vs 2.0%/2.55%/0.55%). A literal 0% false-positive rate is reachable at a 5% floor, but only by also pushing two of the three real variables' own signal below threshold — not a real improvement. Full test suite passes unchanged (158/158) with the new default.

Fix the live plate_solve test's fixture: a single fake star can't solve Testing the .env loader with a real ASTROMETRY_API_KEY for the first time actually ran this test live instead of skipping it — and it failed: astrometry.net job "failed to solve". Not a bug in plate_solve itself; the synthetic fixture (one Gaussian source in noise) has no real star pattern for the service's geometric matching to identify. Replaced it with the real ZTF frame already used for this exact live validation (see plate_solve's own docstring) — its real WCS is never read by plate_solve, which only uploads pixels, so this is a legitimate blind-solve test. Skips (distinctly from the missing-API-key case) if that real, gitignored frame isn't present locally. Verified against the live service with a real key: solves successfully, returning a WCSTransform with plausible sky coordinates. Full suite: 158 pass, 0 broken, 0 errors — first time this project has had a fully clean run with a real API key present.

Load ASTROMETRY_API_KEY from a gitignored .env for local test runs The live plate_solve round-trip test was always skipped in practice — nothing set ASTROMETRY_API_KEY without manually exporting it by hand each session. test/runtests.jl now reads a repo-root .env (gitignored; .env.example documents the expected format) before the test suite runs, without overriding a real environment variable if one's already set. Not read by src/ itself — plate_solve takes api_key as an explicit argument, this is test-only convenience. Verified end to end: a real (intentionally invalid) key in .env made the test stop skipping and actually hit the live service, which correctly rejected it — confirms the loader, not just its parsing logic in isolation.

Wire build_reference's workers= into real_data_demo.jl The reference-build step this script documents as "roughly half an hour" was still running sequentially — the new multi-process path existed but nothing used it. Spawns Sys.CPU_THREADS worker processes around the build_reference call, torn down in a finally block regardless of outcome. Verified end to end against real data: full script (baseline pipeline, worker spawn + parallel reference build, ZOGY, both SkyBoT crossmatches) now completes in ~11 minutes, and recovers the same known objects as before (133/2 baseline, 667/2 ZOGY) — the speedup didn't change results.

Merge feature/distributed-build-reference: opt-in multi-process parallelism for build_reference

Update build_reference's benchmark numbers with the real, integrated result The earlier 331.8s/2.2x came from a standalone script that reimplemented the WCS-header-string technique by hand, before build_reference's own workers keyword existed. Re-measured end to end on the actual function, same real 30-frame field-451 set, sequential and distributed back to back: 951.31s vs. 295.92s, 3.21x — better than the earlier estimate, and output (image/sigma/mask) confirmed exactly identical between the two paths.

Add a real CI workflow, fix README badge, remove deploy-plumbing from the wiki There was no workflow that actually ran the test suite — Documenter.yml only builds/deploys docs. Added CI.yml (setup-julia + buildpkg + runtest) and pointed the README's badge at it instead. Also moved the GitHub Pages/mirror troubleshooting story out of design-refinements.md into a local, gitignored PAGES_NOTES.md: the wiki documents the pipeline itself, not its own CI plumbing. Retitled the page (it covers more than "two limitations" now) and fixed the now-stale link text pointing to it from investigation-log.md.

Fix: build docs step needs GITHUB_TOKEN even without deploydocs Removing deploydocs also removed GITHUB_TOKEN from the build step, reasoning it was only needed for the git push. Wrong: makedocs itself decides "dev" vs root base path from Documenter's deploy-criteria check, which needs the token present to pass. Without it, the first real deploy built for root, breaking every asset path once nested under dev/ (a real, live, unstyled site — traced via the run's own log showing "Deploying: ✘" and Bases that will be built: [""]).

Add opt-in multi-process parallelism to build_reference Threads.@threads on the reprojection loop segfaulted (wcslib isn't thread-safe across concurrent calls from one process, documented previously). Distributed.jl processes avoid that hazard, but passing a WCSTransform itself to a worker crashes too: it holds pointers into per-process wcslib memory that don't survive serialization. Fixed by never sending a WCSTransform across the wire — convert to a FITS header string (WCS.to_header) and have each worker rebuild its own local WCS (load_wcs) before reprojecting. Real result on the 30-frame field-451 reference set, 8 worker processes: 331.8s vs. ~720s sequential, ~2.2x. New workers keyword, opt-in and off by default; the caller supplies already-running worker processes rather than build_reference spawning its own.

Deploy docs via GitHub Actions Pages artifact, not a gh-pages branch push The GitHub mirror's docs site kept going 404 despite a working CI run: confirmed via GitHub's own repo events that gh-pages gets deleted on every Forgejo mirror sync (same-second DeleteEvent alongside every mirrored push, and even on syncs with no new commits) — the mirror forces GitHub to exactly match Forgejo's origin, which never had that branch to begin with, since it's only ever created by CI running on GitHub itself. No fix on the push side can outrun that; the durable fix is to stop using a branch as the deploy target. docs/make.jl no longer calls deploydocs. The workflow now assembles the already-built Vitepress site (docs/build/1) into the dev/ layout DocumenterVitepress used to write via git, uploads it as a Pages artifact, and publishes with actions/deploy-pages — nothing left for the mirror to touch.

Add ADES/MPC export; measure and document photometric_outlier_threshold and build_reference's real performance - New ades_psv: formats an astrometric_calibrate candidate table as an ADES PSV observation table, the format MPC currently requires for astrometric submissions. Deliberately not the legacy 80-column format (see its docstring for why: that needs a real MPC-assigned designation to pack correctly, which this pipeline doesn't have). Each real tracklet's own id becomes ADES's trkSub (base-36 encoded), which is exactly the identifier ADES needs for undesignated objects. julian_date_to_iso8601 (JD -> ISO 8601 UTC, ADES's obsTime format) is verified against two independent, exactly known reference points (J2000.0, the Unix epoch), not just trusted from the algorithm. - photometric_outlier_threshold: measured directly against the one real, confirmed anomaly this project has (the field-451 likely-passing-cloud frame quality_max_std catches) rather than left as "never validated." Real result: that frame's raw-path photometric_scale differs from the field median by only 0.64%, far under any reasonable threshold -- a real negative result showing this check and quality_max_std are sensitive to different failure modes, not redundant safety nets. - build_reference: profiled directly rather than just described as slow. The per-pixel median-combine loop's real cost (a fresh array allocated per pixel) is fixed with a reusable buffer -- a real, measured 2x speedup with byte-identical output. But it was never the actual bottleneck: Reproject.reproject itself is ~24s/frame, two orders of magnitude more than the entire combine step. A multi-threaded reprojection attempt (Threads.@threads across frames, reasoned safe since Reproject.jl itself has no shared state) segfaulted on a real run: WCS.jl's pix_to_world! wraps wcslib via ccall, and concurrent calls into it aren't safe. Reverted, documented directly in build_reference's own docstring so the same mistake isn't repeated. Full narrative and real measured numbers for all three in docs/src/design-refinements.md.

Correct the .nojekyll fix: it needs to be at gh-pages' real root, not inside the per-version build dir The previous commit touched .nojekyll into docs/build/<n>/, reasoning that directory was "the site root" — it isn't; deploydocs copies it into a nested dev/ subfolder of the actual gh-pages branch, with versions.js/index.html generated separately at the true root, which no local directory corresponds to. Confirmed by listing the live branch's root via the GitHub API after the wrong version had already deployed: dev/, index.html, versions.js — no .nojekyll anywhere. Fixed by adding .nojekyll directly to gh-pages' root via the API instead (already done, this commit just documents it and removes the ineffective build-time code) — a one-time fix, not build-time logic, since deploydocs' own push only touches specific known paths on each run rather than wiping the branch, so a root file added once persists. Also documents the actual root cause of the GitHub Pages outage this was found alongside: Pages was never enabled at the repository level (Source stuck on "None"), independent of gh-pages existing with a real, working build — the green "success" on every workflow run was correctly reporting the deploy step's own success, not whether Pages was serving anything.

Add .nojekyll to each deployed docs version, working around a disabled line in DocumenterVitepress DocumenterVitepress has the code for this (writer.jl) but it's commented out in the installed version (0.3.5) — confirmed by reading the package source. Nothing was actually broken by its absence (the live site verified working via gh api/curl after fixing the real cause: GitHub Pages was never actually enabled, Source was stuck on "None" despite gh-pages existing with real content), but GitHub's default Jekyll processing on a branch-based Pages deploy silently ignores any underscore-prefixed path, which is exactly the kind of thing a future Vitepress internal-asset convention could introduce.

Reorganize the wiki: split investigation-log into topic pages, add real-data figures, force dark theme - docs/src/index.md restructured: status summary table, admonition boxes for Known limitations, real-data validation grouped into its own section with ZTF/variable-star/IASC subsections. - Investigation Log split from one large page into four: the original ZTF field 451 page, plus new variable-star-validation.md, iasc-campaign-validation.md, and design-refinements.md — all linked from the sidebar. Every existing anchor link across src/*.jl docstrings and index.md updated to the new locations. - docs/make_figures.jl (CairoMakie) generates two real figures from this session's actual measured numbers — the systematic_error_fraction sweep (false-positive rate vs. real-variable sensitivity) and the IASC match_radius retuning (tracklet counts per field, before/after) — run automatically before makedocs on every build, local or CI, so they can't drift from the numbers in the text. - docs/src/.vitepress/config.mts: custom config (appearance: 'force-dark') so the wiki is dark-only, no toggle — verified against VitePress's own source that this actually suppresses the switch component, not just guessed from docs. - docs/Project.toml gains CairoMakie so CI's Pkg.instantiate() picks it up automatically; no workflow YAML changes needed.

Improve two intentional-design limitations: relax estimate_psf's isolation filter before falling back; warn when zogy_subtract's V_ast is silently skipped - estimate_psf: on an empty stamp list, halve min_separation and retry (relaxation_attempts, default 2) before falling back to the analytic Moffat fit — a real empirical PSF from fewer, closer stars still beats a parametric approximation, which the previous hard binary threw away. - zogy_subtract: warn (once per session) when both n_sources/r_sources are left unset, so omitting V_ast is a visible choice for a direct caller instead of a silent default they could miss. run_pipeline always supplies both, so this never fires on the normal pipeline path. Both are real, tested improvements to items previously documented as "intentional design, not a bug" — the design itself was still the right call in each case, but neither was as good as it could be within that design. Full narrative in docs/src/investigation-log.md, including a real fit_moffat_psf stamp-overlap limitation surfaced while writing the regression tests.

Fix sub-pixel centroiding, auto min_frames, batched cross-matches; validate against real variable-star and IASC campaign data - detect_sources: refine PeakMesh's integer peak to a sub-pixel flux-weighted centroid before centering the aperture (returned x/y stay the original integers, so nothing position-dependent changes). - run_pipeline/search_field: min_frames/variability_min_frames now auto-adjust for quality_max_std-gated frames instead of silently making every tracklet unreachable. - crossmatch_catalog(:vsx/:simbad): batch OR-chained TAP queries (ceil(N/50) requests instead of N); retry once on the HTTP.ParseError confirmed to be a transient connection-reuse quirk, not a real outage. - crossmatch_catalog(:skybot): concurrent requests (measured ~8x speedup) plus a retry on transient HTTP errors, replacing a fully sequential loop that died to a connection error on a real multi-hour run. - find_variable_sources: systematic_error_fraction default raised 0.01 -> 0.02 after sweeping it against both real stationary stars (false-positive side) and a real confirmed variable (sensitivity side) — 2% is the largest floor checked that doesn't cost real detections. - load_wcs: fixed a real wcslib bug on Pan-STARRS1 headers where legacy CNPIX1/CNPIX2 keywords make wcslib build a degenerate implicit WCS and fail the whole parse, even though the header's real WCS is valid; retries after stripping just those two keywords. New examples/variable_star_demo.jl validates search_field end to end against a real, independently-confirmed variable star (ASASSN-V J183620.31) for the first time this project has done so. New examples/iasc_demo.jl validates run_pipeline against 5 real IASC Pan-STARRS1 practice campaign fields (not just ZTF), recovering 9 distinct known objects; also required a BLANK-sentinel pixel cleanup step and a match_radius retuned to the survey's own real astrometric precision (PERROR in the headers) instead of a value carried over from ZTF's pixel scale. Full narrative for all of the above in docs/src/investigation-log.md.

Make README.md minimal, pointing to the docs site; promote its former content to docs/src/index.md README.md was carrying the full Purpose/Status/Known limitations/examples content, duplicating what the docs site should be for — now that the site is actually live (richard7987.github.io/AsteroidPipeline.jl/dev/), that content moves there permanently (docs/src/index.md, a real committed page, same pattern already used for the investigation log) instead of being copied in from README.md at build time. README.md becomes a short landing page: badges, one-paragraph description, a prominent link to the docs site, install instructions, and a short contributing note — everything else is one click away instead of duplicated. Two stray "see README" comments (src/variables.jl, examples/real_data_demo.jl) updated to point at the docs site instead, since that's where the content actually is now.

Fix wrong GitHub username in docs/make.jl (ale-bnes -> Richard7987) The real GitHub mirror is Richard7987/AsteroidPipeline.jl. The wrong username almost certainly made deploydocs' own repo-match safety check silently skip the actual gh-pages push on the first real CI run (job reported success, but no gh-pages branch was created).

Fix CI: julia compat claimed 1.10 but Statistics/Printf compat require 1.11+ Statistics = "1.11.1" and Printf = "1.11.0" are stdlibs whose version tracks the Julia release they ship with — 1.11.x only exists in Julia 1.11+, so the existing julia = "1.10" compat bound was already inconsistent with the package's own other constraints, just never exercised since local development has used Julia 1.12 all along. Surfaced by the real CI run (Statistics unsatisfiable against Julia 1.10.12). Also removes an unjustified version: '1.10' pin I'd added to .github/workflows/Documenter.yml's setup-julia step, not present in DocumenterVitepress.jl's own reference workflow, and bumps setup-julia/cache to the same major versions (v3) that reference workflow uses.

Make INVESTIGATION_LOG.md a permanent docs page; add direnv support INVESTIGATION_LOG.md moves to docs/src/investigation-log.md as a real, committed page instead of being copied there from the repo root at build time — it's the docs site's own content now, not duplicated. README.md's links and docs/make.jl updated accordingly; .gitignore no longer treats that path as a build artifact. Also: .envrc (direnv + nix-direnv, loads flake.nix's devShell — JULIA_PROJECT, julia-bin — automatically on cd, approved via `direnv allow`); examples/.zed/settings.json suppresses julia.lint.missingrefs for that directory, since LanguageServer.jl's static analysis doesn't resolve a package's own self-referential `using` in a standalone script outside its module tree (confirmed a false positive: examples/real_data_demo.jl runs correctly, same environment, all session); .direnv/ (nix-direnv's local build cache) gitignored.

Set up a Documenter.jl + DocumenterVitepress.jl docs site with GitHub Actions deploy docs/make.jl copies README.md and INVESTIGATION_LOG.md into docs/src/ at build time (gitignored, never hand-edited) so the site can't drift out of sync with them; API reference pages are organized by source file, mirroring the module's own structure, via @autodocs. Deploys to gh-pages on push to main and on tags through .github/workflows/Documenter.yml, mirroring DocumenterVitepress.jl's own real deploy workflow. Building the docs surfaced two real broken @ref cross-references that had never been checked before (crossmatch_catalog's docstring linking to two undocumented internal helpers, light_curve's linking to an external Photometry.jl function Documenter has no binding for) and a VitePress rendering bug: a '%' in a section heading (auto-slugified into an anchor link) broke VitePress's URI decoder. All three fixed. Full docs build verified locally end to end (Julia + Vitepress), not just that make.jl runs without a Julia-side error.

Fix detect_sources's transposed aperture bug; migrate cross-match off X-Match; recalibrate variable-source detection against real data detect_sources built CircularAperture(row.x, row.y, r) directly from PeakMesh's output, but Photometry.jl's CircularAperture indexes the array in the opposite (x=1st dim, y=2nd) sense internally from PeakMesh's own (x=2nd dim, y=1st) convention -- every measured flux was silently wrong (~140x too small in a clean test case) except when a source's row and column happened to coincide. Positions were never affected, so every prior position-based result (tracklet linking, WCS calibration, the real ZTF baseline/ZOGY tracklet counts) stands unchanged; only flux/flux_err were wrong, since nothing depended on them quantitatively until find_variable_sources. Found and fixed by comparing measured flux against the analytic enclosed-energy integral for an isolated synthetic source. That fix invalidated the real-data numbers find_variable_sources's constants were originally calibrated against (a false ~29% median flux_err/flux, a false ~2-of-120 S/N pass rate, a false selection-bias explanation for a photometric-scale mismatch that's actually ordinary seeing-dependent aperture correction) -- re-measured everything against the same real ZTF frames post-fix and rewrote photometric_scale's and find_variable_sources's docstrings and chi2_threshold default (3 -> 50) against the corrected measurements, including the honestly-reported residual false-positive rate that remains. Also: crossmatch_catalog's :vsx/:simbad backend moved from the CDS X-Match service (extended, total outages) to direct SIMBAD/VizieR TAP queries; detect_sources gained an optional gain parameter for Poisson-aware flux_err; photometric_scale added as a raw-path quality signal in run_pipeline, honestly documented as unvalidated against a known-real anomaly (unlike quality_max_std). Full test suite green (118 pass, 1 broken -- the live plate_solve round-trip, which needs an API key not set in this environment); real ZTF demo re-run post-fix, confirming identical baseline/ZOGY tracklet counts (133/667, both recovering the same 2 known objects) since the fix never touched detection positions.