- Description:
- An open-source Julia pipeline for asteroid search campaigns, developed for use with the International Astronomical Search Collaboration (IASC).
- Last Change:
- Clone URL:
https://git.nezzontli.xyz/clone/asteroidpipeline.jl.git
Commit Briefs
Add digest2 NEO scoring and cross-night tracklet linking (main)
Investigated 4 candidate technologies for improving the pipeline; GPU reprojection and local plate-solving were rejected with real reasons (no GPU path in Reproject.jl/wcslib; multi-GB index files for marginal gain), documented in Design refinements. Built the other two: - digest2_score wraps the MPC's own external digest2 classifier (verified by actually cloning, building, and running it). Real verification against ZTF field 451 surfaced a genuinely useful, non-obvious finding: it correctly scored the field's 2 known Main Belt objects low on neo_score, while 131 of 133 tracklets scored 100 — bogus links of stationary stars, an artifact of that demo's own loose match_radius, not something digest2 got wrong. Written up in full on its own wiki page. - link_across_nights extends link_candidates' within-night linear linking across multiple observing nights (fit-and-extrapolate on sky coordinates, union-find grouping). Validated synthetically; a real multi-night same-object case was searched for (checked the existing 5-field IASC/PS1 dataset, then queried SkyBoT directly) but not found within reasonable effort — stated plainly as an open gap rather than skipped past. 188/188 tests pass (digest2_score's live test ran for real against a locally-built binary, not just skipped).
Add mpc80_report: legacy MPC 80-column format export
IASC's coordinator confirmed (email reply) that reports must be in the 80-column format specifically, not ADES — which ades_psv's docstring had previously (and correctly, for MPC's own preference) assumed was the only format worth implementing. Column layout verified against the MPC's own published spec (OpticalObs.html, PackedDes.html), not guessed. Reuses the same observer-assigned temporary designation this pipeline's ades_psv trkSub already relies on for undesignated candidates (base-36 tracklet id), one character shorter (7, not 8) to fit the format's designation slot. Refactored julian_date_to_iso8601's Gregorian calendar math into a shared _jd_to_calendar, reused by the new MPC date field formatter.
Profile the pipeline for another build_reference-sized win; mostly not found
Profiled detect_sources, _detect_all_frames, find_variable_sources, photometric_scale, and link_candidates directly on a real, unusually dense dataset (V1012 Mon field, ~200-580 detections/frame). Most stages are already fast (find_variable_sources 0.20s, photometric_scale 0.08s for 34 frames). link_candidates' real cost there (0.57s) is a known, already-mitigated design tradeoff (pairwise seed search scaling with detection density, already why threshold gets tuned up for dense fields), not a hidden bug. One real fix found: _detect_all_frames' detections_per_frame was an untyped Vector{Any}, even though detect_sources always returns the same concrete Table type (confirmed directly, not assumed). Typed it explicitly. Measured no significant wall-clock change from this alone — kept as a real, zero-risk type-stability improvement, not reported as a speedup it didn't produce. Full test suite passes unchanged (158/158).
Raise find_variable_sources' systematic_error_fraction default: 2% -> 3%
The 2% default rested on one real field (451) and one real confirmed variable from a different field. Found two more real, independently confirmed variables (V1012 Mon, ASASSN-V J072906.85-090518.2) via a real VSX/IRSA search for bright short-period eclipsing binaries with dense real same-night ZTF coverage, then repeated the exact same floor sweep against each: 197 and 182 real matched stationary stars respectively. At 3% (already a measured point in the original field-451 sweep), all three confirmed variables stay above chi2_threshold=10.0 (reduced chi2 13.8/61.2/17.5), while the false-positive rate is the same or better than 2% in every field (1.3%/2.04%/0.55% vs 2.0%/2.55%/0.55%). A literal 0% false-positive rate is reachable at a 5% floor, but only by also pushing two of the three real variables' own signal below threshold — not a real improvement. Full test suite passes unchanged (158/158) with the new default.
Fix the live plate_solve test's fixture: a single fake star can't solve
Testing the .env loader with a real ASTROMETRY_API_KEY for the first time actually ran this test live instead of skipping it — and it failed: astrometry.net job "failed to solve". Not a bug in plate_solve itself; the synthetic fixture (one Gaussian source in noise) has no real star pattern for the service's geometric matching to identify. Replaced it with the real ZTF frame already used for this exact live validation (see plate_solve's own docstring) — its real WCS is never read by plate_solve, which only uploads pixels, so this is a legitimate blind-solve test. Skips (distinctly from the missing-API-key case) if that real, gitignored frame isn't present locally. Verified against the live service with a real key: solves successfully, returning a WCSTransform with plausible sky coordinates. Full suite: 158 pass, 0 broken, 0 errors — first time this project has had a fully clean run with a real API key present.
Load ASTROMETRY_API_KEY from a gitignored .env for local test runs
The live plate_solve round-trip test was always skipped in practice — nothing set ASTROMETRY_API_KEY without manually exporting it by hand each session. test/runtests.jl now reads a repo-root .env (gitignored; .env.example documents the expected format) before the test suite runs, without overriding a real environment variable if one's already set. Not read by src/ itself — plate_solve takes api_key as an explicit argument, this is test-only convenience. Verified end to end: a real (intentionally invalid) key in .env made the test stop skipping and actually hit the live service, which correctly rejected it — confirms the loader, not just its parsing logic in isolation.
Wire build_reference's workers= into real_data_demo.jl
The reference-build step this script documents as "roughly half an hour" was still running sequentially — the new multi-process path existed but nothing used it. Spawns Sys.CPU_THREADS worker processes around the build_reference call, torn down in a finally block regardless of outcome. Verified end to end against real data: full script (baseline pipeline, worker spawn + parallel reference build, ZOGY, both SkyBoT crossmatches) now completes in ~11 minutes, and recovers the same known objects as before (133/2 baseline, 667/2 ZOGY) — the speedup didn't change results.
Update build_reference's benchmark numbers with the real, integrated result
The earlier 331.8s/2.2x came from a standalone script that reimplemented the WCS-header-string technique by hand, before build_reference's own workers keyword existed. Re-measured end to end on the actual function, same real 30-frame field-451 set, sequential and distributed back to back: 951.31s vs. 295.92s, 3.21x — better than the earlier estimate, and output (image/sigma/mask) confirmed exactly identical between the two paths.
Add a real CI workflow, fix README badge, remove deploy-plumbing from the wiki
There was no workflow that actually ran the test suite — Documenter.yml only builds/deploys docs. Added CI.yml (setup-julia + buildpkg + runtest) and pointed the README's badge at it instead. Also moved the GitHub Pages/mirror troubleshooting story out of design-refinements.md into a local, gitignored PAGES_NOTES.md: the wiki documents the pipeline itself, not its own CI plumbing. Retitled the page (it covers more than "two limitations" now) and fixed the now-stale link text pointing to it from investigation-log.md.
Branches
Tree
README.md
# AsteroidPipeline.jl
[](https://github.com/Richard7987/AsteroidPipeline.jl/actions/workflows/CI.yml)
[](https://richard7987.github.io/AsteroidPipeline.jl/dev/)
An open-source Julia pipeline for asteroid search campaigns, developed for
use with the [International Astronomical Search Collaboration
(IASC)](https://iasc.cosmosearch.org/). Detects moving and variable
sources across a sequence of FITS frames, optionally via ZOGY difference
imaging, and calibrates and cross-matches the result against known-object
catalogs (SkyBoT, VSX, SIMBAD).
**Full documentation, including project status, real-data validation
results, known limitations, and the complete API reference, is at
<https://richard7987.github.io/AsteroidPipeline.jl/dev/>.**
## Installation
```julia
using Pkg
Pkg.activate(".")
Pkg.instantiate()
```
A reproducible development environment is also provided via Nix
(`flake.nix`):
```
nix develop
```
## Contributing
`docs/make.jl` builds the documentation site
([Documenter.jl](https://github.com/JuliaDocs/Documenter.jl) +
[DocumenterVitepress.jl](https://github.com/LuxDL/DocumenterVitepress.jl));
build it locally with:
```
julia --project=docs docs/make.jl
```
`.github/workflows/Documenter.yml` builds and deploys it to GitHub Pages
on every push to `main`.
## License
MIT. See `LICENSE`.
