Commit Briefs

857b086fcd ale

Add digest2 NEO scoring and cross-night tracklet linking (main)

Investigated 4 candidate technologies for improving the pipeline; GPU reprojection and local plate-solving were rejected with real reasons (no GPU path in Reproject.jl/wcslib; multi-GB index files for marginal gain), documented in Design refinements. Built the other two: - digest2_score wraps the MPC's own external digest2 classifier (verified by actually cloning, building, and running it). Real verification against ZTF field 451 surfaced a genuinely useful, non-obvious finding: it correctly scored the field's 2 known Main Belt objects low on neo_score, while 131 of 133 tracklets scored 100 — bogus links of stationary stars, an artifact of that demo's own loose match_radius, not something digest2 got wrong. Written up in full on its own wiki page. - link_across_nights extends link_candidates' within-night linear linking across multiple observing nights (fit-and-extrapolate on sky coordinates, union-find grouping). Validated synthetically; a real multi-night same-object case was searched for (checked the existing 5-field IASC/PS1 dataset, then queried SkyBoT directly) but not found within reasonable effort — stated plainly as an open gap rather than skipped past. 188/188 tests pass (digest2_score's live test ran for real against a locally-built binary, not just skipped).


1acf0bc705 ale

Add mpc80_report: legacy MPC 80-column format export

IASC's coordinator confirmed (email reply) that reports must be in the 80-column format specifically, not ADES — which ades_psv's docstring had previously (and correctly, for MPC's own preference) assumed was the only format worth implementing. Column layout verified against the MPC's own published spec (OpticalObs.html, PackedDes.html), not guessed. Reuses the same observer-assigned temporary designation this pipeline's ades_psv trkSub already relies on for undesignated candidates (base-36 tracklet id), one character shorter (7, not 8) to fit the format's designation slot. Refactored julian_date_to_iso8601's Gregorian calendar math into a shared _jd_to_calendar, reused by the new MPC date field formatter.


716323a0ae ale

Profile the pipeline for another build_reference-sized win; mostly not found

Profiled detect_sources, _detect_all_frames, find_variable_sources, photometric_scale, and link_candidates directly on a real, unusually dense dataset (V1012 Mon field, ~200-580 detections/frame). Most stages are already fast (find_variable_sources 0.20s, photometric_scale 0.08s for 34 frames). link_candidates' real cost there (0.57s) is a known, already-mitigated design tradeoff (pairwise seed search scaling with detection density, already why threshold gets tuned up for dense fields), not a hidden bug. One real fix found: _detect_all_frames' detections_per_frame was an untyped Vector{Any}, even though detect_sources always returns the same concrete Table type (confirmed directly, not assumed). Typed it explicitly. Measured no significant wall-clock change from this alone — kept as a real, zero-risk type-stability improvement, not reported as a speedup it didn't produce. Full test suite passes unchanged (158/158).


25619044a1 ale

Raise find_variable_sources' systematic_error_fraction default: 2% -> 3%

The 2% default rested on one real field (451) and one real confirmed variable from a different field. Found two more real, independently confirmed variables (V1012 Mon, ASASSN-V J072906.85-090518.2) via a real VSX/IRSA search for bright short-period eclipsing binaries with dense real same-night ZTF coverage, then repeated the exact same floor sweep against each: 197 and 182 real matched stationary stars respectively. At 3% (already a measured point in the original field-451 sweep), all three confirmed variables stay above chi2_threshold=10.0 (reduced chi2 13.8/61.2/17.5), while the false-positive rate is the same or better than 2% in every field (1.3%/2.04%/0.55% vs 2.0%/2.55%/0.55%). A literal 0% false-positive rate is reachable at a 5% floor, but only by also pushing two of the three real variables' own signal below threshold — not a real improvement. Full test suite passes unchanged (158/158) with the new default.


75a8a7aa40 ale

Fix the live plate_solve test's fixture: a single fake star can't solve

Testing the .env loader with a real ASTROMETRY_API_KEY for the first time actually ran this test live instead of skipping it — and it failed: astrometry.net job "failed to solve". Not a bug in plate_solve itself; the synthetic fixture (one Gaussian source in noise) has no real star pattern for the service's geometric matching to identify. Replaced it with the real ZTF frame already used for this exact live validation (see plate_solve's own docstring) — its real WCS is never read by plate_solve, which only uploads pixels, so this is a legitimate blind-solve test. Skips (distinctly from the missing-API-key case) if that real, gitignored frame isn't present locally. Verified against the live service with a real key: solves successfully, returning a WCSTransform with plausible sky coordinates. Full suite: 158 pass, 0 broken, 0 errors — first time this project has had a fully clean run with a real API key present.


4d124c4c49 ale

Load ASTROMETRY_API_KEY from a gitignored .env for local test runs

The live plate_solve round-trip test was always skipped in practice — nothing set ASTROMETRY_API_KEY without manually exporting it by hand each session. test/runtests.jl now reads a repo-root .env (gitignored; .env.example documents the expected format) before the test suite runs, without overriding a real environment variable if one's already set. Not read by src/ itself — plate_solve takes api_key as an explicit argument, this is test-only convenience. Verified end to end: a real (intentionally invalid) key in .env made the test stop skipping and actually hit the live service, which correctly rejected it — confirms the loader, not just its parsing logic in isolation.


6610d9676c ale

Wire build_reference's workers= into real_data_demo.jl

The reference-build step this script documents as "roughly half an hour" was still running sequentially — the new multi-process path existed but nothing used it. Spawns Sys.CPU_THREADS worker processes around the build_reference call, torn down in a finally block regardless of outcome. Verified end to end against real data: full script (baseline pipeline, worker spawn + parallel reference build, ZOGY, both SkyBoT crossmatches) now completes in ~11 minutes, and recovers the same known objects as before (133/2 baseline, 667/2 ZOGY) — the speedup didn't change results.



5dd14c0e4f ale

Update build_reference's benchmark numbers with the real, integrated result

The earlier 331.8s/2.2x came from a standalone script that reimplemented the WCS-header-string technique by hand, before build_reference's own workers keyword existed. Re-measured end to end on the actual function, same real 30-frame field-451 set, sequential and distributed back to back: 951.31s vs. 295.92s, 3.21x — better than the earlier estimate, and output (image/sigma/mask) confirmed exactly identical between the two paths.


6af67de2a5 ale

Add a real CI workflow, fix README badge, remove deploy-plumbing from the wiki

There was no workflow that actually ran the test suite — Documenter.yml only builds/deploys docs. Added CI.yml (setup-julia + buildpkg + runtest) and pointed the README's badge at it instead. Also moved the GitHub Pages/mirror troubleshooting story out of design-refinements.md into a local, gitignored PAGES_NOTES.md: the wiki documents the pipeline itself, not its own CI plumbing. Retitled the page (it covers more than "two limitations" now) and fixed the now-stale link text pointing to it from investigation-log.md.


Branches

Tags

This repository contains no tags

Tree

.env.examplecommits | blame
.envrccommits | blame
.github/
.gitignorecommits | blame
LICENSEcommits | blame
Project.tomlcommits | blame
README.mdcommits | blame
docs/
examples/
flake.lockcommits | blame
flake.nixcommits | blame
src/
test/

README.md

# AsteroidPipeline.jl

[![CI](https://github.com/Richard7987/AsteroidPipeline.jl/actions/workflows/CI.yml/badge.svg)](https://github.com/Richard7987/AsteroidPipeline.jl/actions/workflows/CI.yml)
[![docs](https://img.shields.io/badge/docs-dev-blue.svg)](https://richard7987.github.io/AsteroidPipeline.jl/dev/)

An open-source Julia pipeline for asteroid search campaigns, developed for
use with the [International Astronomical Search Collaboration
(IASC)](https://iasc.cosmosearch.org/). Detects moving and variable
sources across a sequence of FITS frames, optionally via ZOGY difference
imaging, and calibrates and cross-matches the result against known-object
catalogs (SkyBoT, VSX, SIMBAD).

**Full documentation, including project status, real-data validation
results, known limitations, and the complete API reference, is at
<https://richard7987.github.io/AsteroidPipeline.jl/dev/>.**

## Installation

```julia
using Pkg
Pkg.activate(".")
Pkg.instantiate()
```

A reproducible development environment is also provided via Nix
(`flake.nix`):

```
nix develop
```

## Contributing

`docs/make.jl` builds the documentation site
([Documenter.jl](https://github.com/JuliaDocs/Documenter.jl) +
[DocumenterVitepress.jl](https://github.com/LuxDL/DocumenterVitepress.jl));
build it locally with:

```
julia --project=docs docs/make.jl
```

`.github/workflows/Documenter.yml` builds and deploys it to GitHub Pages
on every push to `main`.

## License

MIT. See `LICENSE`.