commit - 25619044a16363a72873b015ccb438393734b447
commit + 716323a0aef2b85a996850c7dc399cb57a0e979a
blob - 75626c03ffee8b288ce67d92dffde952d81e7cab
blob + c29d9e2f32b246d5f916e654ca09cd2ee294589a
--- docs/src/design-refinements.md
+++ docs/src/design-refinements.md
split onto its own page since these aren't bugs found on one specific
real dataset the way the other pages here are: two are design decisions
revisited to see if they could be improved without abandoning the choice
-behind them, and two are validation/performance investigations that
+behind them, and the rest are validation/performance investigations that
don't belong to any single dataset-specific page.
## `estimate_psf`'s empirical-vs-fallback split was a hard binary
accepting that cost, which is why it isn't the target.
+
+## Looking for another `build_reference`-sized win — profiled, and mostly not found
+
+Asked directly whether anything else in the pipeline was worth
+parallelizing or otherwise speeding up, the way `build_reference`'s
+reprojection loop was. Profiled the real stages of `_detect_all_frames`,
+`find_variable_sources`, `photometric_scale`, and `link_candidates`
+directly (`@time`, not guessed) against the V1012 Mon dataset above —
+useful for this specifically because its field is unusually dense
+(~200-580 detections/frame even at `threshold=100`, denser than field
+451's own ~130), making it a real stress case rather than a typical one.
+
+**Most stages are already fast.** `find_variable_sources` (0.20s) and
+`photometric_scale` (0.08s) across all 34 frames were not remotely close
+to being a bottleneck, despite an initial worry that their per-star
+position-matching loop might scale badly with detection count.
+`detect_sources` itself: 0.06s for a single dense frame (523 sources).
+
+**One real, safe fix found: `_detect_all_frames`'s `detections_per_frame`
+was an untyped `Vector{Any}`.** `detect_sources` always returns the exact
+same concrete `TypedTables.Table` type on every call — confirmed
+directly (`typeof(...)` compared for both its empty and non-empty return
+paths, not assumed from reading the source alone) — so there was no
+reason for the container holding those results to be `Any`-typed. Fixed
+by typing it explicitly from that same concrete type. Measured before
+and after on this same dense dataset: no significant wall-clock change
+here (within noise, ~5.4-5.6s either way) — `_detect_all_frames`'s real
+cost is FITS I/O and `detect_sources`'s own internal allocations, not
+this container's typing. Kept anyway as a real, zero-risk type-stability
+correctness improvement (every downstream consumer of
+`detections_per_frame` — `find_variable_sources`, `link_candidates`,
+`photometric_scale` — now sees a concretely-typed vector instead of
+`Any`), not reported as a performance win it didn't measurably produce.
+
+**`link_candidates`'s real cost on this dense field (0.57s, 52.6% GC
+time) is a known, already-mitigated design tradeoff, not a hidden
+bug.** Its seed-tracklet search is a pairwise loop over every
+frame-1/frame-2 detection pair — confirmed directly: 523 × 365 = 190,895
+pairs checked here, of which only 658 survive the `max_speed` filter and
+go on to the more expensive per-candidate work (a linear
+`_closest_detection` scan across every other frame, plus a
+least-squares refit that itself allocates three small temporary arrays
+per candidate). On a typical field (like field 451's own ~130
+detections/frame) the pair count drops by roughly (130/523)² ≈ 6%,
+putting this well under 50ms — this 0.57s figure is specific to an
+unusually dense field, and `threshold` (already tuned up for exactly
+this kind of dense field in both this dataset and the earlier field-487
+case) is the pipeline's existing, deliberate answer to it, not something
+this profiling pass found reason to change.
+
+No second `build_reference`-sized win turned up. The pipeline's real
+remaining cost, at typical real field densities, is dominated by what
+was already found and fixed.
blob - fabf87a8f9d9999654220eeeff28551d25f27188
blob + bbca5cfa18bdb044701bc89fb0d6c7f62b8c8091
--- src/pipeline.jl
+++ src/pipeline.jl
`0` on the raw path, which has no such gate) — both callers use it to
auto-adjust their own `min_frames` default, since a gated frame otherwise
silently makes no tracklet/variable-candidate reachable at all.
+
+`detections_per_frame` is concretely typed from `detect_sources`'s own
+return type (confirmed identical on both its empty and non-empty return
+paths, not assumed) rather than left as `Vector{Any}` — a real
+type-stability fix found while profiling for other speedups, though it
+didn't measurably change this function's own real-data timing; see
+[Design refinements](https://richard7987.github.io/AsteroidPipeline.jl/dev/design-refinements).
"""
function _detect_all_frames(fits_paths::AbstractVector{<:AbstractString};
timestamp_key::AbstractString="MJD-OBS",
reference=nothing, psf_threshold::Real=20.0, psf_min_separation::Real=40.0,
quality_max_std::Real=1.5, plate_solve_api_key::Union{Nothing,AbstractString}=nothing,
photometric_outlier_threshold::Real=0.2)
- detections_per_frame = []
+ detections_per_frame = typeof(Table(x=Int[], y=Int[], peak=Float64[], flux=Float64[], flux_err=Float64[]))[]
wcs_per_frame = WCSTransform[]
timestamps = Float64[]
n_gated = 0
