commit 1acf0bc70592e4401c60ae2d4b010f082fea8205 from: ale date: Sat Aug 29 20:41:54 2026 UTC Add mpc80_report: legacy MPC 80-column format export IASC's coordinator confirmed (email reply) that reports must be in the 80-column format specifically, not ADES — which ades_psv's docstring had previously (and correctly, for MPC's own preference) assumed was the only format worth implementing. Column layout verified against the MPC's own published spec (OpticalObs.html, PackedDes.html), not guessed. Reuses the same observer-assigned temporary designation this pipeline's ades_psv trkSub already relies on for undesignated candidates (base-36 tracklet id), one character shorter (7, not 8) to fit the format's designation slot. Refactored julian_date_to_iso8601's Gregorian calendar math into a shared _jd_to_calendar, reused by the new MPC date field formatter. commit - 716323a0aef2b85a996850c7dc399cb57a0e979a commit + 1acf0bc70592e4401c60ae2d4b010f082fea8205 blob - 1f98f50c93f2b89fcac1d1d1a472cd732c6df9d8 blob + daa8fb74c95966efe409c4ca65d868adbdf91d1b --- docs/src/api/mpc-export.md +++ docs/src/api/mpc-export.md @@ -1,11 +1,13 @@ # MPC / ADES Export -Formats an [`astrometric_calibrate`](@ref) candidate table as an ADES -PSV observation table (`ades_psv`) — the format the Minor Planet Center -currently requires for astrometric submissions. Not the legacy 80-column -format; see `ades_psv`'s own docstring for why. +Formats an [`astrometric_calibrate`](@ref) candidate table as either an +ADES PSV observation table (`ades_psv`) — the format the Minor Planet +Center currently requires for astrometric submissions — or the legacy +fixed-width 80-column format (`mpc80_report`), still required by some +programs (e.g. IASC, as of 2026) even though the MPC's own submissions +now prefer ADES. -## Example +## ADES example ```julia ades_psv(candidates, "I41") @@ -24,6 +26,30 @@ per row, ready to write to a `.psv` file.) Both rows s `candidates` share the same `trkSub`, which is how the Minor Planet Center correlates them back into one tracklet. +## 80-column example + +```julia +mpc80_report(candidates, "I41") +``` + +produces (one fixed-width 80-character line per row, `·` standing in +here for a literal space so the column boundaries stay visible): + +``` +·····1·······C2000·01·01.50000010·00·29.630+20·59·15.56·····························I41 +·····1·······C2000·01·01.51000010·00·29.654+20·59·15.92·····························I41 +·····2·······C2000·01·01.50000013·22·00.000-10·15·00.00·····························I41 +``` + +Columns 1-5 (the permanent-number field) are always blank — this +pipeline's candidates are locally-numbered tracklets, never +MPC-numbered objects — and columns 6-12 carry the same `id`-derived +tracking label as ADES's `trkSub` (`1` for both rows above), just +capped one character shorter (7, not 8) and left-justified in a fixed +field instead of pipe-delimited. See `mpc80_report`'s own docstring for +the full column layout and the same real gaps `ades_psv` has (no +magnitude, no discovery asterisk). + ```@autodocs Modules = [AsteroidPipeline] Pages = ["mpc_export.jl"] blob - 9b3aec538c2e9ca78762449a69e516726f0626a7 blob + 17d1bcf1ff5643cd933349eb71333d684b8c8853 --- docs/src/index.md +++ docs/src/index.md @@ -48,8 +48,10 @@ periodogram applies directly to a `find_variable_sourc the format the Minor Planet Center currently requires for astrometric submissions — so a real discovery's candidates can go straight from `run_pipeline`'s output to a submittable file, `id`-per-tracklet mapped -directly to ADES's own `trkSub` tracking-designation field. Only ADES, -not the legacy 80-column format — see its docstring for why. +directly to ADES's own `trkSub` tracking-designation field. `mpc80_report` +does the same for the legacy fixed-width 80-column format, still +required by some programs (e.g. IASC, as of 2026) even though the MPC's +own submissions now prefer ADES. ## Status @@ -179,8 +181,13 @@ for the full story of each: #### Using real IASC campaign data Not yet attempted on a live campaign — practice data above is the -closest real-world test so far. General guidance for pointing this -pipeline at a real campaign, or any other survey's data: +closest real-world test so far. IASC's own coordinator confirmed +(2026-08) that testing an outside program during a real campaign is +possible, on the condition that submitted reports use the 80-column +format specifically — not just ADES, which the MPC itself now prefers +but this pipeline's own submission format (`mpc80_report`) covers as of +this confirmation. General guidance for pointing this pipeline at a +real campaign, or any other survey's data: - Point `run_pipeline` (or `examples/iasc_demo.jl`'s pattern) at the local file paths directly; no fetch script is needed for files you blob - 48cd9ba929eccd672e426582091e8323dc747ad5 blob + a3264f088ab0d4d2124bd3df972cdcd3cef28e6c --- src/AsteroidPipeline.jl +++ src/AsteroidPipeline.jl @@ -33,6 +33,6 @@ export detect_sources, link_candidates, load_wcs, pix_ crossmatch_catalog, run_pipeline, build_reference, load_frame, estimate_psf, fit_moffat_psf, zogy_subtract, light_curve, recover_rotation_period, plate_solve, search_field, find_variable_sources, variability_chi2, photometric_scale, - ades_psv, julian_date_to_iso8601 + ades_psv, mpc80_report, julian_date_to_iso8601 end # module AsteroidPipeline blob - 5349356777c3cdb1d0df0bae77ae5e2deb9291c0 blob + 4d79d5485a4a854d5c4a0a408338f5f44d9de777 --- src/mpc_export.jl +++ src/mpc_export.jl @@ -1,16 +1,16 @@ """ - julian_date_to_iso8601(jd::Real) -> String + _jd_to_calendar(jd::Real) -> (year, month, day, day_fraction) -Convert a Julian Date (UTC) to an ISO 8601 UTC timestamp -(`"YYYY-MM-DDTHH:MM:SS.sssZ"`), the format ADES requires for `obsTime`. - -Uses the standard Gregorian-calendar algorithm (Meeus, *Astronomical -Algorithms*, ch. 7) — verified here against two independent, exactly -known reference points, not just trusted from memory: `2451545.0` is -the J2000.0 epoch (2000-01-01T12:00:00Z) and `2440587.5` is the Unix -epoch (1970-01-01T00:00:00Z); both are exercised in the test suite. +Break a Julian Date (UTC) into its Gregorian calendar year/month/integer +day plus the leftover fraction of that day, via the standard algorithm +(Meeus, *Astronomical Algorithms*, ch. 7). Shared by +[`julian_date_to_iso8601`](@ref) and [`mpc80_report`](@ref)'s date +field, which need the same calendar date at different sub-day +precisions (milliseconds vs. 1e-6 day) — each rounds `day_fraction` to +its own precision itself, since rounding here would fix one caller's +precision at the expense of the other's. """ -function julian_date_to_iso8601(jd::Real) +function _jd_to_calendar(jd::Real) Z = floor(Int, jd + 0.5) F = (jd + 0.5) - Z @@ -31,11 +31,28 @@ function julian_date_to_iso8601(jd::Real) month = E < 14 ? E - 1 : E - 13 year = month > 2 ? C - 4716 : C - 4715 + return year, month, day, day_frac - day +end + +""" + julian_date_to_iso8601(jd::Real) -> String + +Convert a Julian Date (UTC) to an ISO 8601 UTC timestamp +(`"YYYY-MM-DDTHH:MM:SS.sssZ"`), the format ADES requires for `obsTime`. + +Verified here against two independent, exactly known reference points, +not just trusted from memory: `2451545.0` is the J2000.0 epoch +(2000-01-01T12:00:00Z) and `2440587.5` is the Unix epoch +(1970-01-01T00:00:00Z); both are exercised in the test suite. +""" +function julian_date_to_iso8601(jd::Real) + year, month, day, day_frac = _jd_to_calendar(jd) + # Round to the nearest millisecond, not truncate — otherwise a # fractional day landing at (e.g.) 23:59:59.9997 truncates to # 23:59:59.999 instead of correctly rolling to the next second. - ms_of_day = round(Int, (day_frac - day) * 86_400_000) - ms_of_day == 86_400_000 && (ms_of_day = 0; day += 1) # (only possible from rounding at the boundary) + ms_of_day = round(Int, day_frac * 86_400_000) + ms_of_day == 86_400_000 && (ms_of_day = 0; day += 1) # (only possible from rounding at the boundary; doesn't cascade into month/year — not worth the complexity for a sub-millisecond edge case that real ZTF timestamps won't hit) hour, rem1 = divrem(ms_of_day, 3_600_000) minute, rem2 = divrem(rem1, 60_000) second, ms = divrem(rem2, 1000) @@ -44,6 +61,63 @@ function julian_date_to_iso8601(jd::Real) end """ + _mpc_date(jd::Real) -> String + +Format a Julian Date (UTC) as the MPC 80-column format's +`"YYYY MM DD.dddddd"` date field (columns 16-32; 1e-6 day ≈ 0.0864s +precision). Unlike [`julian_date_to_iso8601`](@ref)'s millisecond +rollover, a rounding carry here is resolved by re-deriving the whole +calendar date at a nudged `jd` (via [`_jd_to_calendar`](@ref)) rather +than hand-incrementing `day`, so it stays correct across a month/year +boundary too. +""" +function _mpc_date(jd::Real) + year, month, day, day_frac = _jd_to_calendar(jd) + micros = round(Int, day_frac * 1_000_000) + if micros == 1_000_000 + year, month, day, day_frac = _jd_to_calendar(jd + 1e-6) + micros = round(Int, day_frac * 1_000_000) + end + return @sprintf("%04d %02d %02d.%06d", year, month, day, micros) +end + +""" + _mpc_ra(ra_deg::Real) -> String + +Format a right ascension in decimal degrees as the MPC 80-column +format's `"HH MM SS.ddd"` field (columns 33-44). Working in integer +milliseconds-of-time and reducing modulo a full day sidesteps the +usual manual carry-the-1 logic for seconds/minutes/hours rollover +entirely. +""" +function _mpc_ra(ra_deg::Real) + hours = mod(ra_deg, 360.0) / 15.0 + total_ms = mod(round(Int, hours * 3_600_000), 24 * 3_600_000) + hour, rem1 = divrem(total_ms, 3_600_000) + minute, rem2 = divrem(rem1, 60_000) + second, ms = divrem(rem2, 1000) + return @sprintf("%02d %02d %02d.%03d", hour, minute, second, ms) +end + +""" + _mpc_dec(dec_deg::Real) -> String + +Format a declination in decimal degrees as the MPC 80-column format's +`"sDD MM SS.dd"` field (columns 45-56; `s` is a mandatory `+`/`-`, per +the spec — MPC does not default a blank sign to positive). Clamped at +the pole rather than left to overflow into an invalid degrees field: +not expected on any real asteroid-search field, but cheap to guard. +""" +function _mpc_dec(dec_deg::Real) + sign_char = dec_deg < 0 ? '-' : '+' + total_cs = min(round(Int, abs(dec_deg) * 360_000), 90 * 360_000) + deg, rem1 = divrem(total_cs, 360_000) + minute, rem2 = divrem(rem1, 6_000) + second, cs = divrem(rem2, 100) + return @sprintf("%c%02d %02d %02d.%02d", sign_char, deg, minute, second, cs) +end + +""" ades_psv(candidates, station::AbstractString; mode::AbstractString="CCD", trksub_prefix::AbstractString="", astCat=nothing, photCat=nothing, band=nothing) -> String @@ -52,21 +126,16 @@ Format `candidates` (an [`astrometric_calibrate`](@ref `id`, `frame`, `x`, `y`, `ra`, `dec`, `epoch`) as an ADES PSV (pipe-separated values) observation table — the format the Minor Planet Center currently requires for astrometric submissions, superseding the -legacy fixed-width 80-column format. +legacy fixed-width 80-column format. See [`mpc80_report`](@ref) for the +80-column format itself, still required by some programs (e.g. IASC, as +of 2026) even though the MPC's own submissions now prefer ADES. -Only the legacy 80-column format's replacement (ADES) is implemented -here, not the 80-column format itself: 80-column records pack a -provisional designation into a specific fixed encoding that needs a -real MPC-assigned designation to round-trip correctly, which nothing in -this pipeline has (candidates are locally-numbered tracklets, not -MPC-designated objects) — guessing at that packing without real -reference examples to check against risked producing output that reads -as well-formed but is subtly wrong, exactly the kind of mistake that -matters for a real submission. ADES has no such requirement: new, -undesignated objects are identified by `trkSub`, an observer-chosen -tracking label (here, each real tracklet's own `id`, base-36 encoded to -stay compact and prefixed with `trksub_prefix` if given), which is -exactly what this pipeline already produces. +New, undesignated objects (this pipeline never produces MPC-designated +ones — candidates are locally-numbered tracklets) are identified by +`trkSub`, an observer-chosen tracking label (here, each real tracklet's +own `id`, base-36 encoded to stay compact and prefixed with +`trksub_prefix` if given), which is exactly what this pipeline already +produces. One row per detection point (i.e. one row per `candidates` row, not one per tracklet) — this is the granularity ADES observation records use; @@ -113,3 +182,67 @@ function ades_psv(candidates, station::AbstractString; return join(lines, "\n") * "\n" end + +""" + mpc80_report(candidates, station::AbstractString; note1::AbstractChar=' ', + note2::AbstractChar='C', trksub_prefix::AbstractString="") -> String + +Format `candidates` (an [`astrometric_calibrate`](@ref) table, same +shape [`ades_psv`](@ref) takes) as the legacy MPC1992 fixed-width +80-column format — verified column-by-column against the Minor Planet +Center's own published specification (`OpticalObs.html`, +`PackedDes.html`), not guessed: columns 1-5 packed permanent number, +6-12 packed provisional/temporary designation, 13 discovery asterisk, +14 note 1, 15 note 2, 16-32 date (`"YYYY MM DD.dddddd"`), 33-44 RA +(`"HH MM SS.ddd"`), 45-56 Dec (`"sDD MM SS.dd"`, sign mandatory — MPC +does *not* default a blank sign to positive), 57-65 blank, 66-71 +magnitude+band, 72-77 blank, 78-80 observatory code. + +Columns 1-5 are always blank: this pipeline's candidates are +locally-numbered tracklets, never MPC-numbered objects. Columns 6-12 +use the same real, spec-sanctioned escape hatch [`ades_psv`](@ref)'s +`trkSub` relies on: the spec requires *some* designation in columns +1-12 ("never leave \\[them\\] blank"), but for a brand-new, +not-yet-designated discovery it explicitly allows an +observer-assigned *temporary* designation in place of an MPC-packed +provisional one — alphanumeric only, max 7 characters here (one +shorter than ADES's 8-character `trkSub` limit, since the packed-number +field this shares a slot with doesn't get the eighth column ADES adds). +Built the same way as `ades_psv`'s `trkSub`: each tracklet's own `id`, +base-36 encoded and optionally prefixed via `trksub_prefix`. + +Three real gaps, same as `ades_psv`'s (deliberately not guessed at): +no discovery asterisk (column 13 always blank — this pipeline doesn't +track which observation of a tracklet was reported first), no +magnitude/band (columns 66-71 always blank — no photometric zeropoint +is calibrated against a reference catalog; a submission without +magnitudes is valid, same as for ADES), and `note1`/`note2` are fixed +per call rather than derived per detection. `note2` defaults to `'C'` +(CCD), matching how ZTF (and most modern digital-sensor surveys) +report; see the MPC spec for the full note2 code table if submitting +from a different observing mode. + +One row (one 80-character line, newline-terminated) per detection +point, same granularity as `ades_psv`. Returns the report content as a +`String`; write it to a file yourself +(e.g. `write("submission.txt", mpc80_report(candidates, "I41"))`). +""" +function mpc80_report(candidates, station::AbstractString; note1::AbstractChar=' ', + note2::AbstractChar='C', trksub_prefix::AbstractString="") + length(station) == 3 || throw(ArgumentError("station must be a 3-character MPC observatory code")) + + lines = String[] + for row in candidates + trksub = trksub_prefix * uppercase(string(row.id; base=36)) + length(trksub) <= 7 || throw(ArgumentError( + "temporary designation \"$trksub\" exceeds the 80-column format's 7-character limit; use a shorter trksub_prefix")) + + line = " "^5 * rpad(trksub, 7) * ' ' * note1 * note2 * + _mpc_date(row.epoch) * _mpc_ra(row.ra) * _mpc_dec(row.dec) * + " "^9 * " "^6 * " "^6 * station + @assert length(line) == 80 + push!(lines, line) + end + + return join(lines, "\n") * "\n" +end blob - 5869706d2f1115ed1a1067179ead0a4419f277ee blob + 951f8032889d36093149f46cd20a5de048b3b2a9 --- test/runtests.jl +++ test/runtests.jl @@ -1082,4 +1082,62 @@ end @test_throws ArgumentError ades_psv(candidates, "I41"; trksub_prefix="TOOLONGPREFIX") end + @testset "mpc80_report" begin + candidates = Table(id=[1, 1, 2], frame=[1, 2, 1], + x=[10.0, 12.0, 20.0], y=[10.0, 12.0, 20.0], + ra=[150.123456789, 150.123556789, 200.5], + dec=[20.987654321, 20.987754321, -10.25], + epoch=[2451545.0, 2451545.01, 2451545.0]) + + @test_throws ArgumentError mpc80_report(candidates, "XX") # not 3 characters + + report = mpc80_report(candidates, "I41") + # chomp, not strip: the leading 5 blank columns on line 1 (no + # permanent number) are real, fixed-width content, not + # incidental whitespace to trim. + lines = split(chomp(report), "\n") + @test length(lines) == 3 + @test all(l -> length(l) == 80, lines) # the whole point of the format + + line1 = lines[1] + @test line1[1:5] == " " # no permanent number + @test strip(line1[6:12]) == uppercase(string(1; base=36)) # temp designation from id=1 + @test line1[15] == 'C' # note2 default (CCD) + @test line1[78:80] == "I41" + + # date/RA/Dec: parse the formatted fields back to numbers and + # compare against the input, within the format's own precision + # — checks the actual round-trip rather than a hand-computed + # expected string. + date_field = line1[16:32] + @test date_field[1:4] == "2000" + @test date_field[6:7] == "01" + @test parse(Float64, date_field[9:end]) ≈ 1.5 atol=1e-6 # J2000.0 = Jan 1.5 + + ra_field = line1[33:44] + ra_deg = 15 * (parse(Float64, ra_field[1:2]) + parse(Float64, ra_field[4:5]) / 60 + + parse(Float64, ra_field[7:end]) / 3600) + # atol matches the field's own precision (0.001s of time ≈ + # 4.2e-6°), not tighter than what the format can represent + @test ra_deg ≈ 150.123456789 atol=1e-5 + + dec_field = line1[45:56] + dec_sign = dec_field[1] == '-' ? -1 : 1 + dec_deg = dec_sign * (parse(Float64, dec_field[2:3]) + parse(Float64, dec_field[5:6]) / 60 + + parse(Float64, dec_field[8:end]) / 3600) + @test dec_deg ≈ 20.987654321 atol=1e-5 + + # both rows sharing id=1 must share the same temp designation + @test strip(lines[2][6:12]) == strip(line1[6:12]) + @test strip(lines[3][6:12]) != strip(line1[6:12]) # id=2 gets a distinct one + + # sign is mandatory even for a negative declination (row 3, dec=-10.25) + @test lines[3][45] == '-' + + # temp designation length limit: mirrors ades_psv's trkSub cap, + # but one character tighter (7, not 8) since this format's + # designation slot has no eighth column to spare + @test_throws ArgumentError mpc80_report(candidates, "I41"; trksub_prefix="TOOLONG") + end + end