tender-db

Ingestion

The importer runs inside the server (ADR-0005): one job at a time, the readers serving throughout. Operators drive it through the /admin API.

Idle — no job running.

Recent runs (20)
JobParamsOutcomeResult
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
registry-contiguityregistry-contiguityokregistry-contiguity (issue 395): 3 source(s) checked, every monthly period sequence is CONTIGUOUS. Nothing is written. A duplicate period is reported per source but is not a hole — TED re-issued its 2025-09 monthly and the dedup held all but one member of ~73,000, which is the system working.
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
repair-member-twinsrepair-member-twins dry-runokrepair-member-twins (issue 404, DRY): 281 twin set(s) planned — keep the OLDEST row of each and drop 281 twin(s), of which 281 caused a Tender version and 0 never folded. 278 set(s) fold into ONE Tender, 3 span more than one; 281 survivor(s) take a new identity. Removing those versions moves 32 head pointer(s) and leaves 3 Tender(s) with no version at all, which the wet arm RETIRES rather than deleting piecemeal. The head moves are harmless for THIS cohort and only for it: twins hold the same bytes, so every current_* column but current_seq is unchanged. NOTHING IS WRITTEN — the plan is stored as `member-twin-repair`, and the wet arm refuses unless a fresh plan matches it set for set.
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
repair-member-twinsrepair-member-twins dry-runokrepair-member-twins (issue 404, DRY): 281 twin set(s) planned — keep the OLDEST row of each and drop 281 twin(s), of which 281 caused a Tender version and 0 never folded. 278 set(s) fold into ONE Tender, 3 span more than one; 281 survivor(s) take a new identity. Removing those versions moves 32 head pointer(s) and leaves 3 Tender(s) with no version at all, which the wet arm would RETIRE rather than delete piecemeal. The head moves are harmless for THIS cohort and only for it: twins hold the same bytes, so every current_* column but current_seq is unchanged. NOTHING IS WRITTEN — the plan is stored as `member-twin-repair` and the wet arm is not built yet (issue 404).
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
member-twin-censusmember-twin-censusokmember-twin-census (issue 404): 14508563 row(s) walked across 3 source(s) in 147 batch(es) of 100000, 388 member name(s) held more than once, of which 281 are ONE payload under 2+ publication_ids, covering 562 notice row(s). By source: [doe 281 set(s)/562 row(s)]. NOTHING IS WRITTEN. A duplicate NAME is not a defect — publishers reuse member names across packages and different bytes are different records; the twin count is the one that matters, and 281 was only ever the floor of one DÖE incident. The ingest path can no longer create this shape (issue 411): a moved identity is now adopted in place, on the row the corpus already hangs off.
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
data-qualitydata-qualityokdata quality measured: 24 eras over 35 windows in 5371s; 0 label(s) unmeasured ()
reindexauto: tender_version_result_winners_org_tenderokdeferred org + tender + notice indexes rebuilt
reveal-recheckreveal recheck (daily slice)okreveal recheck: 299156 withheld field(s) in corpus; slice 25535052..26788048: 100004 section(s), 1029 due, of 1029 checked 54 revealed at head, 600 awaiting a later version, 375 BROKEN (later version still withholds)
projectrebuild=falseok5053 notices → 4890 tenders (527 islands), 8436 versions; 4890 tenders written, 0 verified unchanged
fetch-ratesecb eurofxref-hist (daily)okrates: 220774 daily rows upserted from 2026-09-17 (Fetched, 639294 bytes) + 21 irrevocable conversion rates seeded; 0 stale rows reconciled away; 277503 rows cached
processfts daily (all)ok3831 members → 446 notices (446 parsed, 0 quarantined, 0 unrecognised, 3385 dup)
probefts daily (probe)okprobed 1 day(s), 1 new
processdoe daily (all)ok46019 members → 1184 notices (1184 parsed, 0 quarantined, 0 unrecognised, 44835 dup)
probedoe daily (probe)okprobed 1 day(s), 1 new

Contents

tenders
8 539 865
tenders (island)
688 933
tender_versions
14 508 091
lots
13 928 209
lot_results
18 179 978
bids
7 268 892
contracts
4 975 998
organizations
6 850 736
organizations (canonical)
1 137 475
organizations (provisional)
5 713 261
organization_mentions
48 417 281
changes
624 967 317

System

Fetching and processing are separate stages, so they go stale separately.

service revision
ba9eb1f53bfb18f8100875b7c66c9a54aa2c69d0
change cursor
624 967 317
newest fetched package
17 h ago
newest ingested notice
17 h ago

Quarantine

TERMINAL STATE EXCEEDED (issue 303): unclaimed-content — a reason grew past its documented ledger; investigate before it accretes.

A notice with content no profile maps is held whole, never partly imported (ADR-0004). The headline counts only confirmed real-notice loss; two large buckets are suspected parser gaps under investigation, and a small remainder is benign non-notice members (issue 30).

328

actionable — real notices held whole

+ 0 suspected parser gap (investigate-then-fix, issues 35/36)

+ 8 benign (non-notice members) — of 336 still held

Of 2 419 529 ever held: 336 still held, 1 817 555 reclaimed (reprocessed into the corpus), 601 638 skipped as duplicates of a sibling already held. Reclaimed and skipped are NOT the same claim — a skipped member never entered the corpus and was never meant to.

ReasonClassWhat it meansCount
unclaimed-contentactionableContent no mapping profile claims — held whole rather than partly imported.2
unreadable zip bundle: invalid Zip archive: Could not find EOCDbenignA corrupt archive entry — the bundle or member cannot be opened at all, so it was never a notice to map. Held whole as evidence (issues 201/202).8
unrepresentable-valueactionableA value did not fit its expected type (a malformed amount, date, or code).326
Most recent 50
ReasonProfileMember
unrepresentable-valueeforms:eforms-sdk-1.1320260917_180/00640904_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260713_132/00480047_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260710_131/00477652_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260710_131/00476401_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260710_131/00476676_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260709_130/00475103_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260706_127/00462758_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260706_127/00464135_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260716_135/00492248_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1206/20250610_2025109.tar.gz/20250610_109/00369509_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250610_2025109.tar.gz/20250610_109/00369371_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1206/20250604_2025106.tar.gz/20250604_106/00360008_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1206/20250605_2025107.tar.gz/20250605_107/00363704_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1206/20250603_2025105.tar.gz/20250603_105/00357830_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250603_2025105.tar.gz/20250603_105/00356035_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1206/20250627_2025121.tar.gz/20250627_121/00415376_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250616_2025113.tar.gz/20250616_113/00387147_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250630_2025122.tar.gz/20250630_122/00419269_2025.xml
unclaimed-contenteforms:eforms-sdk-1.1306/20250613_2025112.tar.gz/20250613_112/00381774_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250613_2025112.tar.gz/20250613_112/00382242_2025.xml
unclaimed-contenteforms:eforms-sdk-1.1306/20250626_2025120.tar.gz/20250626_120/00412845_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1306/20250606_2025108.tar.gz/20250606_108/00368703_2025.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260910_175/00625952_2026.xml
unrepresentable-valueeforms:eforms-de-2.1ce892970-bc90-4a12-9438-7ac11ea247fe-01.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260903_170/00608816_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260903_170/00607048_2026.xml
unrepresentable-valueeforms:eforms-de-2.1f96871e5-3e7a-4fab-8a18-ada74e0891c3-01.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260901_168/00602012_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260825_163/00587020_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260825_163/00586705_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260821_161/00579615_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260817_157/00568977_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260813_155/00563141_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260811_153/00554310_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260804_148/00540004_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260803_147/00534290_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1220260729_144/00524869_2026.xml
unrepresentable-valueeforms:eforms-sdk-1.1320260723_140/00509964_2026.xml
unrepresentable-valueeforms:eforms-de-2.1019ef9aa-833e-42fc-a57f-d41743324239-01.xml
unrepresentable-valueeforms:eforms-de-2.17e72f639-f8b3-4eb3-8d79-6fdaf7cfed96-01.xml
unrepresentable-valueeforms:eforms-de-2.19ee39409-1f44-4300-92e9-573fd81a50de-01.xml
unrepresentable-valueeforms:eforms-de-2.1805ac13f-81c7-4688-b521-d0252d977014-01.xml
unrepresentable-valueeforms:eforms-de-2.17cb4beae-5392-4abe-9adc-c6d7619e2596-01.xml
unrepresentable-valueeforms:eforms-de-2.1a511567a-3902-457d-bf94-b526edf14abe-01.xml
unrepresentable-valueeforms:eforms-de-2.115669b35-dcf9-46b6-81e3-1a41a20f2b19-01.xml
unrepresentable-valueeforms:eforms-de-2.0bbf79407-0cc7-43bb-bad3-6d7ebfa4beb4-01.xml
unrepresentable-valueeforms:eforms-de-2.023115380-a7f0-4377-8389-00897b1a61c8-01.xml
unrepresentable-valueeforms:eforms-de-2.00332a9ae-fcb3-439a-97b6-b69f99548346-01.xml
unrepresentable-valueeforms:eforms-de-2.0c1dba5c9-a510-4d64-914d-7bf081bf4ba9-01.xml
unrepresentable-valueeforms:eforms-de-2.01fb55ff8-d678-4497-9cc3-8acd26118754-01.xml

Resolved categories

Gaps we diagnosed and fixed. As the archive reprocesses, the held count falls to zero — the record stays here: what the gap was, the change that closed it, and how many notices have come back.

CategoryDiagnosisFixReclaimedResolved
Form copies diverging from their originalThe r209/r208 overlay walked secondary form copies as translations that must replicate the primary original's structure, rejecting the member whole on any divergence - 94 members. 92 were bilingual buyers' CO-ORIGINALS (Belgian/Bolzano style) whose second original legitimately carries more: extra organisations, F14 change blocks, lots. The last 2 were TED pipeline defects where the ORIGINAL was the impoverished copy - an ES award notice carrying one award where all 23 translations carry three, an F14 whose original has no CHANGE block at all. Every firing of the guard was recoverable content, so secondary copies now ADOPT missing sections (fully emitted - nothing in them is shared with the primary) and the rejection reason is retired.issue 201 (eb05139 + follow-up)942026-08-14
2010 records split by quoted reference numbersThe 2010-era structured forms quote the buyer's file reference under 'IV.3.1) reference number' - '3.131/2010', '331.08/01' - alone on an INDENTED line, which matched the <digits>.<digits>/<digits> record-marker shape: each such line split a real notice mid-body and the ND-less tail held as missing-publication-id (110 rows, every one a 2010 member; the bucket was never anything else). Anchoring the marker at column 0 (1,488/1,488 real markers are column-0 in a full 2010 daily; the only 2 indented matches were both false) merges the tails back: the fuller records entered as versions of their publication ids, and the reason is now empty.issue 200 (ac484f5)1082026-08-14
eForms selection criteria inside AppealTermsA 2022 DOE tool on eforms-sdk-1.0 (9 PINs) writes the SelectionCriteria extension block inside the procedure TenderingTerms' cac:AppealTerms - one mount deeper than any SDK declares. Grafted from the lot TenderingTerms anchor, the same family as the TenderingProcess mount.issue 195 (2d3202f)92026-08-14
DOE 0.1 subcontracting value leavesThe pre-release result model publishes cac:TenderResult/cac:SubcontractTerms with cbc:Amount (10 members, the BT-553 shape) or cbc:Rate (2, the BT-555 share) beside the conditions code the empirical inventory already knew. Inventory additions - sdk-0.1 has no SDK artifact, so observed paths ARE its inventory.issue 195 (2d3202f)122026-08-14
DOE 0.1 party registration idsPartyLegalEntity/cbc:CompanyID under three party mounts the empirical inventory lacked (lot AppealReceiverParty, MediationParty, AdditionalInformationParty) - one member advanced from mount to mount across re-parses, the issue-87 rewrite naming each precisely. Inventory additions.issue 195 (2d3202f)22026-08-14
Additional procurement types under unlisted listsThe SDK enumerates ProcurementAdditionalType per listName, so a code under any other list matched no branch and held the member whole: a 2023 eSender's listName="supplies" carrying a contract-nature value (7, sdk-1.8) and BT-775's dead SDK-1.0 social-procurement shape still published in 2024 (3, sdk-1.10). A predicate-free gap-filled branch claims them as published; declared lists keep their BT ids. Deliberately reverses the issue-144 negative control - publisher noise on old minors is not a mapping-decision signal.issue 195 (be4fcb1)102026-08-14
FieldsPrivacy hoisted to the root extensionFour sdk-1.9 members write withheld-field blocks (pro-fea BT-88, pro-typ BT-105, awa-cri-typ BT-539) directly under the root EformsExtension instead of beside the element each suppresses. Grafted from the two anchored families; the published FieldIdentifierCode predicate keeps BT-195..198 ids exact.issue 195 (8a82813)42026-08-14
Framework expected-participant countTwo sdk-1.10 members publish cbc:ExpectedOperatorQuantity beside the declared BT-113 maximum - no SDK minor or dialect declares the element. Claimed as published (UBL-ExpectedOperatorQuantity); BT-113 keeps its id.issue 195 (8a82813)22026-08-14
A service provider's own service providerTwo sdk-1.12 members nest a second, self-referential ServiceProviderParty level under the contracting party (the eSender registering itself as its own provider); the SDK models one level. The declared level grafts onto the nested mount.issue 195 (8a82813)22026-08-14
CVD flag on an award criterionTwo sdk-1.12 members mount efac:StrategicProcurement (ApplicableLegalBasis, BT-717's leaf) inside the AwardingCriterion extension beside the declared parameter blocks. Grafted from the lot TenderingTerms anchor. Rider: OPT-999 - UBL 2.3's forced DUMMY TenderResult/AwardDate - joined the zoneless-reads-as-UTC relaxation, which one of these members advanced into.issue 195 (8a82813)22026-08-14
EU-scheme business registration on 1.8An sdk-1.8 BRIN registers its PartyLegalEntity with schemeName='EU' - SDK <=1.8 declares only the 'national' BT-500 branch (the -European twin enters at 1.9), and the AdditionalDocumentReference block without its cbc:ID (OPP-124 enters later). Both later-SDK shapes claimed gap-filled on the early minors.issue 195 (cd0fb1c)12026-08-14
BT-707 published before its SDKA Romanian sdk-1.6 eSender publishes the lot documents-justification code that enters the vendored line at 1.7.0. Claimed gap-filled with the declared 1.7+ shape.issue 195 (cd0fb1c)12026-08-14
The organisation inlined under ContractingPartyA Norwegian sdk-1.7 eSender inlines the contracting organisation in full - name, address, website, contact person - under cac:ContractingParty/cac:Party, where the SDK models only the register reference, and fills UBL 2.3's forced TenderResult block with real data (tender count, low/high amounts, zoneless start date, winner reference). Company graft plus UBL- claims, all as published.issue 195 (cd0fb1c)12026-08-14
Notice-subtype free textAn Italian sdk-1.12 eSender writes efbc:SubTypeDescription beside the subtype code - declared by no minor or dialect. Claimed as UBL-SubTypeDescription.issue 195 (cd0fb1c)12026-08-14
Inline mediation body (DOE uuid channel)An sdk-1.0 uuid-channel member inlines the mediation body as a full party at BOTH procedure and lot level; the Company graft gains both targets.issue 195 (cd0fb1c)12026-08-14
eForms CVD statistics at the Lot2025 dailies (sdk 1.10-1.13) publish the Clean Vehicles Directive statistics block - BT-723 AssetCategoryCode plus the OPT-155/OPT-156 StatisticsCode/Numeric pairs under StrategicProcurementInformation/ProcurementDetails - inside the LOT's TenderingTerms extension, forward-looking at notice time, while the SDK declares that block only at LotResult. 81 contract-award notices were held whole for the unclaimed lot-level block. An ALIASES graft mirrors the LotResult declarations onto the lot TenderingTerms mount (the mirror of the pre-existing Lot-to-LotResult graft), and the population reclaimed.issue 195 (342f11e)812026-08-14
eForms selection criteria under TenderingProcessPublishers on sdk 1.8-1.13 mount the lot SelectionCriteria extension block (BT-40/747/748/749/750/752 family) under TenderingProcess where the SDK mounts it under TenderingTerms; the sampled publisher duplicates the identical block at BOTH mounts, so the stored duplication is source content, kept as published. 75 members reclaimed via a TenderingTerms-to-TenderingProcess alias; the same graft drained 2 sibling rows held on the nested AwardingCriterionTypeCode and 1 on the nested UBLExtensions wrapper (their keys are not separable from still-open classes, so they ride this entry's story).issue 195 (342f11e)752026-08-14
eForms framework maximum at the root84 sdk-1.7 members publish efbc:FrameworkMaximumAmount (BT-271, the framework's ceiling value) directly under the ROOT UBLExtensions/EformsExtension, a mount no SDK minor declares - the field belongs in the lot TenderingTerms extension. Claimed at the root position as published; the tender layer's value routing already prefers the declared positions when both exist.issue 195 (a12422f)842026-08-14
Design-contest shortlist under TenderingProcess11 sdk-1.7 French design-contest notices publish the EconomicOperatorShortList (BT-47 pre-selected participants) under TenderingProcess, beside the declared shortlist quantities, instead of under TenderingTerms; this publisher also fills the party NAMES with selection-criteria prose. Both quirks are stored as published (the notice-parsed layer keeps the source's own terms): a TenderingTerms-to-TenderingProcess graft claims the block at the sibling mount.issue 195 (c5351fc)112026-08-14
Part-level realized-location descriptions7 sdk-1.7 Latvian PINs describe the delivery location (RealizedLocation/Address/Description) on planned PARTS. A procedure-level claim for the element existed, and procedure fields alias onto Lots, and Lot fields alias onto Parts - but aliases do not COMPOSE, so the procedure entry never reached Part depth and the members held. The entry is now also written at Lot level, which the existing Lot-to-Part/LotsGroup aliases mirror.issue 195 (c5351fc)72026-08-14
Lot renewals flag from the DE dialect5 sdk-1.7 German members declare the plain EU customization but carry eForms-DE tailoring: ContractExtension/RenewalsIndicator on the lot (the dialect's BT-58 shape). Claimed as UBL-RenewalsIndicator behind the predicate-free-construct guard, so members declaring a DE customization keep their own DE field ids and only the cross-dialect strays land here.issue 195 (c5351fc)52026-08-14
2008 concession summaries (INTERNAL_OJS)7 INTERNAL_OJS R2.0.5 members publish concession awards under CONTRACT_CONCESSIONAIRE_SUM element names the internal-ojs rule registry lacked (the summary twin of the concessionaire family it already knew). Aliases added from the mined member shapes; the population reclaimed with the 194 defence-form waves.issue 194 (69d869f)72026-08-13
Text-era CF companions (UTF-8)The text era ships CF<n> companion files republishing the day's ORG records (plus stragglers the ORG missed). Unrecognised by the member naming, CF fell through to the XML path; the UTF-8-encoded CF members died there on the first text token. 1,900 held members - all real notice records. Recognising the CF delivery class (and the 1999 C<nn> spelling) sent them through the text parser: genuinely-new records reclaimed, the rest confirmed already held via the same day's ORG.issue 181 (9772c3c, 8dd7ebb)89 · 1 811 duplicates skipped2026-08-13
Text-era CF meta files ('part' root)753 meta_cf members wrap CF bookkeeping in an XML-ish document whose root element is a bare 'part' - no notice content. Held as unknown-root since ingestion; now recognised as part of the CF delivery class and skipped under the meta-variant policy, the same disposition as the other text-era meta members.issue 181 (9772c3c)0 · 753 duplicates skipped2026-08-13
1999/text-era ISO members held as not-utf8ISO-8859 members (CF companions and early-1999 deliveries) fell past the text dispatcher into the XML path, whose UTF-8 check rejects their bytes wholesale - a profile-level hold with no detail. 1,657 members: English mains and companions reclaimed (the CF drain alone recovered 1,417 notices the ORG files never carried and confirmed ~135k records already held), non-English language siblings skipped by policy as duplicates of the reclaimed English originals.issues 181 + 180 (9772c3c, 8dd7ebb)54 · 1 603 duplicates skipped2026-08-14
1999 COR corrected re-issuesThe 1999-07-10 and 1999-07-22 dailies were re-delivered whole as ISO_COR members - each a corrected re-issue of the day's ORG delivery (623 records with an identical ND set, of which 4 records byte-differ: the actual corrections). Held as not-utf8 like their era-mates. Dispatched as a companion-style class: the 4 corrected records entered as notice VERSIONS of their originals, the ~620 unchanged records deduped as already parsed; English rows reclaimed, sibling languages skipped.issue 180 (bd7eda5)2 · 20 duplicates skipped2026-08-14
Monthly 2026-06 inner dailies recorded wholeFetch 2 (ted monthly 2026-06) was processed 2026-07-19 with the day-one walker, one day before nested-monthly descent landed (1e4df1c) - its 21 inner daily tarballs were recorded as raw binary MEMBERS, one not-utf8 row per whole 06/<day>.tar.gz. Once the walker descended containers those paths never surfaced as members again, so the rows were unreachable by any reprocess. Fixed by dispatching members INSIDE a held container and stamping through a container address: 78,313 records deduped against the daily fetches, 3 notices the dailies never carried were recovered into the corpus, and 164 sub-cent records joined the unrepresentable-value kept class under their own precise rows.issue 196 (fe8979a)212026-08-14
2008 language siblings (duplicates)TED's 2008 monthly packages ship each notice once per language, all with a DTD the parser declines. The non-English members are duplicates of an English original we already hold and parsed, so they are not lost data and were never a coverage gap - they are marked skipped-by-policy rather than reclaimed, because reclaiming them would claim ~593k notices entered the corpus that never did. Measured: 592,856 markable, verified against the English original being present AND parsed for every one.issue 84 / #290 · 593 010 duplicates skipped2026-08-05
2008 siblings whose original did not parse154 of the 2008 held siblings had an English original that was held but did NOT parse, so the duplicate argument could not be applied to them: they were tracked as their own OUTSTANDING finding, excluded from the marker by its parsed-original guard (and briefly swept by a guard-free flag pass, repaired same day - issue 190). RESOLVED 2026-08-13: their 7 English originals were the concession summaries issue 194 fixed (six missing _SUM aliases); with the originals parsed and folded, the guard accepts these 154 and the marker recorded them skipped-by-policy - the truthful terminal state, reached through the guard rather than around it.issue 84 / #29 · guard: issue 190 · originals parsed: issue 1940 · 593 010 duplicates skipped2026-08-13
2010-03 DTD failures (not siblings)1,898 rows from a single fetch - TED monthly 2010-03 - that are not language siblings at all: one daily (2010-03-10, OJS edition 048) was converted by TED to per-notice TED_EXPORT XML behind an inline DOCTYPE, the only such day in the text era. The dispatcher's XXE-safe DTD strip let them dispatch as notices, but the deep parse fed raw bytes to the parser and re-quarantined every one - the strip is now applied on the deep-parse path too, and the whole day is reclaimed into the tender layer. These were real notices held whole, though never a coverage hole: 2010 is a text-era year the text profile already covers at ~99.5%, so the reclaim is largely language-twin content and honest bookkeeping. Found only by attributing the remainder: every earlier query was scoped to 2008, and that scoping hid them.issue 84 (found) · issue 139 (resolved: r209 deep-parse DTD strip, 145277a)1 8982026-08-12
r208 award @REASONAnnex-D negotiated-procedure awards publish their justification as a @REASON attribute no rule claimed, so the whole award was held; REASON is now a captured attribute.issue 31 · 5858159242026-07-21
text RP co-financingThe RP authority code was scalar, but international-financing records list each co-financier on a continuation line; RP is now per-line, so every institution is kept.issue 31 · 58581594 8612026-07-21
text OC/ON object CPVThe 1995-98 text vintages carry the main object CPV as OC (paired with the English description ON), which no rule claimed, quarantining the whole notice; OC is now claimed as CPV codes and ON as its description.issue 35576 7432026-07-21
2008 OPOCE INTERNAL_OJS (DTD)The 2008 OPOCE export is real S-series notices behind a DTD (INTERNAL_OJS R2.0.5) that roxmltree refused wholesale; the DTD is now safely stripped. The era ships each notice ONCE PER LANGUAGE, so the ~593k that remain held are the ~22 non-English duplicate siblings of the ~27k English notices, which are themselves present and parsed — verified row by row, 5,000 of 5,000 sampled, on two snapshots 29h apart. These are duplicates already ingested, NOT missing data, and no reprocess can move them: dispatch skips a non-English sibling by the one-notice-per-language policy, so it yields no record to reclaim. Their outstanding count therefore overstated the remaining work; issue 84 marked them skipped-by-policy, and since issue 186 this row is keyed to the English originals (the reclaimed ~27k plus the 7 still held), while the sibling rows carry the ~593k story under their own key.issue 36 · issue 8426 9482026-07-21
2008 English originals under the DTD label7 rows in the 2008 DTD population were the English originals themselves (member paths carry the .en language code) - the very rows the sibling guard exists to protect from being marked duplicates of themselves. They stayed outstanding under the DTD label until the 2026-08-12 reprocess relabeled them to their true failure (unclaimed CONTRACT_CONCESSIONAIRE_SUM), and issue 194 then mined the six missing internal-ojs _SUM aliases from the real payloads: all 7 parse and are folded into the tender layer (2026-08-13). Their 154 language siblings followed them into guard-accepted skipped-by-policy, closing the 2008 DTD story end to end.issue 84 (protected) · issue 194 (parsed: _SUM aliases)26 9482026-08-13
2008 OPOCE INTERNAL_OJS (parser)The INTERNAL_OJS R2.0.5 era (~28k opoce-only S-series notices, the whole 2008 coverage gap) had no mapping profile and was held as a tracked unmapped-era. It is now parsed by the internal-ojs profile: a new envelope reader for the BIB_INFO/BIB_DOC_S coded backbone plus an explicit _SUM alias shim that reuses the r209 walker for the 85%-shared form body. Reprocessing the bucket reclaims the era into notices.issue 4102026-07-23
eForms SDK 1.0-1.11 (unvendored)eForms notices declaring an SDK version (1.0-1.11) whose fields.json we had not vendored were held whole as unknown-customization - by far the largest bucket. All versions are now vendored (1.8-1.11 directly; 1.0-1.7 also needed an xpath grammar extension for descendant-axis and boolean-or predicates plus a predicate-free BT-165 binding), and the reprocess mechanism reclaims each held notice from the archive in place. The recovery rebuild (ADR-0009) folds them into the tender layer. The post-vendoring re-parse residue (71,707 rows) was then driven down in four diagnosed waves - the BT-803 transmission stamp (issue 141), OPT-060 on pre-1.7 minors and the bare ProcessJustification (issues 141/142), the 63K tail's five wild constructs (issue 143), and the six mechanical value-era fixes (issue 144) - leaving a tail whose remaining causes are value-domain, not structural. That tail carries stale structural reasons pending the final relabel pass and keep-policy decision, issue 184.issues 71/74/76; re-parse waves 140-144 (d7ba5ec, c6a7382, 0dfe428, 825dd64); tail owned by issue 184980 2942026-07-29
sub-cent amounts (kept by policy)Notices whose published amounts carry more than two fraction digits - sub-cent precision the integer-cents representation cannot hold exactly (ADR-0010 confirmed the representation against exactly this population; rounding was rejected because it would introduce values the source never published, issue 131's line). These are real notices deliberately kept in quarantine: coverage loss by decision, not by defect. They are the dominant share of the reclaim campaign's terminal tail - the 2026-08-11 relabel pass (issues 87/184) drained the stale unknown-customization labels into true current reasons: 1,661 of 1,907 landed here, alongside 206 unclaimed-content, 23 duplicate-section-id, 14 ambiguous-field and 3 genuinely-unknown sdk-1.2 rows. With that pass the campaign is complete in issue 144's sense: everything structural is fixed and reclaimed, and what remains is either owned by this policy or small, precisely named residues. Reopen via claim-and-store (ADR-0010's named alternative) if this population ever grows past a nuisance.ADR-0010 · issues 144/1843 2492026-08-11
eForms-DE 1.x (German dialect)German eForms-DE 1.0/1.1/1.2 notices were held as unknown-customization: the national 1.x line is spec-and-schematron only, with no published field metadata to parse against. The inventory was rebuilt empirically from the archived corpus (issue 75), with grafts for the DÖE serializer's inlined parties (issue 78), then the cohort was reclaimed and projected. These tenders now carry their German title and description, CPV, NUTS, lots, deadlines and estimated values, the buyer, and the other named organization roles — review body, information and document providers, tender recipient, evaluator, mediator. AWARD WINNERS ARE NOT RESOLVED for this dialect (issue 100): the result graph references sections by their published ids while those sections are keyed synthetically, so the LotResult → LotTender → TenderingParty chain does not link. Award notices therefore show their lots, contracts and awarded values, and bidders where a notice names them, but not which bidder won. That is a parse-layer defect: fixing it requires re-parsing the cohort, so it is deliberately not part of this batch. The residual ~241 that stayed held after this batch was re-run once issue 87 landed (failed reclaims now re-record their CURRENT failure): this key drained to zero outstanding, and the rows that still fail carry their true value-domain reasons under unrepresentable-value (issues 144/184).issues 75/78/76/85/98/99 — award winners pending, issue 100218 6382026-08-02
2011 US-spelled defence forms (VEAT et al.)The Sept-Dec 2011 dailies publish the defence-form family with US spelling (AWARD_OF_CONTRACT_DEFENSE, CONTRACT_DEFENSE, FD_ wrappers, and 21 descendants) while the mirrored XSDs - and the rule registry built from them - write DEFENCE. Every such notice was held whole as unclaimed-content: 2,380 VEAT (voluntary ex-ante transparency) notices plus ~84 contract/award/prior-information defence forms. 25 S-spelled twins now sit beside their DEFENCE twins in the registry (r208-observed inventory entries, mined from real members), and the population is reclaimed into the tender layer.issue 194 (b97decb + 31c4318)2 4622026-08-13
Text-era correction sheets (CS files)The 2000-2009 dailies ship per-language CS<n>.TXT members (4,386 CS1 + 44 CS2 + 11 CS3) holding plain-text ND:/FLD:/OLD:/NEW: records - field-level corrections to earlier notices (reference-number renumberings, country fixes, deadline moves), one tiny file duplicated across every language edition, never a notice body. Unrecognised by the text-era naming (no variant token before ~2004, .TXT extension in the member name), every one fell through to the XML path and quarantined as unparsable-xml. Now recognised as a text-era class and skipped by policy (text-era-correction-sheet); the correction content stays reachable in the raw archive - applying it to stored notices would be its own feature.issue 1800 · 4 441 duplicates skipped2026-08-14
Corrupt zip bundles in the TED archive (EOCD missing)Eight daily-delivery zip bundles across 1996-2009 are truncated in the archived TED distribution - the end-of-central-directory record is missing, so the bundle cannot be opened at all and is held whole. Seven are non-English deliveries (SV 1996-02-08, CS 2005-02-01 + 2005-04-01, ET 2005-06-22, NL 2005-10-06 + 2005-10-08, and the LV companion 2009-06-04) whose records the English-only ingestion policy takes from the English edition of the same days - their content is represented, nothing is lost. The eighth was the English UTF8 MAIN delivery for 2005-04-09, and its mere presence by name made the readable ISO twin skip itself as superseded - losing the whole day. Ledger-held unreadable bundles are now excluded from the supersedence decision, judged per publication day (a monthly carries many days, and one day's corrupt UTF8 must not sweep another day's ISO), and the day's 932 notices are ingested from the ISO copy. The eight rows stay held: they record the unreadable bytes themselves, whose content is covered elsewhere.issue 202 (b4a38c5 + b3f67b6)0 · 8 still held2026-08-14
Text-era column-0 wrap artifacts58 records across 1993-2010 held on a line at column 0 the parser could not attach. Extraction of every family showed TED's own line-wrapper emitting a wrapped tail flush-left - after a '!' inside the text ('nicht offnen ! ' then '" zu versehen...'), a closing parenthesis ('(EXCLUDING FOOTWEAR... OF WOOD' then ')'), a fixed-width mid-word sever ('TELECOMMUNICATI' / 'ON'), or an indent defect on per-line fields (RC: ES511 then 'ES512', RG: BARCELONA then 'GIRONA' - real second values) - plus the correction margin marker '!' and 2010 column-0 boilerplate and award justifications. All wrapped CONTENT, never structure, so a column-0 non-tag line now continues its open field (margin marker stripped; per-line lines are their own values; a scalar still rejects, and a line before any field still holds). The four 1994 mainframe search-command echoes ('.S F=ALL;R=NNNN TO 1;SORT=PD;ND;HC', between fields) are production residue, consumed as layout. All 58 reclaimed.issue 199 (bd6a5bf)4 9192026-08-14
F19 concession award-criteria sentenceThe F19 sub-contract concession form (defence, 2012) writes the award-criteria sentence as AWARD_CRITERIA_DETAIL's bare text where every other form nests CRITERIA/WEIGHTING children there - exactly 2 members in 30 years of corpus, the last unclaimed TED_EXPORT rows (issue 194's residue). The rule moved from Group to TextGroup so both shapes are consumed; both members reclaimed.issue 194 (23507fb)22026-08-14
Sub-cent amount precisionThe integer-cents policy refused any amount with more than two fraction digits, and that held the last FED quarantine bucket: 3,249 members of publisher mills (555.242), float-serialization artifacts (893513.4400000001) and deep trailing zeros, plus 1,574 pre-fix OPT-999 zoneless dates the issue-195 exception had already made parseable. cents() now rounds half-away-from-zero to the cent (error <= half a cent; the archived member stays byte-faithful, the canonical layer is a projection). Reprocess job 308 reclaimed 4,898 of 5,196; the 298 still held are the genuine garbage class (10^50-magnitude integers, e.g. BT-113 with fifty zeros) and hold by design.issue 2683 2492026-08-22

Award linkage

Legacy Tenders chain by transitive OJS references; a missed link strands an award as a single-notice Tender. Research predicts ≈17% unchained for the R2.0.9 era.

A high rate is only a defect where the source publishes a cross-reference to chain on. The German platform (eforms-sdk-0.1) reads near-100% because ~85% of its award notices publish an empty procedure identifier and carry no prior-notice reference at all — those awards are genuinely standalone records, so their rate is the honest number, not a lost link.

eraaward tendersunchainedrate
eforms:eforms-de-1.01900.0 %
eforms:eforms-de-1.165 489180.0 %
eforms:eforms-de-1.229 01300.0 %
eforms:eforms-de-2.033 79300.0 %
eforms:eforms-de-2.140 716110.0 %
eforms:eforms-sdk-0.1142 736139 98598.1 %
eforms:eforms-sdk-1.071371199.7 %
eforms:eforms-sdk-1.1071 71316 64823.2 %
eforms:eforms-sdk-1.1141 67612 80030.7 %
eforms:eforms-sdk-1.12105 19636 31534.5 %
eforms:eforms-sdk-1.13188 03553 73128.6 %
eforms:eforms-sdk-1.1420 6589 44945.7 %
eforms:eforms-sdk-1.32 19061428.0 %
eforms:eforms-sdk-1.618 1934 21923.2 %
eforms:eforms-sdk-1.792 34229 29531.7 %
eforms:eforms-sdk-1.862 74819 78131.5 %
eforms:eforms-sdk-1.937 80912 01331.8 %
fts:ocds-1.15 6265 15591.6 %
internal-ojs9 5123 76939.6 %
ted-export-r2081 013 676352 57734.8 %
ted-export-r2091 652 331333 85920.2 %
text1 134 607534 34347.1 %

Data quality

Weekly measurement, per era — measured 12 h ago · deltas vs the previous run

eraversionsshellsvaluenamedlinkageVAT statednegativeEUR conv
eforms:eforms-de-1.0310.0%0.0%100.0%
eforms:eforms-de-1.1145 8590.0%30.0%93.5%100.0%0.0%0.0%100.0%
eforms:eforms-de-1.272 9860.0%40.5%97.4%100.0%0.0%0.1%100.0%
eforms:eforms-de-2.089 3290.0%36.1%100.0%100.0%0.0%0.2%100.0%
eforms:eforms-de-2.1149 4780.0%43.5%100.0%100.0%0.0%0.1%100.0%
eforms:eforms-sdk-0.1676 2870.0%0.6%99.9%1.9%0.0%0.0%100.0%
eforms:eforms-sdk-1.03 5460.0%1.2%100.0%0.3%0.0%0.0%100.0%
eforms:eforms-sdk-1.10262 1750.0%62.9%95.9%76.8%0.0%0.7%99.8%
eforms:eforms-sdk-1.11110 9950.1%62.4%96.5%69.3%0.0%0.1%99.5%
eforms:eforms-sdk-1.12407 4710.0%64.3%99.9%65.5%0.0%0.7%99.4%
eforms:eforms-sdk-1.13841 5520.0%70.9%100.0%71.4%0.0%0.3%99.8%
eforms:eforms-sdk-1.14118 4090.1%62.1%100.0%54.3% ▲0.20.0%0.0%96.7%
eforms:eforms-sdk-1.230.0%0.0%
eforms:eforms-sdk-1.34 8340.0%77.3%100.0%72.0%0.0%0.7%99.3%
eforms:eforms-sdk-1.5350.0%14.3%0.0%0.0%100.0%
eforms:eforms-sdk-1.637 6480.0%66.0%99.7%76.8%0.0%0.0%100.0%
eforms:eforms-sdk-1.7344 5050.0%55.8%96.3%68.3%0.0%0.1%100.0%
eforms:eforms-sdk-1.8140 8200.0%80.0%100.0%68.5%0.0%0.0%100.0%
eforms:eforms-sdk-1.987 4100.0%70.1%91.7%68.2%0.0%0.1%100.0%
fts:ocds-1.111 0460.0%35.3% ▼0.28.4% ▲0.2100.0%0.0%97.1%
internal-ojs26 9550.0%24.3%99.9%60.4%55.5%0.0%100.0%
ted-export-r2082 699 2130.0%43.5%99.8%65.2%15.9%0.0%99.7%
ted-export-r2094 490 5490.0%65.8%99.9%79.8%0.0%0.0%99.9%
text3 786 9550.0%11.0%100.0%52.9%77.5%0.0%100.0%

Pipeline

Where each source is in the import — fetched packages, then notices processed out of them, then Tenders projected. A source whose packages are exchange rates rather than notices has no notices and no Tenders to count (issue 401): its last two cells read — and its Fetched cell says how current the rate table is instead.

SourcePublishedFetchedProcessedProjected
doe103 pkgs (daily 2026-07-17 … 2026-09-16 · monthly 2022-12 … 2026-07) · fetch complete ✓1 137 900673 815
ecb22 pkgs (2026-08-27 … 2026-09-17) · rates through 2026-09-16
eurostat1 pkgs (rates-ecu-bil 1993-1998 … 1993-1998 · rates-ecu-h 1993-1998 … 1993-1998) · rates through 2026-09-16
fts11 pkgs (daily 2026-09-07 … 2026-09-16 · monthly 2025-06 … 2025-06) · fetch complete ✓11 0469 610
ted13 201 520459 pkgs (daily 2026-00124 … 2026-00180 · monthly 1993-01 … 2026-06) · fetch complete ✓ · re-issued: 2025-0913 359 4737 856 440

Coverage

Coverage is notices held ÷ notices that year is known to have published (docs/research/ted-access-channels.md §6): 100 % means we hold the whole year. One collapsible row per source and profile era — expand for the per-year breakdown. A dash means no ground-truth denominator exists (any source but TED, or a year outside the reference counts).

During the historical backfill these ratios climb: the importer works through the eras in the background and in learn-order, not calendar-order, so an old year can sit well below 100 % simply because its packages have not been processed yet. A low ratio here is usually work still in progress — but not always, which is why the Pipeline panel above now names any monthly package the fetch registry is missing (issue 395). A year whose package is named there is a permanent gap until that package is fetched, not a backfill in flight.

ted · text3 786 955 held · 3 802 937 published † · 100.34 %
YearHeldPublishedCoverage
2010389 599391 397100.03 %†
2009363 236363 230100.00 %
2008313 059339 534100.14 %†
2007307 248307 255100.00 %
2006268 081268 060100.01 %
2005249 454249 437100.01 %
2004222 616221 786100.37 %
2003224 192224 144100.02 %
2002202 685202 684100.00 %
2001172 201172 194100.00 %
2000161 237161 228100.01 %
1999162 887162 861100.02 %
1998163 395160 892101.56 %
1997155 185152 339101.87 %
1996140 576138 533101.47 %
1995127 772126 385101.10 %
199496 34094 457101.99 %
199367 19266 521101.01 %
ted · internal-ojs26 955 held · 339 534 published † · 100.14 %
YearHeldPublishedCoverage
200826 955339 534100.14 %†
ted · ted-export-r2082 699 213 held · 8 421 040 published † · 100.00 %
YearHeldPublishedCoverage
2024256801 444100.00 %†
20234 357795 680100.00 %†
20224 470735 067100.00 %†
20214 444676 734100.00 %†
20205 903643 552100.00 %†
20198 388622 786100.00 %†
201859 725578 501100.00 %†
2017117 951528 975100.00 %†
2016312 027466 898100.00 %†
2015463 614463 821100.00 %†
2014446 416446 419100.00 %
2013443 078443 079100.00 %
2012414 836414 837100.00 %
2011411 850411 850100.00 %
20101 898391 397100.03 %†
ted · ted-export-r2094 490 549 held · 6 313 458 published † · 100.00 %
YearHeldPublishedCoverage
202428 667801 444100.00 %†
2023722 082795 680100.00 %†
2022730 587735 067100.00 %†
2021672 289676 734100.00 %†
2020637 649643 552100.00 %†
2019614 398622 786100.00 %†
2018518 776578 501100.00 %†
2017411 024528 975100.00 %†
2016154 871466 898100.00 %†
2015206463 821100.00 %†
doe · eforms:eforms-sdk-0.1676 287 held · — published · —
YearHeldPublishedCoverage
202686 308
2025140 703
2024141 796
2023185 820
2022121 660
doe · eforms:eforms-sdk-1.03 546 held · — published · —
YearHeldPublishedCoverage
2026438
2025633
2024742
20231 446
2022287
ted · eforms:eforms-sdk-1.23 held · 735 067 published † · 100.00 %
YearHeldPublishedCoverage
20223735 067100.00 %†
ted · eforms:eforms-sdk-1.34 834 held · 2 332 191 published † · 100.00 %
YearHeldPublishedCoverage
2024380801 444100.00 %†
20234 447795 680100.00 %†
20227735 067100.00 %†
doe · eforms:eforms-de-1.031 held · — published · —
YearHeldPublishedCoverage
202412
202319
doe · eforms:eforms-de-1.1145 859 held · — published · —
YearHeldPublishedCoverage
202511 308
2024113 583
202320 968
ted · eforms:eforms-sdk-1.10262 103 held · 2 966 064 published † · 104.89 % *
YearHeldPublishedCoverage
20261497 791 ‡129.15 % *†
202593 812871 149100.00 %†
2024164 868801 444100.00 %†
20233 422795 680100.00 %†
ted · eforms:eforms-sdk-1.535 held · 1 597 124 published † · 100.00 %
YearHeldPublishedCoverage
202412801 444100.00 %†
202323795 680100.00 %†
ted · eforms:eforms-sdk-1.637 669 held · 1 597 124 published † · 100.00 %
YearHeldPublishedCoverage
202432 559801 444100.00 %†
20235 110795 680100.00 %†
ted · eforms:eforms-sdk-1.7344 518 held · 2 966 064 published † · 104.89 % *
YearHeldPublishedCoverage
202611497 791 ‡129.15 % *†
202539 648871 149100.00 %†
2024260 166801 444100.00 %†
202344 693795 680100.00 %†
ted · eforms:eforms-sdk-1.8140 834 held · 2 468 273 published † · 100.00 %
YearHeldPublishedCoverage
202520 051871 149100.00 %†
2024113 763801 444100.00 %†
20237 020795 680100.00 %†
ted · eforms:eforms-sdk-1.987 438 held · 2 468 273 published † · 100.00 %
YearHeldPublishedCoverage
202514 110871 149100.00 %†
202468 802801 444100.00 %†
20234 526795 680100.00 %†
doe · eforms:eforms-de-1.272 986 held · — published · —
YearHeldPublishedCoverage
202530 662
202442 324
doe · eforms:eforms-de-2.089 338 held · — published · —
YearHeldPublishedCoverage
202617 119
202569 245
20242 974
doe · eforms:eforms-sdk-1.10102 held · — published · —
YearHeldPublishedCoverage
2024102
doe · eforms:eforms-sdk-1.1287 held · — published · —
YearHeldPublishedCoverage
202568
202419
ted · eforms:eforms-sdk-1.11111 018 held · 2 170 384 published † · 106.68 % *
YearHeldPublishedCoverage
202610497 791 ‡129.15 % *†
202538 646871 149100.00 %†
202472 362801 444100.00 %†
ted · eforms:eforms-sdk-1.12407 470 held · 2 170 384 published † · 106.68 % *
YearHeldPublishedCoverage
2026109 340497 791 ‡129.15 % *†
2025238 521871 149100.00 %†
202459 609801 444100.00 %†
doe · eforms:eforms-de-2.1149 487 held · — published · —
YearHeldPublishedCoverage
202699 526
202549 961
doe · eforms:eforms-sdk-1.13164 held · — published · —
YearHeldPublishedCoverage
202616
2025148
fts · fts:ocds-1.111 046 held · — published · —
YearHeldPublishedCoverage
20263 803
20257 243
ted · eforms:eforms-sdk-1.13841 483 held · 1 368 940 published † · 110.60 % *
YearHeldPublishedCoverage
2026415 121497 791 ‡129.15 % *†
2025426 362871 149100.00 %†
doe · eforms:eforms-sdk-1.1413 held · — published · —
YearHeldPublishedCoverage
202613
ted · eforms:eforms-sdk-1.14118 396 held · 497 791 published † · 129.15 % *
YearHeldPublishedCoverage
2026118 396497 791 ‡129.15 % *†

* the year is not over, and its published count is a snapshot (‡, hover for its date): a shortfall is the calendar, and a figure above 100 % is the corpus having grown past the snapshot — neither is a gap, and neither is a duplicate.

† the year is served by more than one profile (an era boundary), so the figure is the whole year across all of them — there is no ground truth for one profile's share of a year. On a collapsed era row the same mark means the era contains such a year, so its percentage is the coverage of the years it SPANS rather than of what it holds: the held count beside it is still the era's own.