Research methodology

Sixteen sources feed daily into intel_jobs (the intelligence corpus, append-only) and jobs (the site-facing table). Each source row is gated by a 160-term ontology with multilingual broad-stem patterns covering 7 EU languages (DE / FR / ES / IT / NL / PT / EN) before persistence, so non-geospatial postings are rejected at the ingestion boundary rather than filtered downstream.

Slug discovery for ATS platforms (Greenhouse / Lever / Ashby / SmartRecruiters / Workable) runs weekly via .github/workflows/ats-discovery-weekly.yml. The candidate company list grows automatically; the daily collector picks up new viable boards without code changes.

• Title normalisation:212 regex patterns (English + multilingual) compiled at startup; ~40% of non-English titles now classify into role families (was <1% before the ESCO multilingual loader landed). ESCO 28-language label map is hydrated from data/intelligence/esco/esco-labels-multilingual.json.
• Skill extraction: regex pass via 509 canonical skills + 1,895 aliases, then an LLM gap fill pass for rows where regex returned 0 skills.
• Salary parsing: robust parser handling k / M suffixes, locale separators (EU 1.500 vs US 1,500 vs decimal 1.5), period inference from text and from magnitude, and per-period bounds validation. Replaced a previous parser that silently truncated $80k-$120k to 80-120 (1000× corruption).
• Geocoding: dictionary lookup → strict alias map (preserves Reading vs London, Rotterdam vs Amsterdam) → Nominatim fallback with cache and rate-limit. Method tracked in the geocode_method column for downstream precision filtering.
• Quality score: recomputed daily as a weighted sum of post-enrichment fields (title + company + description length + salary + geocoding + skill count + source authority). Tier A / B / C bands feed downstream marts.
• FX provenance: every salary_usd update writes a row to salary_normalisation_audit with the rate, rate_date, source, quality grade, and pipeline_run_id — so historical conversions are fully reproducible.

13 benchmark sources back the salary surfaces with citation-grade provenance. Every row in intel_salary_benchmarks carries a JSONB provenance object containing retrieval URL, retrieval date, dataset version, methodology URL, licence, and authority weight.

Two complementary collectors feed intel_papers:

• OpenAlex — concept-graph driven. 22 concept IDs covering core geospatial fields plus workforce / labour / pay-transparency. Pulls preferred labels in 7 languages.
• Crossref — bibliographic-text driven. 13 boolean queries explicitly target the workforce and EU Pay Transparency Directive literature. Cursor-paginated, polite-pool mailto.

DOIs are normalised on every write via a database trigger (doi_normalised): lowercase, with the leading https://doi.org/ prefix stripped. When the same normalised DOI appears under two distinct sources, the row is flagged cross_source_verified = true — a high-confidence signal for downstream consumers.

Daily snapshot. data_quality_snapshot captures volume, coverage, freshness, stale-source alerts, and six dimension scores (D1-D6) per day. Citable in every release; surfaced publicly at /research/quality.

Mart parity gate. CI test asserts that public.mart_* views and public_analytics.mart_* tables match on row counts and column counts; fails the workflow on schema drift.

Quarantine layer. stg_intel_jobs_rejected classifies every dropped row by reason (no_title, no_company, low_source, no_description, no_geo, no_salary, composite_low) so quality loss is observable instead of silent.

RPC hardening. 23 SECURITY DEFINER functions in public have explicit search_path, REVOKE PUBLIC and GRANT-only-to-needed-roles. Documented in migration 20260505130000.

• Source code: every collector, normaliser, and dbt model is in the public geospatial-careers repository.
• Open dataset: a GeoParquet snapshot of mart_geo_jobs publishes weekly to Hugging Face under CC-BY 4.0. Each release tagged with the snapshot_date + snapshot_version from data_quality_snapshot for traceability.
• Underlying licences: Eurostat (CC-BY 4.0), BLS / ONS / DOL OFLC / StatCan (public records), GDELT (open), OpenAlex (CC0), Crossref (open). Per-row licence in the provenance JSONB.
• No commercial scraping: the platform never ingests scraped commercial data. All sources are official APIs, public-record datasets, or licensed feeds.

From June 2026 the EU Pay Transparency Directive requires employers to disclose salary ranges. This dataset is positioned to back compliance + market-intelligence workflows with:

• 30-country Eurostat SES 2022 panel (annual EUR by ISCO-08 group)
• 871 US H-1B LCA records with exact federal-mandated wages
• ESCO 28-language occupation labels for cross-language pay equity comparison (DE Geomatiker = FR Géomaticien = ES Geomático)
• Citation-grade provenance on every benchmark row, mappable to ISCO / SOC / KldB / PCS / NACE codes
• Daily quality scorecard with stale-source alerts so downstream consumers know freshness state