{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "title-and-safeguards",
   "metadata": {},
   "source": [
    "# DataPulse MY — Trust Layer in a Notebook\n",
    "\n",
    "Learn the sequence: **check health → inspect evidence → decide whether to use → fetch official data**.\n",
    "\n",
    "## Safeguards\n",
    "\n",
    "1. **Verification-first artifact rule.** Every recipe, notebook, case study, or badge must show the dataset ID, canonical official source, DataPulse status, `checked_at`, freshness evidence, and any caveat before showing the data result. An artifact missing any field does not publish.\n",
    "\n",
    "2. **Fail closed in examples.** Examples may proceed automatically only for statuses explicitly allowed by the recipe. `stale`, `degraded`, `unreachable`, `unknown`, and `unknown-freshness` must produce a visible stop or warning; `browser-dependent` and `Volatile` must explain their limitations. No status may be collapsed into a generic green “available” label.\n",
    "\n",
    "3. **Trust-first information architecture.** The homepage hero and primary CTA remain “verified health” and “view live status/connect your agent.” Outreach artifacts live under a secondary “Learn by example” path and may not replace the live status distribution, taxonomy, or verification mechanism above the fold.\n",
    "\n",
    "4. **Contribution boundary.** Community work is accepted only if it improves at least one recorded trust metric: verified coverage, content-date extraction, schema-drift detection, probe reliability, provenance, licence evidence, or reproducible failure documentation. Requests for charts, data cleaning, hosted querying, alerts, or unrelated APIs are declined or logged as paid-layer discovery.\n",
    "\n",
    "5. **Maintainer-controlled evidence.** No community submission may directly alter production probe policy or status classification. It requires review, a reproducible fixture, source evidence, and a successful canary run. Recognition is granted only after acceptance.\n",
    "\n",
    "6. **Canonical-source rule.** All artifacts consume the same published health envelope and taxonomy; no notebook, badge, or page maintains its own status logic. A taxonomy change must update the schema and all affected artifacts together.\n",
    "\n",
    "7. **No free convenience creep.** The outreach layer may teach how to check trust and then access an official source. It may not host normalized datasets, promise stable schemas, run persistent alerts, provide cross-source joins, or guarantee delivery. Those are convenience-layer capabilities and require an explicit product decision.\n",
    "\n",
    "8. **Evidence over audience metrics.** The 30-day scorecard contains: external citations/backlinks to a reproducible artifact, successful notebook runs, MCP queries that include health/provenance, accepted probe improvements, and unknown-freshness cases resolved. Followers, page views, submissions, and demo count are diagnostic only and cannot justify more scope.\n",
    "\n",
    "9. **Fixed outreach budget.** Until a paid convenience offer is validated, outreach maintenance is capped at one maintainer-day per week and may add no standalone database, queue, account system, or always-on service. Work exceeding either limit pauses for a layer-boundary review.\n",
    "\n",
    "10. **ODIN claim discipline.** Every use of the ranking must say what it measures—coverage/openness—and must not imply that ODIN certifies freshness, DataPulse, or individual datasets. DataPulse is an independent verification layer, not an official ODIN or Malaysian-government endorsement.\n",
    "\n",
    "11. **Public limitation statement.** Each artifact must state that DataPulse verifies defined observable signals, not the semantic truth or fitness-for-purpose of every record. This prevents an honest health status from being read as a blanket data-quality guarantee.\n",
    "\n",
    "12. **One-way escalation rule.** Repeated requests for wrappers, cleaned feeds, stable schemas, or alerts are interviewed as potential paid convenience demand; they are not added to the free trust layer. Build only after a named buyer, use case, and willingness to pay are established.\n",
    "\n",
    "**What this notebook proves:** verified MY open data is usable right now, and the trust surface shows you exactly when it isn't."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "why-this-matters",
   "metadata": {},
   "source": [
    "## Why this matters\n",
    "\n",
    "Malaysia ranks #1 in the world for open data coverage and openness (ODIN 2024/25, 99/100 for openness). But \"published\" ≠ \"fresh\" — most endpoints don't send Last-Modified headers, and 200 OK can hide years of staleness. ODIN does not certify freshness, DataPulse, or individual datasets.\n",
    "\n",
    "DataPulse independently probes **335 official datasets every 15 minutes** and classifies each into one of 8 honest statuses. This notebook shows you how to read that surface and decide which datasets to actually use."
   ]
  },
  {
   "cell_type": "code",
   "id": "load-health-envelope",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests, pandas as pd\n",
    "HEALTH_URL = \"https://data-pulse.my/health/latest.json\"\n",
    "health = requests.get(HEALTH_URL, timeout=10).json()\n",
    "ts = health[\"_trust_summary\"]\n",
    "ds = health[\"datasets\"]\n",
    "print(f\"Total datasets: {ts['datasets_total']}\")\n",
    "print(f\"Checked at:    {ts['checked_at']}\")\n",
    "print(f\"Status distribution: {ts['by_status']}\")"
   ]
  },
  {
   "cell_type": "code",
   "id": "status-distribution",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Render the 9-status taxonomy\n",
    "status_df = pd.DataFrame([\n",
    "    {\"status\": k, \"count\": v, \"percent\": f\"{v/ts['datasets_total']*100:.1f}%\"}\n",
    "    for k, v in sorted(ts['by_status'].items(), key=lambda kv: -kv[1])\n",
    "])\n",
    "status_df.style.hide(axis='index').format({'count': '{:,}'})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fresh-example-intro",
   "metadata": {},
   "source": [
    "## Example 1: a fresh dataset you can use right now\n",
    "\n",
    "Use `fuelprice` — daily fuel prices from Malaysia's Ministry of Finance. The live evidence below reports when it was last probed, its content freshness date, official source, and current status. Because status can change, the next cell refuses to fetch unless the live status is `fresh` or `aging`."
   ]
  },
  {
   "cell_type": "code",
   "id": "inspect-fresh-evidence",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "fresh_example = \"fuelprice\"\n",
    "fresh_rec = next(d for d in ds if d[\"dataset_id\"] == fresh_example)\n",
    "print(f\"Dataset:     {fresh_rec['dataset_id']}\")\n",
    "print(f\"Status:      {fresh_rec['status']}\")\n",
    "print(f\"Last check:  {fresh_rec['last_checked']}\")\n",
    "print(f\"Fresh date:  {fresh_rec['content_freshness_date']}\")\n",
    "print(f\"Rows:        {fresh_rec['record_count']:,}\" if isinstance(fresh_rec['record_count'], int) else f\"Rows: {fresh_rec['record_count']}\")\n",
    "print(f\"Source URL:  {fresh_rec['request_url']}\")\n",
    "print(\"Caveat:      Status verifies observable freshness signals, not every value.\")"
   ]
  },
  {
   "cell_type": "code",
   "id": "fetch-fresh-data",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Verification-first: refuse to use this dataset unless status is acceptable\n",
    "ALLOWED = {\"fresh\", \"aging\"}\n",
    "if fresh_rec[\"status\"] not in ALLOWED:\n",
    "    raise SystemExit(f\"BLOCKED: {fresh_example} status is {fresh_rec['status']}; not safe to fetch\")\n",
    "\n",
    "print(f\"✓ Status '{fresh_rec['status']}' is allowed. Fetching official data...\")\n",
    "source_response = requests.get(fresh_rec[\"request_url\"], timeout=30)\n",
    "source_response.raise_for_status()\n",
    "df = pd.DataFrame(source_response.json())\n",
    "print(f\"Fetched {len(df):,} rows × {len(df.columns)} columns\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "stale-example-intro",
   "metadata": {},
   "source": [
    "## Example 2: a dataset you should NOT use right now\n",
    "\n",
    "Find the oldest stale dataset by its live `staleness_days` evidence. Show its dataset ID, status, last check, freshness evidence, official source, and caveat. Refuse to fetch."
   ]
  },
  {
   "cell_type": "code",
   "id": "block-stale-data",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def require_current_dataset(record):\n",
    "    blocked = {\"stale\", \"degraded\", \"unreachable\", \"unknown\", \"unknown_freshness\"}\n",
    "    if record[\"status\"] in blocked:\n",
    "        raise SystemExit(f\"BLOCKED: '{record['status']}' status; do NOT use this dataset for current decisions.\")\n",
    "\n",
    "stale_examples = [d for d in ds if d[\"status\"] == \"stale\"]\n",
    "oldest = max(stale_examples, key=lambda d: (d.get(\"staleness_days\") or -1, d[\"dataset_id\"]))\n",
    "print(f\"Dataset:    {oldest['dataset_id']}\")\n",
    "print(f\"Status:     {oldest['status']}\")\n",
    "print(f\"Last check: {oldest['last_checked']}\")\n",
    "print(f\"Fresh date: {oldest.get('content_freshness_date', 'N/A')}\")\n",
    "print(f\"Last mod:   {oldest.get('last_modified', 'N/A')}\")\n",
    "print(f\"Days old:   {oldest.get('staleness_days', 'N/A')}\")\n",
    "print(f\"Source URL: {oldest['request_url']}\")\n",
    "print(\"Caveat:     Suitable only for explicitly dated historical research.\")\n",
    "print()\n",
    "\n",
    "# Fail-closed: expose the stop visibly without fetching stale data.\n",
    "try:\n",
    "    require_current_dataset(oldest)\n",
    "except SystemExit as error:\n",
    "    print(f\"⛔ {error}\")\n",
    "    evidence_date = oldest.get('content_freshness_date') or oldest.get('last_modified', 'unknown')\n",
    "    print(f\"   For historical research, cite explicitly: \\\"as of {evidence_date}\\\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "usable-datasets-intro",
   "metadata": {},
   "source": [
    "## Bonus: find 10 datasets you can use right now (fresh or aging only)"
   ]
  },
  {
   "cell_type": "code",
   "id": "filter-usable-datasets",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "usable = [d for d in ds if d[\"status\"] in (\"fresh\", \"aging\")]\n",
    "print(f\"Usable right now: {len(usable)} of {ts['datasets_total']} ({len(usable)/ts['datasets_total']*100:.1f}%)\")\n",
    "print(f\"\\nFirst 10 (sorted by most-recently-checked):\")\n",
    "sorted_usable = sorted(usable, key=lambda d: d[\"last_checked\"], reverse=True)\n",
    "for d in sorted_usable[:10]:\n",
    "    print(f\"  {d['dataset_id']:35}  status={d['status']:8}  last_checked={d['last_checked']}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "limitations-and-citation",
   "metadata": {},
   "source": [
    "## Limitations\n",
    "\n",
    "DataPulse verifies defined observable signals (HTTP status, header freshness, content-date extraction, record-count stability, schema shape). It does NOT verify the semantic truth or fitness-for-purpose of every record. A fresh dataset is one you can use **knowing the publisher's update cadence is being honored**, not one whose every value is correct.\n",
    "\n",
    "**Cite this notebook:**\n",
    "> DataPulse MY. (2026). *Trust Layer in a Notebook*. Retrieved from\n",
    "> https://data-pulse.my/docs/trust-layer-notebook.ipynb"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "name": "DataPulse MY — Trust Layer in a Notebook",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}