{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "56071b81",
   "metadata": {},
   "source": [
    "# API Reliability and Production Patterns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bff62782",
   "metadata": {},
   "source": [
    "**Learning goals** \u2014 By the end of this section you will be able to:\n",
    "\n",
    "- Use authentication headers and environment variables safely\n",
    "- Handle paginated APIs with a loop that accumulates results\n",
    "- Implement retry logic with exponential backoff for transient failures\n",
    "- Validate JSON response contracts defensively before downstream processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5166c443",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import time\n",
    "import random\n",
    "import requests"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81fbd654",
   "metadata": {},
   "source": [
    "## POST Requests\n",
    "\n",
    "A **GET** request retrieves data from a server. A **POST** request sends\n",
    "data to a server \u2014 typically to create a resource or submit a form."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "444f7ba0",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "response = requests.post('https://httpbin.org/post', json={'name': 'Alice', 'role': 'student'})\n",
    "response.raise_for_status()\n",
    "data = response.json()\n",
    "print('Status:', response.status_code)\n",
    "print('Server received JSON:', data['json'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6a91f79",
   "metadata": {},
   "source": [
    "## Headers and API Keys\n",
    "\n",
    "Use `headers={}` for auth metadata. Never hard-code real keys in source files.\n",
    "Load secrets from environment variables (for example, `os.environ`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8120adaa",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "api_key = os.environ.get('DEMO_API_KEY', 'demo-key')\n",
    "headers = {'Authorization': f'Bearer {api_key}', 'Accept': 'application/json'}\n",
    "resp = requests.get('https://httpbin.org/headers', headers=headers, timeout=10)\n",
    "resp.raise_for_status()\n",
    "print(resp.json()['headers'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c743e594",
   "metadata": {},
   "source": [
    "## Pagination Patterns\n",
    "\n",
    "Most APIs return data in chunks. Loop until there is no next page token."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61aec0a3",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Mock paginated API responses for teaching\n",
    "pages = {\n",
    "    1: {'items': ['a', 'b'], 'next_page': 2},\n",
    "    2: {'items': ['c', 'd'], 'next_page': 3},\n",
    "    3: {'items': ['e'], 'next_page': None},\n",
    "}\n",
    "\n",
    "all_items = []\n",
    "page = 1\n",
    "while page is not None:\n",
    "    payload = pages[page]\n",
    "    all_items.extend(payload['items'])\n",
    "    page = payload['next_page']\n",
    "\n",
    "print('Collected:', all_items)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53c22eb3",
   "metadata": {},
   "source": [
    "## Retry with Exponential Backoff\n",
    "\n",
    "Retry transient failures (timeouts, 429, 5xx) with increasing wait times."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d65ac82b",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def get_with_retry(url, params=None, retries=3, timeout=5):\n",
    "    delay = 0.5\n",
    "    for attempt in range(1, retries + 1):\n",
    "        try:\n",
    "            resp = requests.get(url, params=params, timeout=timeout)\n",
    "            if resp.status_code in (429, 500, 502, 503, 504):\n",
    "                raise requests.HTTPError(f'Transient HTTP {resp.status_code}', response=resp)\n",
    "            resp.raise_for_status()\n",
    "            return resp\n",
    "        except (requests.Timeout, requests.ConnectionError, requests.HTTPError) as e:\n",
    "            if attempt == retries:\n",
    "                raise\n",
    "            sleep_for = delay * (2 ** (attempt - 1)) + random.uniform(0, 0.2)\n",
    "            print(f'Retry {attempt}/{retries - 1} after {sleep_for:.2f}s -> {type(e).__name__}')\n",
    "            time.sleep(sleep_for)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff1965ae",
   "metadata": {},
   "source": [
    "## Response Contract Validation\n",
    "\n",
    "Validate required keys before downstream processing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc85b670",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def parse_weather_response(payload):\n",
    "    if 'current_weather' not in payload:\n",
    "        raise ValueError('Missing current_weather in API response')\n",
    "\n",
    "    current = payload['current_weather']\n",
    "    if 'temperature' not in current or 'windspeed' not in current:\n",
    "        raise ValueError('Missing required weather fields: temperature/windspeed')\n",
    "\n",
    "    return {'temperature': current['temperature'], 'windspeed': current['windspeed']}\n",
    "\n",
    "sample = {'current_weather': {'temperature': 21.3, 'windspeed': 5.8}}\n",
    "print(parse_weather_response(sample))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7b303a98",
   "metadata": {
    "tags": [
     "thebe-interactive"
    ]
   },
   "outputs": [],
   "source": [
    "### Exercise: Build a Resilient API Client\n",
    "# 1) Write fetch_all_pages(get_page_fn) that keeps requesting until no next page\n",
    "# 2) Wrap calls with retry for Timeout and transient HTTP errors\n",
    "# 3) Validate each page has an items key before extending results\n",
    "\n",
    "# Your code here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6187920d",
   "metadata": {
    "tags": [
     "hide-input"
    ]
   },
   "outputs": [],
   "source": [
    "### Solution\n",
    "def fetch_all_pages(get_page_fn, retries=3):\n",
    "    items = []\n",
    "    page = 1\n",
    "\n",
    "    while page is not None:\n",
    "        delay = 0.5\n",
    "        for attempt in range(1, retries + 1):\n",
    "            try:\n",
    "                payload = get_page_fn(page)\n",
    "                if 'items' not in payload:\n",
    "                    raise ValueError('Page payload missing items key')\n",
    "                items.extend(payload['items'])\n",
    "                page = payload.get('next_page')\n",
    "                break\n",
    "            except Exception:\n",
    "                if attempt == retries:\n",
    "                    raise\n",
    "                time.sleep(delay * (2 ** (attempt - 1)))\n",
    "\n",
    "    return items\n",
    "\n",
    "mock_pages = {1: {'items': [10, 20], 'next_page': 2}, 2: {'items': [30], 'next_page': None}}\n",
    "\n",
    "def mock_get_page(p):\n",
    "    return mock_pages[p]\n",
    "\n",
    "print(fetch_all_pages(mock_get_page))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a723693f",
   "metadata": {},
   "source": [
    "## Summary\n",
    "\n",
    "| Concept | Key idea |\n",
    "|---|---|\n",
    "| Auth headers + env vars | Keep secrets out of source code; load from environment |\n",
    "| Pagination loop | Request pages until no next page/cursor remains |\n",
    "| Retry + backoff | Retry transient failures with increasing wait times |\n",
    "| Defensive validation | Check required JSON keys before business logic |"
   ]
  }
 ],
 "metadata": {
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}