{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exploring groundwater data from Lizard\n",
    "\n",
    "In this notebook, you will experiment how to use the `hydropandas` package to access, visualize and explore meta data from Lizard, a cloud datawarehouse that can be used to store groundwater observations. Data sources:\n",
    "\n",
    "1. Vitens (default, public access): Vitens is the largest drinking water company in the Netherlands, and it has more than 10.000 groundwater wells and more than 50.000 timeseries in its datawarehouse. The data spans from the 1930’s to the present, and it is constantly updated with new observations. Vitens also validates the data using ArtDiver and provides quality flags and comments for each observation. The data is open to the public and you can find more information at https://vitens.lizard.net.\n",
    "2. Others: If you have access to other Lizard data sources, you can specify the `organisation` parameter in the `lizard` functions to access those data sources. Currently, only \"vitens\" is officially supported. Note that you then may need to specify authentication as well by means of a personal API key. This has been implemented for the municipality of Rotterdam.\n",
    "\n",
    "---\n",
    "\n",
    "Feel free to customize and expand upon this introduction as needed.\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id=top></a>0. Notebook contents\n",
    "\n",
    "1. [Read groundwater timeseries](#read)\n",
    "2. [Analyse groundwater observations](#gwobs)\n",
    "3. [Build a Pastas model](#pastas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pastas as ps\n",
    "\n",
    "import hydropandas as hpd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hpd.util.get_color_logger(\"INFO\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Read groundwater timeseries<a id=read></a>\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Authentication\n",
    "The `vitens` data source is used by default, and does not require authentication. If you have access to other Lizard data sources, you can specify the `organisation` parameter in the `read_lizard` function to access those data sources. Currently, only \"vitens\" is officially supported.\n",
    "\n",
    "For others than Vitens, you may need authentication. You can use the `auth` parameter to pass your credentials. According to the [Lizard documentation](https://docs.lizard.net/c_endpoints.html#authentication), you should create a personal API key at `https://{organisation}.lizard.net/management/` and enter that in the cell below. If you do not have access to other data sources, you can use `auth=None` to skip authentication."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Lizard uses the organisation parameter to specify the data source.\n",
    "organisation = \"vitens\"  # or: \"rotterdam\"\n",
    "auth = None\n",
    "\n",
    "## UNCOMMENT AND SET YOUR API KEY IF YOU NEED TO SPECIFY CREDENTIALS\n",
    "# your_api_key = \"your_api_key_here\"\n",
    "# auth = (\"__key__\", your_api_key)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Which timeseries to include? (`which_timeseries`)\n",
    "In the Lizard database, the follow WNS codes are used:\n",
    "\n",
    "1. **WNS9040.hand**:    Hand measurements               *(in Hydropandas: \"hand\")*\n",
    "2. **WNS9040**:         Diver measurements              *(in Hydropandas: \"diver\")*\n",
    "3. **WNS9040.val**:     Validated diver measurements    *(in Hydropandas: \"diver_validated\")*\n",
    "\n",
    "Use of WNS codes per organisation:\n",
    "\n",
    "- Vitens uses only categories 1 and 2. So for consistency with previous versions, Hydropandas will use \"hand\" and \"diver\" measurements when `which_timeseries` is not specified.\n",
    "- Rotterdam uses all three categories. So for Rotterdam you can use `which_timeseries=[\"hand\", \"diver\", \"diver_validated\"]` to get both the validated and unvalidated diver measurements."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Get observations from extent <a id=extent></a>\n",
    "\n",
    "Use `read_lizard` to find monitoring wells by specifying a geographical extent in Rijksdriehoeks coordinates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_extent = (137_000, 138_000, 458_000, 459_000)\n",
    "oc = hpd.read_lizard(\n",
    "    extent=my_extent,\n",
    "    which_timeseries=[\"hand\", \"diver\", \"diver_validated\"],\n",
    "    datafilters=None,\n",
    "    combine_method=\"merge\",\n",
    "    organisation=organisation,\n",
    "    auth=auth,\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize all groundwater wells inside the extent on a map (visualize the `ObsCollection`). The markers are clickable to show a preview of the availables observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc.plots.interactive_map(\n",
    "    color=\"red\", zoom_start=15, tiles=\"Esri.WorldImagery\", popup_width=350\n",
    ")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print all the retrieved groundwater wells and tubes, and make a plot of the observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc.obs.values[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc.plots.section_plot(plot_obs=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Analyse Groundwater observations <a id=gwobs></a>\n",
    "\n",
    "Now lets download the groundwater level observation using the `from_lizard` function of a `GroundwaterObs` object. The code below reads the groundwater level timeseries for the well UPWP016 from Lizard and makes a plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gw_lizard = hpd.GroundwaterObs.from_lizard(\n",
    "    \"UPWP016\",\n",
    "    tmin=\"1900-01-01\",\n",
    "    tmax=\"2030-01-01\",\n",
    "    organisation=\"vitens\",\n",
    "    auth=auth,\n",
    ")\n",
    "print(gw_lizard)\n",
    "\n",
    "ax = gw_lizard[\"value\"].plot(\n",
    "    figsize=(12, 5),\n",
    "    marker=\".\",\n",
    "    grid=True,\n",
    "    label=gw_lizard.name,\n",
    "    legend=True,\n",
    "    xlabel=\"Date\",\n",
    "    ylabel=\"m NAP\",\n",
    "    title=\"Groundwater observations for \" + gw_lizard.name,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The groundwater observations contain a validation flag per timestamp. These can 'betrouwbaar' (reliable), 'onbetrouwbaar' (unreliable) en 'onbeslist' (unvalidated). Below flags of the timeseries are shown as a percentage, and the unreliable timestamps are printed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(gw_lizard[\"flag\"].value_counts(normalize=True) * 100)\n",
    "gw_lizard[gw_lizard[\"flag\"] == \"onbetrouwbaar\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Create a Pastas model <a id=pastas></a>\n",
    "\n",
    "Lets make a Pastas model for this groundwater well (starting from 2015) and use the nearest KNMI station for meteorological data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the precipitation and evaporation data from the KNMI\n",
    "precipitation = hpd.PrecipitationObs.from_knmi(\n",
    "    xy=(gw_lizard.x, gw_lizard.y),\n",
    "    start=gw_lizard.index[0],\n",
    "    end=gw_lizard.index[-1],\n",
    "    fill_missing_obs=True,\n",
    ")\n",
    "evaporation = hpd.EvaporationObs.from_knmi(\n",
    "    xy=(gw_lizard.x, gw_lizard.y),\n",
    "    meteo_var=\"EV24\",\n",
    "    start=gw_lizard.index[0],\n",
    "    end=gw_lizard.index[-1],\n",
    "    fill_missing_obs=True,\n",
    ")\n",
    "\n",
    "# Create a Pastas Model\n",
    "ml = ps.Model(\n",
    "    gw_lizard.loc[gw_lizard[\"origin\"] == \"diver\", \"value\"], name=gw_lizard.name\n",
    ")\n",
    "\n",
    "# Add the recharge data as explanatory variable\n",
    "ts1 = ps.RechargeModel(\n",
    "    precipitation[\"RH\"].resample(\"D\").first(),\n",
    "    evaporation[\"EV24\"].resample(\"D\").first(),\n",
    "    ps.Gamma(),\n",
    "    name=\"rainevap\",\n",
    "    settings=(\"prec\", \"evap\"),\n",
    ")\n",
    "\n",
    "# Add the stressmodel to the model and solve for period after 2015\n",
    "ml.add_stressmodel(ts1)\n",
    "ml.solve(tmin=\"2015\")\n",
    "ml.plots.results(figsize=(10, 6))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}