{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading waterinfo observations\n",
    "\n",
    "This notebook introduces how to use the `hydropandas` package to read, process and visualise data from the Waterinfo database. In this notebook the `ddlpy` package (https://github.com/Deltares/ddlpy) is used to access the waterinfo api. This package can be installed using `pip install rws-ddlpy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import contextily as ctx\n",
    "import pandas as pd\n",
    "\n",
    "import hydropandas as hpd\n",
    "from hydropandas.io.waterinfo import get_locations_gdf\n",
    "\n",
    "# enabling debug logging so we can see what happens in the background\n",
    "hpd.util.get_color_logger(\"INFO\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# settings\n",
    "grootheid_code = None\n",
    "locatie = \"schoonhoven\"\n",
    "proces_type = \"meting\"\n",
    "tmin = pd.Timestamp(\"2020-1-1\")\n",
    "tmax = pd.Timestamp(\"2020-1-3\")\n",
    "extent = (110000, 125000, 429550, 449900)  # Schoonhoven"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get waterinfo observations within an extent\n",
    "oc = hpd.read_waterinfo(\n",
    "    extent=extent,\n",
    "    grootheid_code=grootheid_code,\n",
    "    proces_type=proces_type,\n",
    "    locatie=locatie,\n",
    "    tmin=tmin,\n",
    "    tmax=tmax,\n",
    "    keep_all_obs=False,\n",
    ")\n",
    "oc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show all measurement types within the extent\n",
    "gdf_meas = get_locations_gdf()\n",
    "gdf_meas.loc[\n",
    "    locatie,\n",
    "    [\n",
    "        \"Grootheid.Code\",\n",
    "        \"Grootheid.Omschrijving\",\n",
    "        \"Groepering.Code\",\n",
    "        \"Groepering.Omschrijving\",\n",
    "        \"Parameter.Code\",\n",
    "        \"Parameter.Omschrijving\",\n",
    "        \"ProcesType\",\n",
    "    ],\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get data from a certain location and grootheid\n",
    "o1 = hpd.WaterlvlObs.from_waterinfo(\n",
    "    locatie=\"schoonhoven\",\n",
    "    grootheid_code=\"WATHTE\",\n",
    "    groepering_code=\"\",\n",
    "    proces_type=\"meting\",\n",
    "    tmin=tmin,\n",
    "    tmax=tmax,\n",
    "    location_gdf=gdf_meas,  # specifying the location_gdf signficantly speeds up the process\n",
    ")\n",
    "o1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get data from a certain location and grootheid\n",
    "o2 = hpd.WaterlvlObs.from_waterinfo(\n",
    "    locatie=\"schoonhoven\",\n",
    "    grootheid_code=\"WATHTE\",\n",
    "    groepering_code=\"\",\n",
    "    proces_type=\"astronomisch\",\n",
    "    tmin=tmin,\n",
    "    tmax=tmax,\n",
    "    location_gdf=gdf_meas,  # specifying the location_gdf signficantly speeds up the process\n",
    ")\n",
    "\n",
    "# plot data\n",
    "ax = o1[\"value\"].plot(ylabel=o1.unit, label=o1.name, legend=True)\n",
    "o2[\"value\"].plot(ylabel=o2.unit, label=o2.name, marker=\"o\", legend=True, ax=ax);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get all measurement points within the Netherlands\n",
    "gdf = hpd.io.waterinfo.get_locations_gdf()\n",
    "gdf = hpd.io.waterinfo.get_locations_within_extent(\n",
    "    gdf, extent=(482.06, 306602.42, 284182.97, 637049.52)\n",
    ")\n",
    "ax = gdf.plot(figsize=(10, 10))\n",
    "ctx.add_basemap(ax=ax, crs=28992, alpha=0.5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Water quality data\n",
    "\n",
    "The Waterinfo database also contains water quality data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get chloride concentration from the Oosterschelde (Bommenede 1 (b))\n",
    "o_cl = hpd.WaterlvlObs.from_waterinfo(\n",
    "    locatie=\"bommenede\",\n",
    "    parameter_code=\"Cl\",\n",
    "    tmin=\"2025-6-25\",\n",
    "    tmax=\"2025-6-30\",\n",
    "    location_gdf=gdf,\n",
    ")\n",
    "\n",
    "# plot data\n",
    "ax = o_cl[\"value\"].plot(ylabel=o_cl.unit, label=o_cl.name, legend=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or download all chloride measurements within a certain extent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc = hpd.read_waterinfo(\n",
    "    extent=(80000, 90000, 429550, 449900),\n",
    "    parameter_code=\"Cl\",\n",
    "    tmin=\"2025-6-1\",\n",
    "    tmax=\"2025-6-10\",\n",
    ")\n",
    "oc.plots.interactive_map(popup_width=350)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Find selection criteria\n",
    "\n",
    "Very often you don't know exactly the names of the location, grootheid_code, groepering_code or parameter_code. To get the data that you want you can follow these steps:\n",
    "\n",
    "1. get a geodataframe with all the locations in the extent\n",
    "2. query the geodataframe to find either a location/grootheid_code/groepering_code or parameter_code\n",
    "3. call `read_waterinfo` with your selection criteria and a tmin and tmax value"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1 download geodataframe with all measurement points in your extent\n",
    "gdf = hpd.io.waterinfo.get_locations_gdf()\n",
    "gdf_locatie = hpd.io.waterinfo.get_locations_within_extent(\n",
    "    gdf, extent=(80000, 90000, 429550, 449900)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 2 query the GeoDataFrame\n",
    "\n",
    "# print unique names\n",
    "print(f\"Unique values of Grootheid Code: \\n{gdf_locatie['Grootheid.Code'].unique()}\\n\")\n",
    "print(\n",
    "    f\"Unique values of Groepering Code: \\n{gdf_locatie['Groepering.Code'].unique()}\\n\"\n",
    ")\n",
    "print(f\"Unique values of Parameter Code: \\n{gdf_locatie['Parameter.Code'].unique()}\\n\")\n",
    "print(f\"Unique values of Proces Type: \\n{gdf_locatie['ProcesType'].unique()}\\n\")\n",
    "\n",
    "# plot locations\n",
    "ax = gdf_locatie.plot(\"Naam\", figsize=(16, 6), legend=True)\n",
    "ctx.add_basemap(ax=ax, crs=28992, alpha=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 3 read data for selection criteria\n",
    "oc = hpd.read_waterinfo(\n",
    "    parameter_code=\"Cl\", tmin=\"2025-6-1\", tmax=\"2025-6-10\", location_gdf=gdf_locatie\n",
    ")\n",
    "oc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieving data from WaterWebservices \n",
    "\n",
    "In the future, the WaterWebservices will become available. Currently (2024-03-20), they are not yet working. When retrieving measurements, it is always indicated that the maximum number of measurements is exceeded. Even when retrieving measurements for only one day.\n",
    "\n",
    "Useful information:\n",
    "\n",
    "- https://waterwebservices.beta.rijkswaterstaat.nl/test/swagger-ui/index.html#/\n",
    "- https://rijkswaterstaatdata.nl/projecten/beta-waterwebservices/\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import json\n",
    "\n",
    "# import requests\n",
    "# from shapely.geometry import Point\n",
    "\n",
    "# import nlmod"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # request catalogus REST API\n",
    "# url = \"https://waterwebservices.beta.rijkswaterstaat.nl/test/METADATASERVICES/OphalenCatalogus\"\n",
    "# body = {\"CatalogusFilter\": {\"Compartimenten\": True, \"Grootheden\": True}}\n",
    "# headers = {\"content-type\": \"application/json\"}\n",
    "# r = requests.post(url, data=json.dumps(body), headers=headers)\n",
    "\n",
    "# out = r.json()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # plot locations from REST API\n",
    "# geometries = [Point(loc[\"Lon\"], loc[\"Lat\"]) for loc in out[\"LocatieLijst\"]]\n",
    "# gdf = gpd.GeoDataFrame(out[\"LocatieLijst\"], geometry=geometries, crs=4258)\n",
    "# gdf.to_crs(28992, inplace=True)\n",
    "# # extent_nl_poly = nlmod.util.extent_to_polygon(\n",
    "#     [482.06, 306602.42, 284182.97, 637049.52]\n",
    "# )\n",
    "# gdf = gdf.loc[gdf.within(extent_nl_poly)]\n",
    "# ax = gdf.plot(figsize=(10, 10))\n",
    "# nlmod.plot.add_background_map(ax=ax)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read REST API using an extent (Schoonhoven zuid-west)\n",
    "# extent_schoon = nlmod.util.polygon_from_extent((117850, 118180, 439550, 439900))\n",
    "# gdf_schoon = gdf.loc[gdf.within(extent_schoon.buffer(10000))]\n",
    "# ax = gdf_schoon.plot(\"Code\", figsize=(10, 10), legend=True)\n",
    "# nlmod.plot.add_background_map(ax=ax, alpha=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # kies code en laat zien welke parameters erbij horen\n",
    "# code = \"schoonhoven\"\n",
    "\n",
    "# df_meta_locatie = pd.DataFrame(out[\"AquoMetadataLocatieLijst\"]).set_index(\n",
    "#     \"Locatie_MessageID\"\n",
    "# )\n",
    "# df_meta = pd.DataFrame(out[\"AquoMetadataLijst\"])\n",
    "\n",
    "# locatie_message_id = gdf.loc[gdf[\"Code\"] == code, \"Locatie_MessageID\"].iloc[0]\n",
    "\n",
    "# AquoMetaData_MessageIDs = df_meta_locatie.loc[\n",
    "#     locatie_message_id, \"AquoMetaData_MessageID\"\n",
    "# ]\n",
    "# if isinstance(AquoMetaData_MessageIDs, int):\n",
    "#     AquoMetaData_MessageIDs = [AquoMetaData_MessageIDs]\n",
    "# df_meta.loc[AquoMetaData_MessageIDs]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # request om metingen op te halen\n",
    "# code = \"ameland.nes\"\n",
    "# aquometadata_message_id = 6\n",
    "\n",
    "# url = \"https://waterwebservices.beta.rijkswaterstaat.nl/test/ONLINEWAARNEMINGENSERVICES/OphalenWaarnemingen\"\n",
    "# body = {\n",
    "#     \"Locatie\": {\"Code\": code},\n",
    "#     \"AquoPlusWaarnemingMetadata\": {\n",
    "#         \"AquoMetadata\": {\n",
    "#             \"Compartiment\": {\n",
    "#                 \"Code\": df_meta.loc[aquometadata_message_id, \"Compartiment\"][\"Code\"]\n",
    "#             },\n",
    "#             \"Grootheid\": {\n",
    "#                 \"Code\": df_meta.loc[aquometadata_message_id, \"Grootheid\"][\"Code\"]\n",
    "#             },\n",
    "#         }\n",
    "#     },\n",
    "#     \"Periode\": {\n",
    "#         \"Begindatumtijd\": \"2024-06-01T00:00:00.000+01:00\",\n",
    "#         \"Einddatumtijd\": \"2025-01-01T00:00:00.000+01:00\",\n",
    "#     },\n",
    "# }\n",
    "# headers = {\"content-type\": \"application/json\"}\n",
    "# r = requests.post(url, data=json.dumps(body), headers=headers)\n",
    "# if r.status_code == 204:\n",
    "#     print(\"No data available\")\n",
    "# else:\n",
    "#     meas = r.json()\n",
    "#     print(meas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # andere poging\n",
    "# code = \"ameland.nes\"\n",
    "# aquometadata_message_id = 6\n",
    "\n",
    "# url = \"https://waterwebservices.beta.rijkswaterstaat.nl/test/ONLINEWAARNEMINGENSERVICES/OphalenWaarnemingen\"\n",
    "# body = {\n",
    "#     \"Locatie\": {\"Code\": code},\n",
    "#     \"AquoPlusWaarnemingMetadata\": {\n",
    "#         \"AquoMetadata\": {\n",
    "#             \"ProcesType\": \"verwachting\",\n",
    "#             \"Grootheid\": {\n",
    "#                 \"Code\": df_meta.loc[aquometadata_message_id, \"Grootheid\"][\"Code\"]\n",
    "#             },\n",
    "#         }\n",
    "#     },\n",
    "#     \"Periode\": {\n",
    "#         \"Begindatumtijd\": \"2024-01-01T00:00:00.000+01:00\",\n",
    "#         \"Einddatumtijd\": \"2024-01-02T00:00:00.000+01:00\",\n",
    "#     },\n",
    "# }\n",
    "# headers = {\"content-type\": \"application/json\"}\n",
    "# r = requests.post(url, data=json.dumps(body), headers=headers)\n",
    "# if r.status_code == 204:\n",
    "#     print(\"No data available\")\n",
    "# else:\n",
    "#     meas = r.json()\n",
    "#     print(meas)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hydropandas",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}