{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Reading groundwater observations\n",
    "\n",
    "This notebook introduces how to use the `hydropandas` package to read, process and visualise groundwater data from Dino and Bro databases."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id=top></a>Notebook contents\n",
    "\n",
    "1. [GroundwaterObs](#GroundwaterObs)\n",
    "2. [ObsCollections](#ObsCollections)\n",
    "3. [Read ObsCollections](#Read-ObsCollections)\n",
    "4. [Write ObsCollections](#Write-ObsCollections)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import contextily as ctx\n",
    "from IPython.display import HTML\n",
    "\n",
    "import hydropandas as hpd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hpd.util.get_color_logger(\"INFO\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GroundwaterObs\n",
    "\n",
    "The hydropandas package has several functions to read groundwater observations at a measurement well. These include reading data from:\n",
    "\n",
    "- dino (from csv-files).\n",
    "- bro (using the bro-api)\n",
    "- fews (xml dumps from the fews database)\n",
    "- wiski (dumps from the wiski database)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reading a dino csv file\n",
    "path = \"data/Grondwaterstanden_Put/B33F0080001_1.csv\"\n",
    "gw_dino = hpd.GroundwaterObs.from_dino(path=path)\n",
    "gw_dino"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# reading the same filter from using the bro api. Specify a groundwater monitoring id (GMW00...) and a filter number (1)\n",
    "gw_bro = hpd.GroundwaterObs.from_bro(\"GMW000000041261\", 1)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have an `GroundwaterObs` object named `gw_bro` and `gw_dino`. Both objects are from the same measurement well in different databases. A `GroundwaterObs` object inherits from a pandas `DataFrame` and has the same attributes and methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gw_bro.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gw_bro"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = gw_dino[\"stand_m_tov_nap\"].plot(\n",
    "    label=\"dinoloket\", figsize=(14, 5), legend=True, marker=\".\", lw=0.2\n",
    ")\n",
    "gw_bro[\"values\"].plot(ax=ax, label=\"bro\", legend=True, ylabel=gw_bro.unit)\n",
    "gw_dino[\"ground_level\"].plot(\n",
    "    ax=ax,\n",
    "    label=\"ground level\",\n",
    "    legend=True,\n",
    "    grid=True,\n",
    "    color=\"green\",\n",
    "    ylabel=gw_dino.unit,\n",
    ")\n",
    "\n",
    "ax.set_title(f\"same tube from Dinoloket {gw_dino.name} and BRO id {gw_bro.name}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GroundwaterObs Attributes\n",
    "\n",
    "Besides the standard `DataFrame` attributes a `GroundwaterObs` has the following additional attributes:\n",
    "\n",
    "- x, y: x- and y-coordinates of the observation point\n",
    "- name: str with the name\n",
    "- filename: str with the filename (only available when the data was loaded from a file)\n",
    "- monitoring_well: the name of the monitoring_well. One monitoring well can have multiple tubes.\n",
    "- tube_nr: the number of the tube. The combination of monitoring_well and tube_nr should be unique\n",
    "- screen_top: the top of the tube screen (bovenkant filter in Dutch)\n",
    "- screen_bottom: the bottom of the tube screen (onderkant filter in Dutch)\n",
    "- ground_level: surface level (maaiveld in Dutch)\n",
    "- tube_top: the top of the tube\n",
    "- metadata_available: boolean indicating whether metadata is available for this observation point\n",
    "- meta: dictionary with additional metadata\n",
    "\n",
    "When dowloading from Dinoloket all levels are in meters NAP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(gw_bro)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### GroundwaterObs methods\n",
    "\n",
    "Besides the standard `DataFrame` methods a `GroundwaterObs` has additional methods. This methods are accessible through submodules:\n",
    "\n",
    "- `geo.get_lat_lon()`, to obtain latitude and longitude\n",
    "- `gwobs.get_modellayer()`, to obtain the modellayer of a modflow model using the filter depth\n",
    "- `stats.get_seasonal_stat()`, to obtain seasonal statistics\n",
    "- `stats.obs_per_year()`, to obtain the number of observations per year\n",
    "- `stats.consecutive_obs_years()`, to obtain the number of consecutive years with more than a minimum number of observations\n",
    "- `plots.interactive_plot()`, to obtain a bokeh plot\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get latitude and longitude with `gw.geo.get_lat_lon()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"latitude and longitude -> {gw_bro.geo.get_lat_lon()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gw_bro.stats.get_seasonal_stat(stat=\"mean\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "p = gw_bro.plots.interactive_plot(\"figure\")\n",
    "HTML(filename=\"figure/{}.html\".format(gw_bro.name))"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ObsCollections\n",
    "\n",
    "`ObsCollections` are a combination of multiple observation objects. The easiest way to construct an `ObsCollections` is from a list of observation objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path1 = \"data/Grondwaterstanden_Put/B33F0080001_1.csv\"\n",
    "path2 = \"data/Grondwaterstanden_Put/B33F0133001_1.csv\"\n",
    "gw1 = hpd.GroundwaterObs.from_dino(path=path1)\n",
    "gw2 = hpd.GroundwaterObs.from_dino(path=path2)\n",
    "\n",
    "# create ObsCollection\n",
    "oc = hpd.ObsCollection([gw1, gw2], name=\"Dino groundwater\")\n",
    "oc"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have an `ObsCollection` object named `oc`. The `ObsCollection` contains all the data from the two `GroundwaterObs` objects. It also stores a reference to the `GroundwaterObs` objects in the 'obs' column. An `ObsCollection` object also inherits from a pandas `DataFrame` and has the same attributes and methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get columns\n",
    "oc.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get individual GroundwaterObs object from an ObsCollection\n",
    "o = oc.loc[\"B33F0133-001\", \"obs\"]\n",
    "o"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get statistics\n",
    "oc.describe()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ObsCollection methods\n",
    "Besides the methods of a pandas `DataFrame` an `ObsCollection` has additional methods stored in submodules.\n",
    "\n",
    "`geo`:\n",
    "\n",
    "- `get_bounding_box` -> get a tuple with (xmin, ymin, xmax, ymax)\n",
    "- `get_extent` -> get a tule with (xmin, xmax, ymin, ymax)\n",
    "- `get_lat_lon` -> to get the lattitudes and longitudes from the x and y coordinates\n",
    "- `within_polygon` -> to select only the observations that lie within a polygon\n",
    "\n",
    "`gwobs`:\n",
    "\n",
    "- `set_tube_nr` -> to set the tube numbers based on the tube screen depth when there are multiple tubes at one monitoring well\n",
    "- `set_tube_nr_monitoring_well` -> find out which observations are at the same location with a different screen depth. Set monitoring_well and tube_nr attributes accordingly.\n",
    "\n",
    "`plots`:\n",
    "\n",
    "- `interactive_figures` -> create bokeh figures for each observation point.\n",
    "- `interactive_map` -> create a folium map with observation points and bokeh figures for each observation point.\n",
    "- `section_plot` -> create a plot of multiple observations and a plot of the well layout.\n",
    "\n",
    "`stats`:\n",
    "\n",
    "- `get_first_last_obs_date()` -> get the first and the last date of the observations for each observation point\n",
    "- `get_no_of_observations()` -> get the number of observations\n",
    "- `get_seasonal_stat()` -> get seasonal stats of the observations"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "E.g. get the bounding box with `gw.geo.get_bounding_box()`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"bounding box -> {oc.geo.get_bounding_box()}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc.geo.set_lat_lon()\n",
    "oc.plots.interactive_map(plot_dir=\"figure\", popup_width=300)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can get an overview of the well layout and observations via `plots.section_plot`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oc.plots.section_plot()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### ObsCollection Attributes\n",
    "\n",
    "An `ObsCollection` also has additional attributes:\n",
    "\n",
    "- name, a str with the name of the collection\n",
    "- meta, a dictionary with additional metadata"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"name is -> {oc.name}\")\n",
    "print(f\"meta is -> {oc.meta}\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read ObsCollections"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Instead of creating the ObsCollection from a list of observation objects. It is also possible to read the data from a source into an ObsCollection at once. The following sources can be read as an ObsCollection:\n",
    "\n",
    "- bro (using the api)\n",
    "- dino (from files)\n",
    "- fews (dumps from the fews database)\n",
    "- wiski (dumps from the wiski database)\n",
    "- menyanthes (a .men file)\n",
    "- modflow (from the heads of a modflow model)\n",
    "- imod (from the heads of an imod model)\n",
    "\n",
    "This notebook won't go into detail on all the sources that can be read. Only the two options for reading data from Dino and BRO are shown below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read using a .zip file with data\n",
    "dinozip = \"data/dino.zip\"\n",
    "dino_gw = hpd.read_dino(\n",
    "    dirname=dinozip, subdir=\"Grondwaterstanden_Put\", suffix=\"1.csv\", keep_all_obs=False\n",
    ")\n",
    "dino_gw"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read from bro using an extent (Schoonhoven zuid-west)\n",
    "oc = hpd.read_bro(extent=(117850, 118180, 439550, 439900), keep_all_obs=False)\n",
    "oc"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# plot wells, use x-coordinate in section plot\n",
    "oc.plots.section_plot(section_colname_x=\"x\", section_label_x=\"x coordinate [m]\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = oc.to_gdf().plot()\n",
    "ctx.add_basemap(ax=ax, crs=28992)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "dev2",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}