{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connected sets\n",
    "\n",
    "## Import the BipartitePandas package\n",
    "\n",
    "Make sure to install it using `pip install bipartitepandas`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import bipartitepandas as bpd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get your data ready\n",
    "\n",
    "For this notebook, we simulate data (we set parameters to make the connected sets interesting)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.255266</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>0.395999</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.273443</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.404883</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-0.783187</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>-0.510697</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>1.994076</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>2.570977</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>0.535605</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>-0.202017</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t\n",
       "0         0  164 -1.255266  0\n",
       "1         0  164  0.395999  1\n",
       "2         0  164 -1.273443  2\n",
       "3         0  164 -1.404883  3\n",
       "4         0  164 -0.783187  4\n",
       "...     ...  ...       ... ..\n",
       "49995  9999  786 -0.510697  0\n",
       "49996  9999  786  1.994076  1\n",
       "49997  9999  786  2.570977  2\n",
       "49998  9999  786  0.535605  3\n",
       "49999  9999  786 -0.202017  4\n",
       "\n",
       "[50000 rows x 4 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "df = bpd.SimBipartite(\n",
    "    bpd.sim_params(\n",
    "        {\n",
    "            'firm_size': 10,\n",
    "            'p_move': 0.05\n",
    "        }\n",
    "    )\n",
    ").simulate()\n",
    "bdf = bpd.BipartiteDataFrame(\n",
    "    i=df['i'], j=df['j'], y=df['y'], t=df['t']\n",
    ")\n",
    "display(bdf)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computing connected sets\n",
    "\n",
    "There are eight connectedness options:\n",
    "\n",
    "- None\n",
    "- Connected\n",
    "- Strongly connected\n",
    "- Leave-out-observation\n",
    "- Leave-out-spell\n",
    "- Leave-out-match\n",
    "- Leave-out-worker\n",
    "- Leave-out-firm\n",
    "\n",
    "These are specified in the cleaning parameters dictionary under the key `'connectedness'`. We will demonstrate `'connectedness' = None` and `'connectedness' = 'leave_out_observation'`.\n",
    "\n",
    "<div class=\"alert alert-info\">\n",
    "\n",
    "Note\n",
    "\n",
    "Leave-out-spell and leave-out-match are distinguished by workers who leave a firm then return to it.\n",
    "\n",
    "</div>\n",
    "\n",
    "Note\n",
    "\n",
    "Stayers who have only a single observation after computing the largest connected set can be dropped by specifying `'drop_single_stayers' = True` in your cleaning parameters dictionary.\n",
    "\n",
    "</div>\n",
    "\n",
    "<div class=\"alert alert-warning\">\n",
    "\n",
    "Warning\n",
    "\n",
    "Connectedness is not necessarily maintained between non-collapsed and collapsed formats. Therefore, if you plan to use connected, collapsed data, it is recommended to set the connectedness level at the level at which you would to collapse your data, and to set `'collapse_at_connectedness_measure' = True` in your cleaning parameters dictionary. An example is given below.\n",
    "\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 'connectedness' = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t</th>\n",
       "      <th>m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.255266</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>0.395999</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.273443</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-1.404883</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>164</td>\n",
       "      <td>-0.783187</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49995</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>-0.510697</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49996</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>1.994076</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49997</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>2.570977</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49998</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>0.535605</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>49999</th>\n",
       "      <td>9999</td>\n",
       "      <td>786</td>\n",
       "      <td>-0.202017</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>50000 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t  m\n",
       "0         0  164 -1.255266  0  0\n",
       "1         0  164  0.395999  1  0\n",
       "2         0  164 -1.273443  2  0\n",
       "3         0  164 -1.404883  3  0\n",
       "4         0  164 -0.783187  4  0\n",
       "...     ...  ...       ... .. ..\n",
       "49995  9999  786 -0.510697  0  0\n",
       "49996  9999  786  1.994076  1  0\n",
       "49997  9999  786  2.570977  2  0\n",
       "49998  9999  786  0.535605  3  0\n",
       "49999  9999  786 -0.202017  4  0\n",
       "\n",
       "[50000 rows x 5 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "conn_none = bdf.clean(\n",
    "    bpd.clean_params(\n",
    "        {\n",
    "            'connectedness': None,\n",
    "            'verbose': False\n",
    "        }\n",
    "    )\n",
    ")\n",
    "display(conn_none)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 'connectedness' = 'leave_out_observation'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t</th>\n",
       "      <th>m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.255266</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0.395999</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.273443</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-1.404883</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.783187</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44856</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>-0.510697</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44857</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>1.994076</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44858</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>2.570977</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44859</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>0.535605</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44860</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>-0.202017</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>44861 rows × 5 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t  m\n",
       "0         0    0 -1.255266  0  0\n",
       "1         0    0  0.395999  1  0\n",
       "2         0    0 -1.273443  2  0\n",
       "3         0    0 -1.404883  3  0\n",
       "4         0    0 -0.783187  4  0\n",
       "...     ...  ...       ... .. ..\n",
       "44856  9014  876 -0.510697  0  0\n",
       "44857  9014  876  1.994076  1  0\n",
       "44858  9014  876  2.570977  2  0\n",
       "44859  9014  876  0.535605  3  0\n",
       "44860  9014  876 -0.202017  4  0\n",
       "\n",
       "[44861 rows x 5 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "conn_loo = bdf.clean(\n",
    "    bpd.clean_params(\n",
    "        {\n",
    "            'connectedness': 'leave_out_observation',\n",
    "            'verbose': False\n",
    "        }\n",
    "    )\n",
    ")\n",
    "display(conn_loo)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connected sets for collapsed data\n",
    "\n",
    "As mentioned above, connectedness is not necessarily maintained between non-collapsed and collapsed formats.\n",
    "\n",
    "Here we show an example that demonstrates this, then show how setting `'collapse_at_connectedness_measure' = True` in your cleaning parameters dictionary will give the correct results, all in one line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>w</th>\n",
       "      <th>m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.864156</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.076316</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.360893</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.256533</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.233623</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10891</th>\n",
       "      <td>9011</td>\n",
       "      <td>697</td>\n",
       "      <td>0.423780</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10892</th>\n",
       "      <td>9012</td>\n",
       "      <td>476</td>\n",
       "      <td>0.385739</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10893</th>\n",
       "      <td>9013</td>\n",
       "      <td>796</td>\n",
       "      <td>1.546216</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10894</th>\n",
       "      <td>9013</td>\n",
       "      <td>381</td>\n",
       "      <td>-0.333299</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10895</th>\n",
       "      <td>9014</td>\n",
       "      <td>876</td>\n",
       "      <td>0.877589</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10896 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t1  t2  w  m\n",
       "0         0    0 -0.864156   0   4  5  0\n",
       "1         1    1 -2.076316   0   4  5  0\n",
       "2         2    2 -0.360893   0   4  5  0\n",
       "3         3    3 -1.256533   0   4  5  0\n",
       "4         4    4  1.233623   1   4  4  0\n",
       "...     ...  ...       ...  ..  .. .. ..\n",
       "10891  9011  697  0.423780   0   4  5  0\n",
       "10892  9012  476  0.385739   0   4  5  0\n",
       "10893  9013  796  1.546216   0   0  1  1\n",
       "10894  9013  381 -0.333299   1   4  4  1\n",
       "10895  9014  876  0.877589   0   4  5  0\n",
       "\n",
       "[10896 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coll_conn_loo_wrong = conn_loo.collapse(level='spell')\n",
    "display(coll_conn_loo_wrong)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>w</th>\n",
       "      <th>m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.864156</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.076316</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.360893</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.256533</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.233623</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10878</th>\n",
       "      <td>8999</td>\n",
       "      <td>696</td>\n",
       "      <td>0.423780</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10879</th>\n",
       "      <td>9000</td>\n",
       "      <td>475</td>\n",
       "      <td>0.385739</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10880</th>\n",
       "      <td>9001</td>\n",
       "      <td>795</td>\n",
       "      <td>1.546216</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10881</th>\n",
       "      <td>9001</td>\n",
       "      <td>380</td>\n",
       "      <td>-0.333299</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10882</th>\n",
       "      <td>9002</td>\n",
       "      <td>875</td>\n",
       "      <td>0.877589</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5.0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t1  t2    w  m\n",
       "0         0    0 -0.864156   0   4  5.0  0\n",
       "1         1    1 -2.076316   0   4  5.0  0\n",
       "2         2    2 -0.360893   0   4  5.0  0\n",
       "3         3    3 -1.256533   0   4  5.0  0\n",
       "4         4    4  1.233623   1   4  4.0  0\n",
       "...     ...  ...       ...  ..  ..  ... ..\n",
       "10878  8999  696  0.423780   0   4  5.0  0\n",
       "10879  9000  475  0.385739   0   4  5.0  0\n",
       "10880  9001  795  1.546216   0   0  1.0  1\n",
       "10881  9001  380 -0.333299   1   4  4.0  1\n",
       "10882  9002  875  0.877589   0   4  5.0  0\n",
       "\n",
       "[10883 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coll_conn_loo_right_1 = bdf.clean(\n",
    "    bpd.clean_params(\n",
    "        {\n",
    "            'connectedness': None,\n",
    "            'verbose': False\n",
    "        }\n",
    "    )\n",
    ").collapse(level='spell').clean(\n",
    "    bpd.clean_params(\n",
    "        {\n",
    "            'connectedness': 'leave_out_observation',\n",
    "            'verbose': False\n",
    "        }\n",
    "    )\n",
    ")\n",
    "display(coll_conn_loo_right_1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simpler code\n",
    "\n",
    "Instead of cleaning, collapsing, then cleaning again, we can do it all at once by specifying `'connectedness' = 'leave_out_spell'` (or `'leave_out_match'`) and `'collapse_at_connectedness_measure' = True`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>i</th>\n",
       "      <th>j</th>\n",
       "      <th>y</th>\n",
       "      <th>t1</th>\n",
       "      <th>t2</th>\n",
       "      <th>w</th>\n",
       "      <th>m</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.864156</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>-2.076316</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.360893</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>-1.256533</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1.233623</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10878</th>\n",
       "      <td>8999</td>\n",
       "      <td>696</td>\n",
       "      <td>0.423780</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10879</th>\n",
       "      <td>9000</td>\n",
       "      <td>475</td>\n",
       "      <td>0.385739</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10880</th>\n",
       "      <td>9001</td>\n",
       "      <td>795</td>\n",
       "      <td>1.546216</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10881</th>\n",
       "      <td>9001</td>\n",
       "      <td>380</td>\n",
       "      <td>-0.333299</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10882</th>\n",
       "      <td>9002</td>\n",
       "      <td>875</td>\n",
       "      <td>0.877589</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>5</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>10883 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          i    j         y  t1  t2  w  m\n",
       "0         0    0 -0.864156   0   4  5  0\n",
       "1         1    1 -2.076316   0   4  5  0\n",
       "2         2    2 -0.360893   0   4  5  0\n",
       "3         3    3 -1.256533   0   4  5  0\n",
       "4         4    4  1.233623   1   4  4  0\n",
       "...     ...  ...       ...  ..  .. .. ..\n",
       "10878  8999  696  0.423780   0   4  5  0\n",
       "10879  9000  475  0.385739   0   4  5  0\n",
       "10880  9001  795  1.546216   0   0  1  1\n",
       "10881  9001  380 -0.333299   1   4  4  1\n",
       "10882  9002  875  0.877589   0   4  5  0\n",
       "\n",
       "[10883 rows x 7 columns]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "coll_conn_loo_right_2 = bdf.clean(\n",
    "    bpd.clean_params(\n",
    "        {\n",
    "            'connectedness': 'leave_out_spell',\n",
    "            'collapse_at_connectedness_measure': True,\n",
    "            'verbose': False\n",
    "        }\n",
    "    )\n",
    ")\n",
    "display(coll_conn_loo_right_2)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.10.4 ('tw-env')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "cc42643cd328f586448453625167b9fc1f3293a5a6342212b4e72707ccae656b"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}