{ "cells": [ { "cell_type": "markdown", "metadata": { "editable": true, "id": "vqv_wAtk0ibM", "slideshow": { "slide_type": "" }, "tags": [] }, "source": [ "# Python for Data Analysis\n", "Example solutions are provided in this file.\n", "\n", "### Research Computing Services\n", "\n", "Instructor: Brian Gregor\n", "Website: [rcs.bu.edu](http://www.bu.edu/tech/support/research/) \n", "Tutorial materials: [http://rcs.bu.edu/examples/python/DataAnalysis](http://rcs.bu.edu/examples/python/DataAnalysis) \n", "Contact us: help@scc.bu.edu" ] }, { "cell_type": "markdown", "metadata": { "id": "M3FbA7um0ibR" }, "source": [ "## Course Content\n", "1. Python packages for data scientists\n", "2. Data manipulation with Pandas\n", "3. Basic data plotting\n", "4. Descriptive statistics\n", "5. Inferential statistics\n", "\n", "\n", "## Tutorial Evaluation\n", "After the tutorial please submit an evaluation by clicking on this link [Tutorial Evaluation](http://scv.bu.edu/survey/tutorial_evaluation.html)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Python packages for data scientists\n", "- [NumPy](https://numpy.org)\n", " - Introduces objects for handling n-dimensional arrays such as vectors (1-D) and matrices (2-D).\n", " - Introduces functions to perform advanced mathematical and statistical operations on these objects.\n", " - Provides vectorization of mathematical operations on arrays and matrices which significantly improves performance.\n", " - Many other Python libraries are built on NumPy\n", "- [SciPy](https://scipy.org)\n", " - An enormous collection of algorithms for statistics, linear algebra, optimization, differential equations, numerical integration, and more.\n", " - Developed and released with Numpy. \n", "- [Pandas](https://pandas.pydata.org)\n", " - Adds data structures and tools designed to work with table-like data (similar to Vectors and Data Frames in R)\n", " - Provides tools for data maniuplation: *reshaping*, *merging*, *sorting*, *slicing*, *aggregation*, etc.\n", " - Easily allows to handle missing data\n", " \n", "- [SciKit-Learn](https://scikit-learn.org/stable/)\n", " - Provides machine learning algorithms: classification, regression, clustering, model validation, etc.\n", " - Built on NumPy, Scipy, and matplotlib.\n", "\n", "- Machine Learning libraries\n", " - [Pytorch](https://pytorch.org/)\n", " - [Tensorflow](https://www.tensorflow.org/)\n", " - [Jax](https://github.com/jax-ml/jax)\n", " - For more info on using these on the SCC see [this page](https://www.bu.edu/tech/support/research/software-and-programming/common-languages/python/python-ml/).\n", "\n", "- Pandas alternatives\n", " - Pandas (used for this tutorials) is very popular and it has some alternatives\n", " - [Dask](https://www.dask.org/) - process large scale data in parallel, built on Pandas.\n", " - [Modin](https://github.com/modin-project/modin) - another library for scaling up Pandas to large datasets.\n", " - [Polars](https://pola.rs/) - Similar functionality (but not built on Pandas), fast, parallel processing, gaining in popularity.\n", " \n", "### Visualization\n", "More in-depth look at visualization in the `Data Visualization in Python` course.\n", "- [matplotlib](https://matplotlib.org/)\n", " - Python 2-D plotting library for pulibcation quality figures in a variety of hardcopy formats\n", " - Functionalities similar to MATLAB\n", " - Line plots, scatter plots, bar charts, histograms, pie charts, etc.\n", " - Effort needed to create advanced visualizations\n", "- [seaborn](https://seaborn.pydata.org/)\n", " - Based on matplotlib\n", " - Provides a high-level interface for drawing attractive statistical graphs\n", " - Similar to the ggplot2 library in R\n", "- [plotly](https://plotly.com/python/)\n", " - over 40 unique chart types covering a wide range of statistical, financial, geographic, scientific, and 3-dimensional use-cases.\n", " - Built on top of the Plotly JavaScript library\n", " - Can create beautiful interactive web-based visualizations\n", "- [Datashader](https://datashader.org/)\n", " - Used to create visualizations and plots from very large datasets." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Loading Python libraries" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Ea4C8rUR0ibU" }, "outputs": [], "source": [ "# Press shift-enter to execute a Jupyter notebook cell\n", "# Import the Python Libraries used in the tutorial\n", "import numpy as np\n", "import scipy as sp\n", "import pandas as pd\n", "import matplotlib.pyplot as plt\n", "import seaborn as sns" ] }, { "cell_type": "markdown", "metadata": { "id": "ca3M2mv_0ibX" }, "source": [ "## Pandas\n", "The main focus of this tutorial is using the Pandas library to manipulate and analyze data.\n", "\n", "Pandas is a python package that deals mostly with :\n", "- **Series** (1-D homogeneous array)\n", " - the array has 1 data type (int, floating point, etc)\n", "- **DataFrame** (2-D labeled heterogeneous array)\n", " - each column has a specific data type\n", "- **MultiIndex** (for hierarchical data)\n", "- **Xarray** (built on top of Pandas for n-D arrays)\n", "\n", "The Pandas content of this tutorial will cover:\n", "* Creating and understanding Series and DataFrames\n", "* Importing/Reading data\n", "* Data selection and filtering\n", "* Data maniuplation via sorting, grouping, and rearranging\n", "* Handling missing data\n", "\n", "\n", "In addition we will also provide information on the following.\n", "* Basic data plotting\n", "* Descriptive statistics (time permitting)\n", "* Inferential statistics (time permitting)" ] }, { "cell_type": "markdown", "metadata": { "id": "vYNDrGEL0ibY" }, "source": [ "### Pandas Series" ] }, { "cell_type": "markdown", "metadata": { "id": "hQpjwDv_0ibZ" }, "source": [ "A Pandas *Series* is a 1-dimensional labeled array containing data of the same type (integers, strings, floating point numbers, Python objects, etc. ). It is a generalized numpy array with an explicit axis called the *index*." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "iPJvEWlG0ibZ" }, "outputs": [], "source": [ "# Example of creating Pandas series :\n", "# Order all S1 together\n", "s1 = pd.Series([-3, -1, 1, 3, 5])\n", "print(s1)" ] }, { "attachments": { "e141912c-cf3c-4599-a21b-4c3c3f2f6785.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAMUAAACUCAYAAAAj6u2iAAAgAElEQVR4Xu1dBVxUWRf/M3QJgigmqIhrt58tiq6ioKyoWKAotZSgYid2gIEtdq2FuoC5roFrx6qIgoktGCA99b77BkeRRZ1RBmZ49/1+X+Dc9+65/3P+95xzU41hxAzoQxGgCHxCQI2SgloDReBLBCgpqEVQBAogQElBTYIiQElBbYAi8G0EqKegFkIRoJ6C2gBFgHoKagMUAbkQUHj4xDD3ML9DbaROYDCvB50SkUs7xVSY6ugHh2QFiTHwrD8fbW/Hwr2W7MZNAS8my/6JauTRUXzMIWhZ94SVHDbwE6KVyKvUU5QI7MpVqaykkLWccrVOfmlkJoWYiYG7YRRcM9agAxMt+f+Nd4mxZ95dnD17Ft09Y3B4jZ1EguxoD9g6rMdNrXZoM8wJVeKDUP5j+CT9Tb1dO/DU6sJl/RqJ52H/vXVUH/xLviGpq8N8tNkgn1eSv/ncfeNbOrod4wkP+wiwOrp+CfDbH4u5dolY6DUc49b9A/2Pel2zJhBZMR7/KavqYfIPk2I4zwHloxOxoKeVxIhHaOeFVq5MGGwa7MfwuDyD5icuhk3t0egQw2COXR6ZWGLZIO8376YJn/7e59keF3tvxP8OumFH3Y3YF2jFXatVYMslOpFDR93cedhwJhCWTOIX+WH+jlKqT2nZ6kS/qvr8MCnyG3eeWx2BcqRnH5zgjnoL6uMEAZEFJr/LnSZ2hx7xIO1IDyR92J5oyUcCsSCzZPuXeB3WY9BHMQiwXuJrOmJ7eVZnhw7dQ/zBgyD/jcgd9bCXdGQdC5CCle5rZVmSqOpT7KTIr4yCoLE9mO2I/ZKwasaZPG9Cn6JH4FukCLHK8/TtIjfA3Zoh3iEanYk3n1UIKaQep7Cyqqy7IidFYeGTJwmfzD+GT9Iwiw2t8oddI6xY15znbVpGt8G4u5M/5ShFbxbc/mJh4ZNUR6w3z99x3Vzsic5TeBJPwRr6Pk8eLjvmDa8XJFfBsqqKcpGTgjV2aaL2kIRJDes6oTNJtKXzFJI8YsRoHEI7ZJDQyYf0SGxewoK9o26iJI+QhmOpE2Lp3IaCLOtrOpIm1CGb1dCkJdBifB+kOt/9lPdJE3QDz8VYvboH9pLk+2tlFSS6wj8rMykULgmtgCKgJAhQUiiJIqgYyoOAUpDi5I0HqGdRAeWNDZQHGQ5Kcu5OEtrUseBgy79scomT4tr9Z/BcsBV6ujpY4t8PTa2qcF4pJQHA4shT2BQVi2EO7RHUx6YkRFCaOkucFMmpGfAN343Eh8+hqamBKW690Lt1PaUBiAuC7D7zL2ZvjJI0dUngIHRqVJMLzf5qG0ucFKxkuXwhgiMO4vTleImgTl1aYurgbpxWTHE1ng1dg1fshkBAdDDEDkNsmxdX1Upbj1KQQopOxJELCN91XPJn68bWmO/eG0b6OkoLnqoLFpf0Ch4kdM3KysEgOzI31N9W1ZtUJPIrFSnYFh25koAZG/+UKKqSuSnWjhqEqmbGRdJY+pHPCDxNScWw+Vvx5m0qOraoi2U+ThSejwgoHSlYue69eAPvsJ0Shenp6SDUrz8dFSlCk03LzIHr/C14/PQ16te2wAbS8WhraRRhDar9KaUkBQspqzgfkoDHJSRJEKbxbtEYGpu/eYfvwrW4hxJP/Mfk4TRELQCt0pJCKuekTTGIPn1N8qd9x6aYPaxn0VgHR78SvO4gjp27CaMyBtg+2Y2GpoXYgdKTgpV524krWEIScHaEhHX3K/37097tB0gtnYtgh743TXJDfQvzH/hK6X9FJUjBqoGdbR29fLckAS9naozVowaiVqVypV9DRdRCOhchO5AqQwq2SeyIiR/JM9gEkU3Ap5GJvu7Na8veWo6WpHMR8ilepUghTcDHkYm+8/8mSlrq79wV7t1bfWo1m6Anp2Vw2ouweUNwP1vJWjI6FyEfIdjSKkcKaRNDth/Fvr/IhgzysOPsC8lEH/sMD9uB2lUrcHZGnO0UOvgtlISYwQN+xcI/jtG5CDl5obKkYNt58PxtzCQTfWwCbl2jMqwJGdiRKjaRPLTAn5OrbllMpq6N/MIM6FyEfKxQaVKwTWVX2QaG70Hah4wvWu7VpxN8HD4fkCAfLKpbWjrkmr8FBUNM1W1d8Uiu8qSQeoyCvSM7Dn98oT/nZmptx4RLwqWCD13bJDuhVJ4U7IhU/+nrJEO1BZ8Qzz6cWobOYmE/Nvw/OLDhZCeSd0mTb9nNg5slVZ4U7Iz3UTJDy+YVBR9LkmMcDPHkjGbzz0WwjWaXcQwmy/AdWtWnk51yWIHKk0LaVja3YP9zk2xWuv3wxacQYtVYV84sJmTziZNkTwrrFQZ2akZ3McpBhPxFSw0pCrafDSWu3X+OMmSSr7CdZIwoFzk5QogYNahr6kBXm/eDEBb9a2Ihn2y8EkAkzvu2upYOdLTUofadqthl963JHmu6B+XndFJqSfEtWMSiHMSuHoqwqGd4m6UD/Rq/YfocN7SupP9zaBbR289jlyJs1Tace2YMLdEHmDYbDL8AP3Sy4n2XGEUkAqc/wzlSMGIBzq5ywYJ/LODq74qGpuk4HDodp0xcsWC0M6zLqZe4QfAz3iDlzVtk5LIkeIPT65bgZm5HuMzwRUsTepSoohXEOVIIci4itHcYeJ6z4OtkDX1yFGTa9eXwCbqMtpNmYWjXapJ/U6bn0b4ghJ4BOnktgVNd5ZJNmXAqKlk4RQoGImScXwrnMckYsWUCHC30kPv8COaOX4Hoi4/QJeRPTBjwC0w0St7wxCKS74hEEIrisX3iHiRVaAuvIHtU0S552YrK+JT1O5wjxYeTC+G8URcB3mY4v3IPLj7SQkf/geAdXwy+TTi8+jeCuYINjxx2D7FARBJpYuBs9swm+xrqUFf/nEq/OBuOJau34fS9XBjWdILvmLFwbKpNc4piYBLHSMEgI3Y+7CdEIUdcBb8FBKBbgyqoXlsbR4IHIbFpGDz7N1E4KcT8m9gaEo6dJxLAN1CHIL0OnCcGwK1XnU+hG//DK7x8lYzUbB4yE3Zjy3UBWvUeh2Gt6CEOiuYFp0jBgpmdsAFewxZD034lZozrgCokVBImR5E94FtRxW8u3B2sYKSm2BCFEWfi9ZPnSH6XBTHrHUT6MLOoDHNTXRSW5gue7sboObHQbeuD+UPqKNomOP99zpFCnPsWB0J+w4Y3gzBjljuamKrj0po+mH6jE2aM90ILS22U9IwFQ8Kqz7RkEL9vAibueo+OXiEY3ZVuIVU0azlHCom3eHceK8dMxdHHfAjUdKBZpwcmBriiXU0TaKgr1kt8T6HsYMCtyAVYuSkad7MMoS5Ih75VZzi5eMG+ZWWY6n1vCu97NdDfv4cAJ0nBgpL+IgGPXn4guQUPOua/oGFV5Zi4Y31EZvJjPHmWggwRmcVmRNAxq4Fa1c2go2RDxd8zLlX9nbOkUFWFUbkVjwAlheIxpjWoGAKUFCqmMCqu4hGgpFA8xrQGFUOAkkLFFEbFVTwClBSKx5jWoGIIUFKomMKouIpHgJJC8RjTGlQMAUoKFVMYFVfxCHCWFAzzHsdCJ2L3tRoYPN8bNlUNS3zNU0F1P/hrCZYtv4OarkEY3qcODOiMtuIZQWrgJCn48TsxKvQijPVfIvpvc4zeNRmD6pkVukK1WLRQoBJR2iWsXPQn7r9+jDu3RKju6I1542xQlpKiWNTBOVKI+C9weNoq3GvQGh1N4zF+1lsMXT0KA5SEFAyTgysbNuBksjbadTHAvkWxyGreH3ODO8KYkoKSQhEIiEVZeBqXDLNG1aF+ORS9Al/BZW0wBioLKcgq2eR7z6FmWh6mOucw2WUH3rVyxXxKCkWYQ6Hf5JynyI9C1tmF6D0uBUOViBT55RO+jMYE/4NI/58L5lFSUFIoGgF2iXbW2UVwLAFSiPnx2BMWgcgzDyDU54GfaQ3HkR4Y2K0W9PKFSMJXMZjgd4CSQtHGUOD71FOwpFg3FoPqliu20SdG9BZ3L91AwpN3EGmpQcQ3Qe2WjVG3ugk085PiNSGFbx4p2PDJiOYUxUIPTpMi99pS9ApIhse2KehrqVMsgMtTiSjzb0xxI3dvtB2BpSObg17/Lg96P16Wc6RgxCJc3zUKq488w5sXz3HjPh+V61ZHZaMO8Jznhg7VypT40OyrSxsRsTUSV5Jy8CThNQRly8PSrDHs3Nzg0qc+DKnH+HGLl+FN7pECYry4cQgXE9KQQw6UMSIxfXZGNoTkyJuWds1haVzyBxekP7uKy9fi8DzDAGUMtaAmzEZamgFqtWiOJvUqQJuSQgbT/vEinCPFj0NF3+QKApQUXNE0bafMCFBSyAwVLcgVBCgpuKJp2k6ZEaCkkBkqWpArCFBScEXTtJ0yI0BJITNUtCBXEKCk4IqmaTtlRoCTpGAYsrlo5lIce5SCbB0DNO07Bm421aBbwocrF9Taqys7sHnLfVR1dIVj5+pfLBaUWcO0oNwIcI4UjEiAWwd/x+ZTlqhQ1QAaWddw5l8reIQEoGuDstBSgtlicU48/lh7DHF3z+Of2GxYDx6NBePpzju5rfsHX+AeKcjap8cXdiG1xgA0MeeBYe5i6UA/POwwHePd2qOSbgkfxc/k4tYf67H32gc0sjHF6a1xEDbvgzljbOjOux80cnlf4xwpCgLEMEnYMLAvrpCrvSb6dUBVvZImhQCPL96BuLIlqptfx5QBW/GG7ryT165/qjznSfH0zFgEzniGntPCMLB9Regq+GovebRFd97Jg1bRleU0KbLjyV1yIxcives8zA7ojGo6ynVLEN15V3SGLs+XOEuKrPg9CJmyFo8sBiN4vAualy/sCkZ5oCz6spQURY+pLF/kJCmEr89i7qQVeFK1H8ZPc0JNJRhxKkxZYv5pTHXZhQ/tPbHMr7Es+qRligABzpFCJErEft/fMXL3czTo0xvNKqiBnyuGmlpD9B3tiObmBsW2V/tr+ntz6wD2Rf2FuKRkXI19gNyKVmjxS0vY/OaIXl2s6HxFERj+tz7BOVKIRY/w17qjuJeVhSx+DnIEIjBigMerC3uv7mhYXr/ESfHu7mHEHItFYqoxTIy0AUEG3r8zRTO77ujank7iKZgT3Dw2U9Gg0u+rNgKc8xSqrS4qfXEgQElRHCjTOlQKAUoKlVIXFbY4EKCkKA6UaR0qhQAlhUqpiwpbHAhQUhQHyipeB8Pcw/wOtZE6gcG8HiW7YLI4oKSkKA6UVbwOeUgRH3MIWtY9YVVLfvL8zLtFCTElRVGiWUq/JSspZC1XGEw/825Rw845UjBiMR7ERuDwlTdI56tDnacF6+5u6N3IuMRnsqXKfZ94Akf/Pov77/SgQTYdmTbsgZ6/NkMlbfl73x81mOxoD9g6rMdNrXZoM8wJVeKDUP5j+HQ7xhMe9hFQb9cO1y8BfvtjMdcuEQu9hmPcun+g//GdNWsCkRXj8Z+yBUMwlhD53201tA6q3Vn3qT62Daw8raP64NpqMdwNo9B4lxh75t3F2bNn0d0zBofX2EmaKpWblY2nVhcu69fAXU6vxUFSiHA7Zg52nH6JtBwNsrzjBZLfNcCwoADYNDVRiv0UKTf2Ytf+o4hL0YeG+A1S3uugZTdfDB7UFObFQAx+4mLYNNiP4XGxEoOS/F17NDrEMJhjFy0xSteMNbAhCynZ37q587DhTCAsmcQvcg8xE/PVstULLMIs6CmkJPj3o7Hv8+ThsmNe/cN5DkhfnIh9gVZk5+TnfKcw2bybJnySVdYOgnOkKAgMw3zAbo/2+Nt6KSb5dkI1/eLrjWVV0u2IoQiLLQuHqUvhWFPx8rEGWW9BfZwghs4ab0GDZf8+dOge4g8eBPlvRO6oh72EJB0LkIJt39fKsoTK/xSsgyXUCO35aHs7Fq5M2CfiWTBfkpL9xs3FbeF6dzLOO0RCj3i3dsRLSB/Wky35SG5Z8aakYJKx1T8AD9oFwsehNSooGSkYRoira6Zg55Oy6PX7OHSsWrKkCLEKk3iRdpEb4G7NEO8Qjc6kN55VCCmkHqewst8jhdTYZ6htxjRmKNj/ZT1DQe9TkBT5ySwrCQqW4yQp2Pvuks5txrFrb/D+5XVcfuCAmaucUaes8uy8S3twmvTUF5D4Ogm3rpZDZ9cAjHAo96N6luu9wsInTxI+mZPwaZrY/QsvcnOxJzpP4Uk8BWvo0jCHzRsKepyCZQsKlf9d9jdWjq7uCajJxKPNhrxQjiUFGz6Vj07Egp7/DZ+k3kVaNv/fsoLAUVKIEXdgOraceIb3gjIwq1kNDVvZo0dzaxjpygqdYsul/LsLO/fG4OoTLZhWq4TqzW3R+X9tUa9i8VzyJU2mH5JQpGFdJ3QmiTY7TyFNqEM2q6FJS6DF+D5Idb77KW6XJroGnouxenUP7CXJ99fKFkQw/7tsks6GbixRZqrFQJpbSD1F/ZkM9kfe+U+izRLJe8RoHEI7ZJDQyYd4NJY88jycJEVBgJ7tcsGgo00xd5oX2lroyYNfsZTlx6+EV+glVOkxHTOdLIulTmWopLBh2sLCp6KWlZKCIPp4uw9cDllh5mxP2FgaFDXGP/29nFtr4DXvKio5BGPugFo//T1V+IDUaxjlG25l5aakUID2GHE6bp+IwuVHH5DL8KAGcn3vtYfQbumKwP7tYG5YsnkFm++8unUK5y7fxmuBNtRFOXj16Bae6zSDs+tQ2NZSvltcFaCmEv0k5zwFI07BiZXzEHktBZlidp5CHQ2cpmJkz6olfisqawkklUTi8XXYvvckHuYaQF2YAbOWg+E5rBesyyh+5KlErVFJKuccKZQEdyqGEiNASaHEylFF0c7dSUKbOhaqKPonmSkpVFp9yiX84shT2BQVi2EO7RHUx0a5hJNDGkoKOcCiRb+OwO4z/2L2xihJgSWBg9CpUU2VhYuSQmVVpzyCn7zxAMErdkMgECJ4iB2G2DZXHuF+QBJKih8Ajb7yGYG4pFfwWLAVWVk5GGTXBuP626o8PJQUKq/CkmvA05RUDJtP7s94m4qOLepimY9TyQlThDVTUhQhmFz6VFpmDlznb8Hjp69Rv7YFNowaBG2t4lmXpWicOU0KhnmHG7E38F69Cho1qwETHeU4jl/84RnibsfjwcsMQI0HsZAPXfO6aNSoASorwQReLl8I7/BduBb3EJXMTfHH5OEw0i89M+2cJQW7nCL51HSMm3wIF7UcsXJzADpVNVR0JyTT97Mur8CY0F24nGIGSxN1iHIzYNK0H0YMH47W1Up+Vjt43UEcO3cTRmUMsH2yG6qaGcvULlUpxFlSCN6exKzABdCqZoUTLxph6rR+ZDGgkVLoLf2fxVh8SQdNHXxgb1XyJMgPinQuQlNTA5smuaG+hblSYFaUQnCSFIzwDY7MDkJEWjeM6vwMi6IN4Tt2CLrUUB5SLLmsh+aO3rCzVB5SlKa5iG+RiJOkeH8xBP4z0/DbzMnoItwG11UM/CYPRVclIUXGuTBM23sf6pY9YEPmwMS6FiTnaYiqRiVHkNI2F0FJ8REBNo8QpyVhY9hEvGs9C2O614DgVjj6hWth3CxPtC1fssvGpYri3z+CVRHbcCQuHdo8IdJyjdHC3hm/D/gV1c2KJ6Fl84bgfrYob2xAblQqfXMRlBRSUpBDAJL2+KLfek0MGtYD9U01kXv3IBae4MHeqTO6d+qC2lUMoaVE1wazomdcX0o82XEY95qFBV5NoKXgO/rY4dYOfgtRztQYwYSIC/84VurmIigppKQQC3F5uz/WHH6OlAx1speCgSgjBY/fMChXwRS9/VdgeHcLlNEouTClMGUJHm6Hb8BhqHfxx4LAVjBUMCkOnr+NqWsjvxCltM1FUFJ8A4Gc8/Nhv0IbE+Z6w7Zq8YQm31IIG+KlPY1Hwv1nSBNpgifm49HFAziZXA1OHiPh1FC/KAdaCv2WdMg1/4/+zl3h3r2VwutWhgo4mWjnBz731k5MjNLAQLeeaF6x5A8tYHfe3T4YirVbDiMh2xA8svOufCtX+PkMQwvz4vFgtmPCJeFSwae0rG36HvE4T4rvAcS139n1TPZjw//TbHZeohNZ3yRNvkszLpQUpVm7P9C2/HMR7OvsMo7BXVrCoVX9UrWUg+YUP2AcXH2FzSdOXo6XeIWBnZqhqVUVzkFBPQXnVP7tBh+5koDWZI91aVrgJ6+KKSnkRYyWL/UIUFKUehXTBsqLACWFvIjR8qUeAUqKUq9i2kB5EaCkIIiJs97iyXs1lCtnDANtnrwYKm353PQ3eJ2SAQ1jc1Q00SHn5tJHFgQ4Twp2WUU2WardP0ITAZOG4teaeXsqst49wwcYw6QMWSCoZGuhZFEsWyYhahpGzTiGqsOWY7lfM8izg5rd7ZealgpGrzLKGRROJ4bJwev7iXidxUBdSx+mlSxR0ejLWsS5qXiS9Aa8MhXInIehXDLI2s6iLsd5UrCA5l5ZAqcVavCbNAzdrfJIcWVlT2wW+iDY3R7V9IpneUVRK/f1xfWYF34GpvbjMWFAHbkOkM65H4PV69eC6b4fQR3/6z35GeS09gsHsXPtFpx7KZTsIXfwXQhfm88btRhxKu7snwH/kGvQ6RiAiGV9UVHBixmLAkPOkkKQ+hiPX2WCL1ID/9Y2jD9UAcEz3NDBNBNPXqbiwmZvHBb3g0vvLqhWRgyUMUd1Mrurw0tH8otMaJU1gZEBWbBHtMCQa33fP38LkX5ZmJTVRe7blxDoloEoNQ3v3r1DjrohzKuyPe7nXlSc/RYvnr9Eao4aeLpmqFqlAgwL3Hya8yEFb9KIwZUtB1NSl6wPk0tkfP0SKR+E0CAymZmZkfc/152b8R7ZQiE5FEGLLD58inQ1bRiZkJ68YhnwBNl4l/wMiVeO4kD0fohbLYRbK22INfVgVK4SqppogxG8xvmd07BkvwaGLlqBnh8vp2QITOScBckjuVLg9j6snxmGF+U74DFaYt1SJ1RWsmX5hWHKSVLkfEhC1Ap/bDmZig8CLajz05Bu5owlK3xQ8/5yjFv6F+48eg6+hhExfD3Sw4qg28obi0b1h7X+aUx1XglNp3HwdWkOM3UGwjfRGOe1H6aOgfBzqY9Ls/rhGK86BAmZSLp3HS+yDdBqxASMdLdBdR0ehFkpuHRwLtbsjMW9tzoQ69VBP38/eHZvBEOtPK/ELgy8EuGNMWufodPYZZjeV/YrqgRPziBieRg2n3iEHPP/wdl/BiZ0r/hJ/4lHF2FHzD/IJMaadng3bvK1ULWJA/xDg9BWcAeRq2ZicdQT8PnkNBHdCjDWzAW/7C+wcZmIGX0skXr3OLasWoz3rRfBuQm5Q4MwQdfQlJDKSOKNWEIIU25j59JFOMZvh0GNkrDqcmOsDu2LygQvZX84RwpGlIFDs/tivWAIZvv2RXVjNWRfXAKXTboIIFtSu1jqk+MfRfhnRS/sEo6Av0tvWJkQNfI0SG6hDjW19zg0zRU7Bf0wKtAFTSuo4cXBUQiIrQ4Pdw90rQ0cmzME83ddR+0R6zHWvRVMbyyH3/JnaOkyEj52FkjY4oeZV03g7DUJjnV18fT4RIzbm4UuQ6ZgWHvTPO9DSHFr9xTM2fESrb2mYSR5T9aHEYshFAqQfScSs3eeBdN8HBb0rfbp9QcnwzBj9Gpktp+EqfOcUeP5Saza8ieSagYi3KUWOVJHiDdxB7Bp+2aIO2/FKFs9Ig8PGhqa0FDPQcKpbdgw5QR4dpa4dOICBDk8mFRqhSGzJsDxlzLk7u/3iItcgDlb9eC9zAOG55dj6vlGWBPWj5JCViUWZzl+5inM7bUV5hNnwMW2Klh1515agr5rePCf+DnRPh9uh21Cb4xyc0RN4y97N378Wgydl0JiaA8MbKmHyAkuuGnxO4YN7QZL3XTETB+Is3qDMdhrIBqQdxnmATa4zcezuoPwe/AvOO41EZf02+A3vy6wIAwQ5ZzDylGnoNPNFWMC28PkY9wtFokgEhNzJBfLqKvLP3YkuHcAMzf/DX6TMZjn9JkUdw7NxrYDD9BkzHr0tVYjG61uYMcKsv2V3xMRUzpBl9Sffucg1m2MgKjbXgTban9SEZtcPzi9HBMD/ob5+IWY/1sNiAUvcGHDSiw8WQOh+3xg8fQCNkWEgm+7BUGdRLi9ZyHGn2+GTYsdUY7mFMVp7t+vSzLSdH87gtzuw35DAOxqmUhGQ3LOL0K/dZrwJ4m2dPQpdsmv+IPxReBQR9QyKXgR+h2sHDAP6T2C4N4zGUvn3EHz/s7o3cqcECBDQorzZd3h6u6I2gYsKR5i47BQPG84AD4jy+EPr3HYei4Z2lUMSK9KBFATIeu9IWy9xsLXs32RJaP8u/sQsvU0hE3/S4od0UloHrwOvauT3Ydp17F99U4S6vTAuo+keH8rEuu3rAfDkqLL5ytjGSYLd49HYPX8JPTYRS59L8e2T4jH5zdh+ZQHsIvwhcaR2Qi5UAezp/ZCeWEy4g5vRfiNXzBrSi/UrlgNZXWVO4TiVPjEkiLn0R6McbuIduETMLAB6bfIHXjn1wVi8klrTJzrgy7V8w5EO7e0OyIy3THetx+sCzlF48amQVj7tBs6GiUgXq8Nevd0QJOKrIHkkeKCqSeGe/dCDZIjCNOiMcntb5Tv6wH/gabY7zMBl0y6wm1iP9RhO2E1NTKHwEpHkm519h6+vIdNiNMyRdA2NIaRnjwDqnnvCx79iVkbT0LYfDxm96rwqddgPcX2qMdoNiYCv5Ek+RMpBIQUk/M8RertA1i7YR1ybXdjSo/Pu/0Yhk88xTKETT2Nxov/hGczIjn5t8STq7AiNA3dQgchbfNQrDirDR0tHvmSCPzMD3iTLoblLw3hMmUzBjf+fgdWkiU4RQoWaDH/MXZN9MZxgxH43bkFjJ7sxbhFfyBBrTdWRgTBplre7ahPD02C/zYxfhsxEB1qlIGangmZADOEtmaeyQpTj2Kmxw6ce6GL4asnoHcDS0koxpLi0J+rL/UAAAODSURBVIxB+IvpgZ797VBTT4wHx2ZhzV1LDHUfA/t6unhwYCymHM+BjYMv7OobgCgBjLYhTI2NoKeVZw5sTnF1ox/Gr3uOjmPCMKWP7Pc98DPf491bQqjEowjffwHCeh4IdqgF47KmklGo+OgQbDnwiNyBvQlO5LA1UdpVbFm+HUcE9tg03VZCCuHrK9i8fBmOZv2KaX5toauuDX1jM1Qoo4mspH+wesEEnNN0x9SRHaCX9gh/7duG8xrOWDS5C4yFuRCKyHdFYoL3K1zftxIhlxthVWg/WOrpK/28D+dIwRpczuM9mBy0EZeTs2BQpx8Ce2Rj2wVTjCCJdweLPE/BiF7jwKwgbDj1CO9FWijT5ncsCuqLuhXyemyJ8U9ywqIXnTF9xih0tMgbMmWYTByfNxyhB+4gVbs89JEFcZ3+mBg0HLbWZHSGHJbAiDLxz+ZpWLXjDB6Ty+21xZmkTF+MDfRCj7p5pGRJcWPneIRseYG2vrMw2t5S5s7z4anVWB62ERdT80bOGHIWLVOlDZy9x8O/c3ncO7EUkcefoaH3Islha6IPcdi/lewDF3RGaGBb6LDkJsn6y5uRWD5jIc4kkzGlcnXQaegkzOxTQ+LTUu5EYeXUufj7tQZEmuVRz9YTM4O7ooLml3MajPgdbv8ZgXlXG2DJzB40p5BZi8VckDW43IxM5AjEUNPURRk9HnIFDBld0SCjK58TWnFuFjJz+BCy4++knKGuDkl4Pw6Zil9ix9jhuF1jFDyGkvOY9D/+Oxs+zRiIM9r94TSExNAkp1Ajs71l9DW/WGbBCHLJnQ7Z4DMkdCKeAhra0CPf19L4XL+IlJHIpakFrQLG9i3IRPwcZGfnQMCo57VHLIJQTQM6OrrQJSGNiJBEKCSz0Jo65Pc8AgjJiJOIlNfR/nzINIsTPyMD2QQnhiT7Wjp60P/4O0sMQWY6svjkNzXym64h+e2/UknO2hIIyHwQj9SvoRJLTTjpKX6Gg8JsMiGXlonnJ5ZhzgldDAjwQZ/GZp+UzXqQqMlOOGfijeG/94G1is6G/wxGqv4uJYWcGkw5E4bxy/7Ev/frwCd8PAa2I7lEvllaSfi0wAOXDAfB2cUBtQyVe6RFzuZzojglhZxqFudm4gM5QU8g1oGxiQE0SY6Q/5HM5mZnkXCFhDxaZLa8wO9yVkeLlwAClBQlADqtUrkR+D95EMGw4MIZngAAAABJRU5ErkJggg==" } }, "cell_type": "markdown", "metadata": { "id": "YPctLCcw0ibb" }, "source": [ "![image.png](attachment:e141912c-cf3c-4599-a21b-4c3c3f2f6785.png)\n", "\n", "We did not pass any index, so by default, it assigned the indexes ranging from `0`to `len(data)-1`. Contrast this with a Python list, which always has an implicit index that counts from 0:\n", "```\n", "x = [10,20,30]\n", "y = x[1] # y --> 20\n", "```\n", "and also with a Python dictionary, where the keys act as an index:\n", "```\n", "x = {'a':10, 'b':20, 'c':30}\n", "y = x['b'] # y --> 20\n", "```" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "v4vAbdqv0ibb" }, "outputs": [], "source": [ "# View index values\n", "print(s1.index)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s100 = pd.Series([-3, -1, 1, 3, 5], index=range(0,10,2))\n", "s100" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "s3I3__b90ibi" }, "outputs": [], "source": [ "x=s1[:2] # First 2 elements\n", "x[0]=-100\n", "print(x)\n", "print(s1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4jsIOlUR0ibi" }, "outputs": [], "source": [ "print(s1[[2,1,0]]) # Elements out of order" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "qGn-aAvx2t9V" }, "outputs": [], "source": [ "type(s1)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "WIft0v6r0ibl" }, "outputs": [], "source": [ "# Can place filtering conditions on series\n", "s1 > 0" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZmlDR_tj0ibd" }, "outputs": [], "source": [ "# Creating Pandas series with a custom index:\n", "# fetch a random number generator object.\n", "rng = np.random.default_rng()\n", "# select 5 points from a normal (Gaussian) distribution.\n", "s2 = pd.Series(rng.normal(size=5), index=['a', 'b', 'c', 'd', 'e'])\n", "print(s2)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "srVyZ9ik0ibe" }, "outputs": [], "source": [ "# View index values\n", "print(s2.index)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Print a value using the custom index\n", "print(s2['a'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "vX-IUZb30ibf" }, "outputs": [], "source": [ "# Create a Series from dictionary. The dictionary keys\n", "# become the index.\n", "data = {'pi': 3.14159, 'e': 2.71828} # dictionary\n", "print(data)\n", "s3 = pd.Series(data)\n", "print(s3)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sOHbkXC10ibg" }, "outputs": [], "source": [ "# Create a new series from a dictionary and reorder the elements\n", "s4 = pd.Series(data, index=['e', 'pi', 'tau'])\n", "print(s4)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s4[s4.notna()].sum()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Drop NaN values\n", "#?s4.dropna\n", "s4 = s4.dropna()\n", "print(s4)" ] }, { "cell_type": "markdown", "metadata": { "id": "DeofA8nZ0ibi" }, "source": [ "NaN (Not a Number) - is used to specify a missing value in Pandas." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "skyr2siA0ibj" }, "outputs": [], "source": [ "# Series can be treated as a 1-D array and you can apply functions to them:\n", "print(\"Median:\", s4.median())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "42_G7Ks-0ibl" }, "outputs": [], "source": [ "# Methods can be used to filter series:\n", "s4[s4 > s4.median()]" ] }, { "cell_type": "markdown", "metadata": { "id": "3f4HD5E90ibn" }, "source": [ "### Attributes and Methods:\n", "An attribute is a variable stored in the object, e.g., index or size with Series.\n", "A method is a function stored in the object, e.g., head() or median() with Series.\n", "\n", "| Attribute/Method | Description |\n", "|-----|-----|\n", "| dtype | data type of values in series |\n", "| empty | True if series is empty |\n", "| size | number of elements |\n", "| values | Returns values as ndarray |\n", "| head() | First n elements |\n", "| tail() | Last n elements |\n", "\n", "Execute *dir(s1)* to see all attributes and methods. \n", "\n", "I recommend using online documentation as well. This will be in a much easier format to read and come with examples.\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "# For more information on a particular method or attribute use the help() function\n", "help(s4.head)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "help(s4.index)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# You can also add a question mark to get help information\n", "s4.head?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s4.index?" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Or see a list of everything built-in to the Series use the dir() function\n", "dir(s4)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Simple plotting\n", "Pandas Series can also [plot](https://pandas.pydata.org/docs/reference/api/pandas.Series.plot.html) themselves. This uses the matplotlib library, called from within the Series object:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Going back to s2\n", "s2.plot()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "One final way to get help is to press shift-tab when you are in the parentheses of a method or after an attribute. Try this in the exercise below." ] }, { "cell_type": "markdown", "metadata": { "id": "FXjjnBOj0ibn" }, "source": [ "### Exercise - Create your own Series\n", "\n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "RUmWlc790ibn" }, "outputs": [], "source": [ "# Create a series with 10 elements containing both positive and negative integers\n", "# Examine the series with the head() method\n", "# Use a Python list. Or a tuple, or any other iterable in Python.\n", "mys = pd.Series([-25,-10,-5,3,6,8,10,15,20,25])\n", "mys.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Series from Numpy or a Series. See the [documentation](https://pandas.pydata.org/docs/reference/api/pandas.Series.html#pandas.Series)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# create a Series from a numpy array\n", "a=np.array(range(6)) # numbers 0 thru 5\n", "print(f'numpy: {a}')\n", "ser = pd.Series(a)\n", "print(f'series:\\n{ser}')\n", "# Change an element of the numpy array\n", "a[0] = -100\n", "# Print the series again\n", "print(f'numpy: {a}')\n", "print(f'series:\\n{ser}')" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Tell pandas to make a copy of the numpy array.\n", "a=np.array([1,2,3,4,5])\n", "ser2 = pd.Series(a, copy=False)\n", "print(f'series2:\\n{ser2}')\n", "# change the array again\n", "a[-1]=1000\n", "# ser2 was built with a copy of \"a\", so no changes.\n", "print(f'series2:\\n{ser2}')" ] }, { "cell_type": "markdown", "metadata": { "id": "PQprwXx40ibo" }, "source": [ "### Pandas DataFrames" ] }, { "cell_type": "markdown", "metadata": { "id": "ZYRivqyM0ibo" }, "source": [ "A Pandas *DataFrame* is a 2-dimensional, size-mutable, heterogeneous tabular data structure with labeled rows and columns. The columns can have different data types. You can think of it as a dictionary-like container to store Python Series objects. It also analogous to a single table in a spreadsheet (like Excel). " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "vWXR_GKo0ibo" }, "outputs": [], "source": [ "# Create a Python dictionary:\n", "my_data = {'Name': ['Alice', 'Bob', 'Chris'], \n", " 'Age': [21, 25, 23]}\n", "print(f'The dictionary:\\n {my_data}')\n", "print('####################')\n", "# Make a dataframe from it\n", "d = pd.DataFrame(my_data)\n", "print(f'The dataframe: \\n{d}')\n", "\n", "# You could also do this in one step:\n", "# d = pd.DataFrame({'Name': pd.Series(['Alice', 'Bob', 'Chris']), 'Age': pd.Series([21, 25, 23])})" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xt6NqHhS0ibo" }, "outputs": [], "source": [ "# Create from 2 lists. The lists are in a row format, but we want to pass them\n", "# to the dataframe as columns. np.array() transforms the list to a 2D numpy array,\n", "# then we transpose it with .T\n", "arr = np.array([['Alice','Bob','Chris'], [21, 25, 23]])\n", "d2 = pd.DataFrame(arr.T, columns=['Name','Age'])\n", "d2\n", "# or in one line:\n", "#d2 = pd.DataFrame(np.array([['Alice','Bob','Chris'], [21, 25, 23]]).T, columns=['Name','Age'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "cZXlWYmn0ibq" }, "outputs": [], "source": [ "# Use the head() method to print the first 5 records in the dataframe (same as with series)\n", "d2.head()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Check out the stored data types. Using the numpy array approach with mixed strings\n", "# and numbers created a numpy array of all strings (numpy arrays only hold 1 type of data)\n", "# so the Age column is now a string. Type \"object\" usually means a Python string\n", "d2.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Convert the Age column to integers\n", "d2['Age'] = d2['Age'].astype(np.int32)\n", "d2.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "d2" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NHte5SMZ0ibq" }, "outputs": [], "source": [ "# Add a new column to d2:\n", "d2['Height'] = pd.Series([5.2, 6.0, 5.6])\n", "d2.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# What type was chosen for Height?\n", "d2.info()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Column datatypes\n", "Pandas has [documentation](https://pandas.pydata.org/docs/user_guide/basics.html#basics-dtypes) on the data type. Numpy [datatypes](https://numpy.org/doc/stable/user/basics.types.html#relationship-between-numpy-data-types-and-c-data-types) are the most common types used by Pandas as each column is essentially a 1D Numpy array.\n", "\n", "You can change datatypes (as seen above), just refer to the docs for advice. For example, floating point numbers with just a few decimal places (say, monetary values or student GPA values) can be stored as np.float32, a 32-bit float instead of a 64-bit float (np.float64). This halves the storage requirement for that column." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Add your own index:\n", "d3 = d2.copy()\n", "d3['my_index'] = ['person1','person2','person3']\n", "# assign the values in my_index as the new index\n", "# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.set_index.html\n", "d3 = d3.set_index('my_index', drop=True) # remove my_index afterwards\n", "d3" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# If you don't like an index, you can remove it and reset it to the usual one 0...len(df)-1\n", "# https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.reset_index.html#pandas.DataFrame.reset_index\n", "d4 = d3.reset_index(drop=True) # What do you get if drop=False?\n", "d4" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Combine dataframes. There's a bunch of ways. Here let's stack d2 onto d3:\n", "#https://pandas.pydata.org/docs/reference/api/pandas.concat.html\n", "d5 = pd.concat([d2,d4], axis=0)\n", "d5\n", "# See here for a discussion of a bunch of ways: https://pandas.pydata.org/pandas-docs/stable/user_guide/merging.html" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The index has gotten weird...fix it.\n", "d5 = d5.reset_index(drop=True)\n", "d5" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Reading data using Pandas\n", "You can read CSV (comma separated values) files using Pandas. The command shown below reads a CSV file into the Pandas dataframe df. This was derived from a [dataset](https://www.kaggle.com/datasets/navjotkaushal/coffee-sales-dataset?resource=download) on Kaggle.com. It's not clear what the denomination was used for the \"money\" column but that's ok." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Hfwv2wzd0ibq" }, "outputs": [], "source": [ "# Read a csv file into Pandas Dataframe\n", "# read_csv also accepts a filename.\n", "coffee_url = \"http://rcs.bu.edu/examples/python/DataAnalysis/coffee_sales.csv\"\n", "df = pd.read_csv(coffee_url)\n", "df" ] }, { "cell_type": "markdown", "metadata": { "id": "2JaiSOAXnKup" }, "source": [ "The above command has many optional arguments that you can find in the [Pandas documentation online](https://pandas.pydata.org/docs/reference/api/pandas.read_csv.html). Let's take a look.\n", "\n", "You can also read many other formats, for instance:\n", "* Excel - pd.read_excel('myfile.xlsx', sheet_name='Sheet1', index_col=None, na_values=['NA'])\n", "* Stata - pd.read_stata('myfile.dta')\n", "* SAS - pd.read_sas('myfile.sas7bdat')\n", "* HDF - pd.read_hdf('myfile.h5', 'df')" ] }, { "cell_type": "markdown", "metadata": { "id": "2JaiSOAXnKup" }, "source": [ "Before we can perform any analysis on the data we need to\n", "\n", "\n", "* Check if the data is correctly imported \n", "* Check the types of each column\n", "* Determine how many missing values each column has\n", "\n", "We can then carefully prepare the data:\n", "\n", "* Remove columns that are not needed in our analysis\n", "* Rename the columns (if necessary)\n", "* Possibly rearrange the columns to make it easier to work with them\n", "* Create new or modify existing columns (e.g., convert into different units) if necessary" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "86icwTdZ0ibr" }, "outputs": [], "source": [ "# Display the first 10 records\n", "df.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "z-lqfY0MiwGP" }, "outputs": [], "source": [ "# Display structure of the data frame\n", "df.size" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### More details on DataFrame data types\n", "\n", "|Pandas Type | Native Python Type | Description |\n", "|------------|--------------------|-------------|\n", "| object | string | The most general dtype. Will be assigned to your column if column has mixed types (numbers and strings).|\n", "| int64 | int | Numeric characters. 64 refers to the memory allocated to hold this character. |\n", "| float64 | float | Numeric characters with decimals. If a column contains numbers and NaNs (see below), pandas will default to float64, in case your missing value has a decimal. |\n", "| datetime64, timedelta\\[ns\\]| N/A (but see the datetime module in Python’s standard library) | Values meant to hold time data. Look into these for time series experiments. |\n", "\n", "\n", "### DataFrame attributes\n", "|df.attribute | Description |\n", "|-------------|-------------|\n", "| dtypes | list the types of the columns |\n", "| columns | list the column names |\n", "| axes | list the row labels and column names |\n", "| ndim | number of dimensions |\n", "| size | number of elements |\n", "| shape | return a tuple representung the dimensionality |\n", "| values | numpy representation of the data |\n", "\n", "### Dataframe methods\n", "|df.method() | Description |\n", "|-------------|-------------|\n", "| head(\\[n\\]), tail(\\[n\\]) | first/last n rows |\n", "| describe() | generate descriptive statistics (for numeric columns only) |\n", "| max(), min() | return max/min values for all numeric columns |\n", "| mean(), median() | return mean/median values for all numeric columns |\n", "| std() | standard deviation |\n", "| sample(\\[n\\]) | returns a random sample of n elements from the data frame |\n", "| dropna() | drop all the records with missing values |" ] }, { "cell_type": "markdown", "metadata": { "id": "v_Gm0ubUmu95" }, "source": [ "Sometimes the column names in the input file are too long or contain special characters. In such cases we rename them to make it easier to work with these columns." ] }, { "cell_type": "markdown", "metadata": { "id": "tj85w0PMttNG" }, "source": [ "### DataFrame Exploration" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pbDGiE0J0ibt" }, "outputs": [], "source": [ "# Identify the type of df_new object\n", "type(df)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pybC4toq0ibt" }, "outputs": [], "source": [ "# Check the data type of the column \"money\"\n", "# We access columns using the brackets, e.g., df['column_name']\n", "df['money'].dtype" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# If the column name has no spaces, complex symbols, and is not the name of an attribute/method\n", "# you can use the syntax df.column_name\n", "df.money.dtype" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1T1wR6qq0ibu" }, "outputs": [], "source": [ "# List the types of all columns\n", "df.dtypes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "YFXUSewc0ibu" }, "outputs": [], "source": [ "# List the column names\n", "df.columns" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_Cu0DFOj0ibv" }, "outputs": [], "source": [ "# List the row labels and the column names\n", "df.axes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "99jPywmg0ibw" }, "outputs": [], "source": [ "# Number of rows and columns\n", "df.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "utYUKYjN0ibv" }, "outputs": [], "source": [ "# Total number of elements in the Data Frame (100 x 5)\n", "df.size" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "jsjjlva70ibw" }, "outputs": [], "source": [ "# Output some descriptive statistics for the numeric columns\n", "# On a large dataframe this can take a long time to calculate\n", "df.sample(10).describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ZFxc7P8w9mJc" }, "outputs": [], "source": [ "# Remember we can use the ? to get help about the function\n", "df.describe?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Adding columns to a dataframe\n", "\n", "Here's two ways to add a column:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5PTMYO9ok8te" }, "outputs": [], "source": [ "# Let's modify the dataframe to create a new Date_Time column\n", "# Sometimes this option is enough:\n", "# df = pd.read_csv(coffee_url, parse_dates=True)\n", "# but here we need to help do this explicitly as the date & time are split into 2 columns\n", "# The current approach (older pandas did this differently) is to make a new column using the pd.to_datetime function\n", "#, parse_dates=[['Date','Time']], date_format= '%m/%d/%Y %H:%M.%S')\n", "# Combine the date & time strings into a Series of strings:\n", "timestamps = df['Date'] + ' ' + df['Time']\n", "print(f'Timestamps.head():\\n{timestamps.head()}\\\\n\\n')\n", "\n", "# Or just do this in one line:\n", "df['Date_Time'] = pd.to_datetime(df['Date'] + ' ' + df['Time'], format='%m/%d/%y %H:%M')\n", "print('df.info():')\n", "df.info()\n", "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "5PTMYO9ok8te" }, "outputs": [], "source": [ "# Temporarily make a copy:\n", "df_new = df.copy()\n", "\n", "# Add a new column derived from the money column\n", "df_new['money_10'] = df_new['money'] * 10\n", "df_new.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XHWfkQX8roey" }, "outputs": [], "source": [ "# Create a new column using the assign method: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.assign.html\n", "# Assign returns a copy of df_new with a new column attached. To keep using df_new we could\n", "# add the optional argument in_place=True or (better) re-assign to df_new\n", "df_new = df_new.assign(money_1000=df_new['money']*1000.0)\n", "df_new.head(10)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "vsyWDUkww3m3" }, "outputs": [], "source": [ "# Check how many unique values are in a column\n", "# There is a rank attribute in DataFrame object so we access using df['rank']\n", "df_new['coffee_name'].unique()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pESwRxLCt5oO" }, "outputs": [], "source": [ "# Get the frequency table for a categorical or binary column.\n", "df_new['coffee_name'].value_counts()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "pBF1lOcY0FLH" }, "outputs": [], "source": [ "# Get the proportions\n", "df_new['coffee_name'].value_counts() / (df['coffee_name'].value_counts().sum())\n", "# value_counts returns a one-column Series which has a built-in .sum() function.\n", "# You could use the built-in Python sum() function:\n", "df_new['coffee_name'].value_counts() / sum((df['coffee_name'].value_counts()))\n", "# Why is this not recommended?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "ITSJWChquiYI" }, "outputs": [], "source": [ "# Alternatively we can use the pandas function crosstab() to calculate a frequency table\n", "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.crosstab.html\n", "# This returns a dataframe\n", "pd.crosstab(index=df_new['coffee_name'], columns=\"count\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Removing Columns\n", "df_new has accumulated some extra columns and has some that aren't necessary anymore. Let's remove them.\n", "There's a couple of ways to do this, check the [drop](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.drop.html) function documentation." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Drop Method 1. This is convenient if there's a lot of columns and you just want to \n", "# drop a few.\n", "df_tmp = df_new.copy() # a copy for demonstration purposes\n", "df_tmp = df_tmp.drop(labels=['Date','Time','money_10','money_1000'], axis=1) # or axis='columns'\n", "df_tmp.head()" ] }, { "cell_type": "raw", "metadata": {}, "source": [ "# Drop Method 2. Subset the dataframe and just keep the ones you want. This is \n", "# convenient when you just want to keep a few columns. It also lets you re-order.\n", "df_tmp = df_new.copy()\n", "df_tmp = df_tmp[['Date_Time','payment_type', 'money','coffee_name']]\n", "df_tmp.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Re-read from the CSV, put it all together\n", "Let's re-read the dataframe, add the Date_Time column, and drop the ones we don't need." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = pd.read_csv(coffee_url)\n", "df['Date_Time'] = pd.to_datetime(df['Date'] + ' ' + df['Time'], format='%m/%d/%y %H:%M')\n", "# drop columns we don't need\n", "df = df[['Date_Time','payment_type', 'money','coffee_name']]\n", "df.head()" ] }, { "attachments": { "b310b122-fed9-4fd8-8bc0-2f8104575b5a.png": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAABsCAYAAAC2EsxOAAAgAElEQVR4Xu2dCZhcVZXH36uu7oQ1gCSEbYgIqCyDmsCA4NAgOo7jPhMccdTuIAGD7AImpJNKOiA7CAMMAZIwMqOSWXVGBh1Ms8iIEEA2AYMiEMjCHszW3VVzfs074XXRna6qrqquqve/33e+qq567777/vecd/9nudVhUKE2efLkpsWLF/c27z3xzlwu+IhdJmcSVuhy6lYIjCwCpuRhc0uf5Lo39kkQSt1HdlJ09RFHoM8uRpk0B7mNG4JcT7fsYsQnRQMoAIGsHZNKhal5G397X0dra2u6q6urp4Dz+h1SsRVgAIK1ttjB6XghUBcIhKkgt/bNYMv9D20ZPW639Lrlv+te99TS7nCLrc2twE7VhEACEYjsYqsDPtwyauyuZhfLNq576sEe2UUCdaHubtmUN8iNrnmCld5r0hIbaGtTKvzkhqfuv3XMAYdv37Suu7fu8NaAhcAgCPSOf0/T63f/86tj2zuvsKjVqane3MyVN3WcN+bwY7dvWvG0dF2ak0gE3C7GtXdekwvDb4TZ3rNXLZp9sewikepQFzfdu0Vz0+uP3P1qy16TzssGuRkWbs30LLt/Ts1GsJxgBangIz1PLb27LlDWIIVACQjsNKXzgmw2e04qTJ+xcuG5l5fQhU4RAg2HgBGsy3K57OmWcPnm6gUdVzfcDeqGGg6B9N4TZ1pRU2fdEKx0Ojxy/RP3d8EElyxZIq++4VQyuTc06YT56aXzT+ge1z73Yis5+VYYps5atXDmJROnXtd8/3VTi87bJxdJ3XkjIeB20RfZzeVOtXTLqSsXzrxSdtFIs9xY9zJp0qT00qVLuy0wNNsyb5m6IVhWL9bas+y+O7w2q7GmRXeTZARaM0vSXZkje8a1dV6UC3JnhanwW6sWdFzqnycZG917chFw/R/bNpdo7mlOsGQXydWJWr9zTwWm9zpoVhBk54hg1fqMaXwNj4AIVsNPsW6wBAREsEoATaeMKAIiWCMKvy4uBN6JgAiWtEIIDG4XimBJO+oFARGsepkpjTMxCIhgJWaqdaNFIKAIVhFg6dCaQEAEqyamQYMQAm8jIIIlbRACimBJB+ofARGs+p9D3UGDISCC1WATqtspCwKKYJUFRnVSRQREsKoIti4lBApBQASrEJR0TNIQEMFK2ozX//2KYNX/HOoOGgwBEawGm1DdTlkQEMEqC4zqpIoIiGBVEWxdSggUgoAIViEo6ZikISCClbQZr//7FcGq/znUHTQYAiJYDTahup2yICCCVRYY1UkVERDBqiLYupQQKAQBEaxCUNIxSUNABCtpM17/9yuCVf9zqDtoMAREsBpsQnU7ZUFABKssMKqTKiIgglVFsHUpIVAIAiJYhaCkY5KGgAhW0ma8/u9XBKv+5jC0IedKHDbn0ko9f6DLDtYnn8fH6sdlSxx7Yk4TwRp0qoejv8M5d7ABDdVn3AbKaXOJsYX4jYpg9T1Ph/P8Hu75xTz/U3ljTeTzXwQrkY+qit/0cEhgxQdX6xcQwar1GSpofCwwciYKgqqwg0SwCsNJR9UOAiJYtTMXmxuJP6w/bgddanKSyZ0mxT7E/97O2cvk0ybdJqUSIT9vG+vjv0zuMpkZ9ece1vb25nSTKSZjovHOtdd7h3Hd+pitYY5SBKsfgK7ju9mnPza50QQ9LlT3/bjj7JzTTL5g8tsizh9oNr3Pq+3LPzH5osnamF432ftek7TJTSYHmHzC5AXpfunGkWCC5fp2gqF3fKRvTxeow/6sHm3HL450kH5KffYzgT6ePe39v5tcZXJD3njQ+dkmR0Qzfre9XmxyzzCvXboCjcCZIlgjAHoJl3SF/hs794cmEKSf5Ck0BsNx8TAwD/l4+5H9sZ/JviYbTJpNSF3gaQ/kbef36ce5cUKcnozGApGKL3o/tb8/ZsIC85TJN0xYJA8xgWQVukCWAFd9nyKC1W/+4g/zx+2bi0xmmTiJ8YP5O6776Cq67cedY+/PN/mAySMmLZHOc0y+nQzUZ9xGfEwQPhyWSSZ/jK7vqcD32t//YNJqsia67u/yjqlvRa3y6BNMsFyHcWKnmxxs8phJ/Bk61LN6Szt+qckfTCD7kH9vm3v+c22arxO8+nXfb++xJcZ1gQnrCY77RJP7TV4xuTk696/sFVshSPCzvLHHhtJYb0Ww6mM+XaE/b8P9RxOI1m0xJd0cWYl7KrfYOZArDACCFW/5C9ZgfXpuHaOEYD0YjQUChdH2mPD+GpOzTfBaaHub4L0sM8Gr2WgyHC+qPmauhFGKYPUDzfVwgn36gMllJvNM/PPN6VBcV8+wc+aYfNiERSHe8mtTNldXyHl8j/4TEXiPyUdMIFg0orr/ZnK0yUMmz5scaYLdPRudq3qsvAko5E8RrAAn4UwTnp+/MXEbGOxZ7TWA6CoEiyjScyafHQDvUp7/77N+cJbJTJBZ8ec/dvEXJjjTOEW0sdH7RyPbwKlp+Oe/CFYhlj3yx8QJFh4BaQ4nWDyskXeZkAbZPVJciA9kLJ4K/Ff7Gy/iQJNPmeBN0FgQumK36dejz29GxkEKhAgYRhqPYP06GgthZzdSold/arK/yUsmhKfXm3zXBPKFB8bioyhWDHR/K4LVDxTXkXfbp+g0D/LOSHdc99HHr5ocZMJi8qLJApOVJv7Q/5a9z5gQwd3KhIjrFia/NPledMV8YnWifY4e44z8nwkOCs31/F/sPfZ0mIkTLJwOomykcC43IWpGapJ0Cotbwy8qEUZlfxHBCr5toKLHEHonWG4D2Ee7yXaRji2xV57r3lzXSY/joLeZ8BxGb1kncDpcN/0V52GqCfbyqskPTIicuf5DsO4zyZhglzTWDOyPteZLJtgvx7MOYS+sPX9mQnSr4W1BBCvSihp/GYxguaKjsCwSO5rgMfA5i82tJn9r4g9/PAsUHHL2l5HybxsZxVn2ekn0GYsUfZKO3NoEo8QDQag76TDBA8GYIUr0B8Gi7WDCQoinQljY05Yc/2UTCOLnTP7TJN9rirpI9osIVr/5H4xgOXEab0f/kwn6Dql5zYSUHQ95nAjS0zQWJlIZC02OMkHvSRPuYsLC8RUTdJQFaycTFoMPmrCQ4SCwgJHagJix2NDyCVY8YhAd0lcvxiLFYiWC5aiU8CqC9Q6C5c9PHG6ey9gKz3/qX3EMrjU52YTnOSTpDhMcXco6sAEaz2ue8dQRos9uV5Cw601YO4i8Ut5BH9gQ/dLyCZaTKZyJ101WmHjqEtvi+qwZOCRvRt81dDRXBCvSlBp/GYxgMWzy3r8y4QHOIvBydC8QKOq0WBBYVGiQsL8zgeQQSULJad83gYhhlHgypDnw2FmcMF5vx9obFjNSlf9hgiF7itAJFsZF/p3vuXY8TdNqf+NZ5ZO52CX0VgSrnw4MRrC83uNKO5pFhAUD3aLtbAIxgsR/LfrsFHslgkodCkSfRYZG6hDvm+gvUS8ahfQfNYG0rY4+YzEhRUnqG7JGw0vH7jyCFffIfaFifETCRLAi0Ep9EcF6B8ECShwMdPn3JtQXenN9/3P74C4THAocblLk1EudawLx4vOfR/qJjhJZmmDCmkJky3WdfjMmFK7jeOBYk/YmRcjn2NBADrN/xrFEvyiIZ2yJyF6IYKE2td82R7AI/VLrRKoOUuONBYjFAXLFbj4aESzqryjAJWQ7yoT0BwbD4uHEh8Xlf01IIUKWIFLrouNZxDA+PJ6BIliQNGqt8HLoL06wvF9qYTImDR8ijnAv6kUEqx9cgxEsf3BD4tFPT1H4yUSb0E8WGHSX2hUWFqJb2IrrPnZCnRSp7782waN/xoRoLkI0Cy+bVPciExYxiBdRLFIwOBT5BCueahTBKkr7Bz9YBGtAgoV+otvoIulub+glEV0caTZbEKWCaKHHbD6iuQ3wLCeKizPNrkB2qZPe5lmOnhO5esMEB/52E0gSBI10O9fMmGB/bqu8+gYTanK59i9MSJ9jK8tNEvHsF8GKNK3GX4ZKETJ8iJMvEBQ0sriw2xBDwUunEQKGXHEsRea+SBH+hUjdaXKMCQZGaoO6KzcWzud40ip4Qp+MrpGfIlQEa5jKJILVD8DBCJbrpaf0iEDtasLDnJQe6Wl0k2Jb0hx44njfpL49je67B/8nsgvqqSBQRG/pH+fDyRKDwpkh7cEx1HcNRrA41hcQEaxh2oOfLoI1aIoQiCBNOMSQGPSUKC4O7TQTnF3sAieCSBcZiLjjSy0Wz342JFH+cYXJqSZkOIjEep0XdsV56DTfD0awGI+vLVyXSDLE6lATMh6JiF4BgggWKNR+yydYECkWBa/5wKNoMyEtAolipx71UqQz8gmW7yLEq3cjIMxMqJcFiR0mLEYU6n7GhK3leDo0QsosVtS54AlhzPEid47ZXA0W6UnSlFyDgnnVYEXAxl9EsPqB4ro/IdJPdhHONfEUHKlpdJVoLSk7UtzUmXAcnjKLDI4COk3klIf8w3m6h9fOIsM1iE6x2FBrAoFigUDvsTV0H08e3cczH6gGy2tKRLAMoHI2Eay+XYRkKw434VlP4xlMKg89J7MAmfmDCdEn9D0/gkW0Fsc7Xi+IwwD58p3f/KYVz2hqGLElJ2M4JJAu0oisAV6DxXXi9btumzg8/JQJkS9S+IkiV0yOCFakpTX+4osMIVyMCYJFYTkNxSVs656K3wqRJn5/B++FXUy0+C5C/vbdffyEAvVWGAlpPa7Dsf6bVXF4MGQa13SCRUQr/jMNg+0idG+eRQxilhhPJg7gUO9FsPohlB/BQkf5mQbaOBPqOkhns2Mp3khTYwPsuMKZQK/5/Sw2ebjTQBQXQvSECaSJhYtoLsXo/N4QKcV4Y9GiTgX9hmxhIywebJvH1uJpDxGsPPCG+6cIVl8Ey3+mwX/+gA0U15n470s5zGQq0Os4wYJE4XxTwE5q3KO0/C4Wz3CclfkmpP+wMdKPq/LmDdKFrZAa5HewIHUZE6/B8t/L2sM+w2mHbLWasOnJf3vON5MMVyVq/nwRrJqfor4B+iLD7juKzPFAKEykeVQID5wts95YINgiTtgXr4dGcTsFvoSTWZS8sfDghfjnE+w9Xj5GR37eGzuuyOvTD78oTBoSr4R6FwzdG0W9EDs8LqILNP8dLHYktprod7BigMXfimD1AyYewULXiEx1RkeQouAhTuovXoNFepDoE9+x2QMy5EW/+cTJFxdSIywqEKMuE37uhNpEdkN5o44EncdBICrmKXd+tZoWdxicYFFYz8KF/msXYQzMYt+KYPU9xyFZHzJ5JsKPaC6fo6++IYOvqLtC75w0sXGJ5zR6SKG7b/Lg2H82mWyCPeFok86DjJ1nQiTXGw436XP/fUOyIRAt1g7sz3UeJwbbIJ1+tEn+by7Sn2qwurqIghfV4vUKRZ041MGTJ09uWrx4cW96r0lWZJ1rtWdZa8+y++7wz4c6v86/9wf3PnYfFKNjHOx0IqSLB83CQ9iWhzkeBFEtokuGU9/23W9G908qhO/Ywg5BI4WCYWKE7CRkl6A3dmYRcSLEC9HCE6HOhevh6fvvphCSps4FAyO6hteEHpDCxKsiJYjRQroIGVMDg9ej6FUM7PhbEax+wPiDmLoSHvr8axpS4uz4I91xR6SPLAYQGFKAe5qQ9vAftYVgQawgYug8OwmJoFK8S2SLiBaLCp45zRcYdBw7I01CzSEpcxwUFiQaC82F0TE4PmwAiT8DsUVf6PDon42+b+it6RE2ZX9JMMHyZ6VvEiJ7gZOLsIkDG8CZYEMTz32OIzvBd9RK8Rxnt2CXCc9giBh9oO84EZAriBr24ddiLcEpoU9IFRtJ6IsUeasJNYj06d+xzuBU8ztYlJpQB8zfpCshdx61IorFWhP/3S37szGbIlj1M6++0JAeJCpFLhyyQvqD6BVePcZCSJfFh1AvJAqC5B4/D3tYMzl2olt4MuwSwRBYtOgr7llAkIhuTTAhpMwulBkmECY3RFImjIfFh/AxxIqGp8/uxXYTjJ5zMWLIVSK8lwiHol9EsN4BmesL+k2KEBIFCcKxgHBlTIhEob+k70iJs0mDOkR+fgSCRKqECDApFsg+kVmIDp42tgBZi5N+vHOiWq0m1GDxPakTIr8+HvQa+yA9T6TWFygnULzyPWNg0cOxke4bCKW0BBOsOFzoGs9ziA3PWxoRW3QVxwJCw/OYyFJX9DnkH8LD7xryHCZ9zjH7m1D0Tp3uTSZea+g7ANnwRMSMulocbHapE60iMuW2whqETeLQ0B8OO04J9bUQMAiVbyahf2oacVKIMDe8ky2CZbNc562cD+xC+xrquKG+r3PIKzt8EayC8S2nnsX7KrXfUs8r+IaTfKAI1jtmv9z65v0N1e9Q3ydZTfvduwhW/akCrJ/mW2d5H/+Mv/MNIL6zyb/3PuLH56cuBuqX4/FEvHEtv178/Pjn8WvGz60/9KswYhGsQUEeSPcH0rO4Hubrfr59eEovXy+L1f24PcVvwPtXWnCYtiOC1QdguZ//cf0c6Pk91DoxmE1ubrYTYwsiWMM0ep0uBMqNgAhWuRFVf42AgAhWI8xisu5BBCtZ8627rQMERLDqYJI0xKojIIJVdch1wWEiIII1TAB1uhAoNwIiWOVGVP01AgIiWI0wi8m6BxGsZM237rYOEBDBqoNJ0hCrjoAIVtUh1wWHiYAI1jAB1OlCoNwIiGCVG1H11wgIiGA1wiwm6x5EsJI137rbOkBABKsOJklDrDoCIlhVh1wXHCYCIljDBFCnC4FyIyCCVW5E1V8jICCC1QizmKx7EMFK1nzrbusAARGsOpgkDbHqCIhgVR1yXXCYCIhgDRNAnS4Eyo2ACFa5EVV/jYCACFYjzGKy7kEEK1nzrbutAwREsOpgkjTEqiMgglV1yHXBYSIggjVMAHW6ECg3AiJY5UZU/TUCAiJYjTCLyboHEaxkzbfutg4QEMGqg0nSEKuOgAhW1SHXBYeJgAjWMAHU6UKg3AiIYJUbUfXXCAiIYDXCLCbrHkSwkjXfuts6QEAEqw4mSUOsOgIiWFWHXBccJgJ1S7DSTcFR659cumS//Sa37Ltv0DtMHHS6EKgZBB4cPz697KqrNoxrn3NJLheeGQbh2asWdVy818knj/rgihU9NTNQDUQIVBEBt4ux7XO/G+SCU1JBcPrKRbOukF1UcRJ0qaIQePDBN9PLlt26oWWfSZlsNjc7CMJMz7L75zjxKqozOzgs9oRCj588eXLT4sWLe9N7Tfy5nXNk2BxM7P7N0gcKPV/HCYF6Q2D8lPPm9vb2dDS1tJy04vrp19Tb+DVeIVAJBHY67rwLsj0956TSLcevvHH6DZW4hvoUAuVEIL33pLODXO7COiBYk5YEQa41CFMdTWHugVw2t1UQhtlygqG+hMCIItA8qmnj4794bcdjZ0xLbbntZ3vffHXRyz+48Act+x42JujeoGjtiE6OLj5iCLhdfPnck1NbbPOp7JqXbnzph5csll2M2IzowkMhkAqbwjB8Pdub/Voul/1SkEplep6q4QhW894T7zVydXCu19aZsGIBs6Fg0/dCoHIIoNc93UF67G5Byw7jg40vLQ+6X34xCNPN5lvkKndd9SwEahkBs4uc2UXLuN2DZrOLDaueC3peWSG7qOU509jeQsCe22FTOgh7N1608emHzqnBFOEtliI8pnerI754U/OYd01e88hdP06lR2dt4GJZUuLGQiCVCrPr16wfvdt7D0hvs92ENx+++yepbXdoDnqzYleNNdO6m6IQsEd9tqc3tdWYcVtMeN9Bbz75wF1hLrvG6rGsHEtNCNQoArnerGUixmyx5/6fWP/cshPXP3Db9V7yVOyIK0Z2Jt9iBOuYY3rHff38a4Js7rjtdt57z6fOO2Z5sQPU8UKgXhDY6fgLbgxy2c9vv+fBez8x4+iX62XcGqcQqCQCO5146ZTsxvU3tmy99dHLrzz19kpeS30LgXIgsPO0y7/Qs/6P/xo2p7+y6rpv35zJ5FKZTPGlTRUhWD6YXdrP+1B30NtlUatt7ELfW7Vo9lf3m5xpGbu6SzVY5dAC9THiCKwe25p6bHFm4/i2ztbeIGeLRy5luwgvW7Vo1pnS9RGfHg1ghBBYs88+4dL587t3n5LZZUM2vMsSLnvaGnBv2PvGR1d+75K1E6dOTW/z1FOK8I7Q/Oiy70TAdXa34zI7bOwJ78iF4f5Wj/XItluPOXzZVae8kbPsm7WidLbsBCs+iLHt8/Dop3ArtuisyzWnPrz6+nMfKpUNSimEQK0h4Po+rn3uTy1t/7E+XQ/Dlc1NLZOW33DO89L1WpsxjacaCLjej2ubd0YuyF5qRtFtjnZzUyr44ooFs27xDEc1xqJrCIFCEHCd3Kl97tetuON6I0cbjU21mMd8/MqFs24oRWfLTrA2GdaUuQca4bsPo7Ilxyrcc1Tn37RqYUdbKUywEIB0jBCoJgKbDLJt3qezQfZHb137LV03j2Le6oWzOkoxymreg64lBMqNgD/fJ5x2+XZrX3vzgVyQe7fZQ4/VXqXN0f7ltAm9h2UyGWUxyg28+isZgU06m1k4eu0fnr/P/t4/prOPbTlht0nPZNrXF8tdyk6wNtVetXfeZIP56lsLDkWNFl4LgvXpsOmwFxae+4A8+5J1QSfWAAJuaOjxNc/Mu9UWkY/36Xpomzj6injD5aNTLQc/t+CcF4o1yhq4PQ1BCJSMwNtrwLyzbav7hbZQ2eYm30KeC1Nh6tiVC2d+X85HyRDrxDIjsMlZnjJ3ajYbXGfP71gqsC81eKIFh64rVmfLSrCcNI1v+85B2bC7y1ImW741UFt0+owsSNlAb7aBfkWLTpk1RN1VFQHX9Z3aOz+VzeV+3O/ika57FEvORFWnRhcbQQRc18dOu2h8uHbDPX3RK18D3l4L7hu13dZHPH/5Geu0DozgZOnSfQjkRVzvNp3dL19nzT14fPTo5sOfvXb6q8XobNkIVl7t1fVWe/V1IlZW3DgqIlgWIrZhhsHa5iDVqiiWtLsREBjX3vkz0/2jvcbkrXt6y6kwbX/Jfrbu4JU3dPxeJKsRZlv3MBQCMcfjdHM8LutbA8KgJYrq9ppN9Ji9jPJaLNnFUIjq+0ojsElnp3QeZ/8e54Z+OmvOsj3NqcUa7bVYxehs2QiWg8DFF6y8YMyGXLr5/eMnvfL4M3d12eAOSwWpz+w4YZ/bVj//2+3TqR3XvDD/hLWVBk79C4FKIoBTMf4bl4zdpnnU+jVrXtvD9PxHRrR2tkXkJHafmHOxTXp0+NDyq2e8XIzXU8kxq28hUA0E9px6wZg3s72jV14/Y9VOU+Zda/p/gtnGjJP2mHnhNc+fP3b7bHrtkwvOWVONsegaQqAQBPY6+cpt31i3Zotpu81Yfe2z8843snWOPcsvmrZHx3R0dtsttlnHbsJC+vJjyk6w8i9uHv7tZlxH2a+iHr7qxhm/KGZwOlYI1AsC+03LbL16bRP6vXfYlDp45Y3nPlovY9c4hUAlERg3Ze6luWxwRhikTlq1aKb+R2clwVbfZUHASj/OtQjsPKsX7LB6wXmldloRgoW3PumE+eml80/oHtvW2fe/CJvS4ZErbujoas0sSXdljuwpdcA6TwjUEgJe9Lj7lAt32ZDbeI/p+jbp5vRhL8yf8USxBZG1dF8aixAYDgL91oD2zius0OVUW6xOtcXqyolTr2u+/7qpPcX+ptBwxqNzhcBQCMR1dlxb52yrxcrYjow5qxZ0ZErV2YoQLG7EF5dNBCtMt65YOOMOLTpDTbO+rycEPB+/24nzdt2wIfsrKzJsDsP0ISsXTP+d0oL1NJMaa7kRcGd6bNvcy63v05xgyckuN9Lqr1wIuG6Oa583y3bAzrHneWbVoo45peqsCFa5Zkb9JBKBfgRrfe5ey9m3BM3BoavmdzwtgpVIldBNRwiIYEkV6g0BEax6mzGNt6EREMFq6OnVzQ0DARGsYYCnU0cEARGsEYFdFxUCAyMwEMFSilDaIgSCQARLWlBvCIhg1duMabwNjcA7CVZuVNAcHqIUYUNPu26uAAREsAoASYfUFAIiWDU1HRpM0hHII1j32f/JaQrT4SH8uKhqsJKuHcm+fxGsZM9/Pd69CFY9zprG3LAIvP0rwN/ZM5fr/rX9YvUrqTA9yXbMrhbBathp140VgIAIVgEg6ZCaQkAEq6amQ4NJOgKb/vfacd/ZK+ztfsT+Tc6LLU1bTXr+xjNeEcFKunYk+/5FsJI9//V49yJY9ThrGnPDItCfYPU8bD80urKlaeuJIlgNO+W6sQIREMEqECgdVjMIiGDVzFRoIEIgCN4ZwQpWiGBJM4SAdhFKB+oPARGs+pszjbiBERDBauDJ1a0NCwFFsIYFn04eAQREsEYAdF1SCAyGwEApwuat0pOWXz3jZdVgSW+SjIAIVpJnvz7vXQSrPudNo25QBDb9TEPbRXtvDNY/GoTB86NHN0969trpr4pgNeik67YKQkAEqyCYdFANISCCVUOToaEIgU0/03Dceftne3sfsX8O+ti+E3o/0JXJ9IhgST+SjIAIVpJnvz7vXQSrPudNo25QBDYRrLa5B2SD4OEwDB+dtkfvgZlMxv5UEwLJRUAEK7lzX693LoJVrzOncTckAiJYDTmtuqkyICCCVQYQ1UVVERDBqircupgQ2DwCIljSECEwMAIiWNKMekOg7gjWuPbO260W5ah0U+qIF2+ceefkW25pWnzMMb31BrzGKwQGQuAdBCsIH542ofeDShFKX5KOwNuLVedltgacHqTCU1Yv6LjKP086Prr/2kNgk85O6ezIZXNzwzA1e9XCmXNL1dmwUrfoRGpc+9x7crng0KZU01ErFpy7RASrUoir35FAYJOeT5l7YC4bPGRF7k8bwdpHBGskZkPXrCUE3l6s5l5ttjEtDFJnrlo087JSF6taujeNpTER2BR1bZ/baf9XdmYQhm7sxWwAAAyISURBVOetXtgxs1SdrQLB6rzJvJfPNQXNR69YNP0+EazGVMyk3pVHsHY9/vx9urt7bsuFwSMn7ZH9nAhWUjVC9+0I+LN+bHvn9DAXdARhbuqqhbNu1hogHalVBDY5zG3zptm/PbsklwrOsqjr1aXqbMUIFgDmb1PXtvVaVSuNSwgIASFQfgS0BpQfU/VYWQTKqbMVJVhOsubMCcJMJtS29crqhXofYQSIZjEE6foIT4QuX1MIsGCxBsyeHeTsZ0xyNTU4DUYIDIBAuXS24gRLsycEhIAQEAJCQAgIgaQhIIKVtBnX/QoBISAEhIAQEAIVR0AEq+IQ6wJCQAgIASEgBIRA0hAQwUrajOt+hYAQEAJCQAgIgYojUCzB6ivitba5gvUm+55CxmKK2hkH53nfxZxbcZB0gcQhgD6i6+jhYEW5fM9xpfxo7nDOTdxk6IZrBoFK2gXPf1+PsCkVw9fMtNf9QFxvN/esdv5RzPPc++XVOU8/vS2GYHknoB1/Pxj6hRzDub6QxfvhZou50brXAN1AzSIwkB4Xqtv5N1WIodcsEBpYohHI1/ly2YWvQfmEaqB1IdEToJsvCYFin9WFHj+YfvY7v1CC5Sd9NCJX/zsIyRpln08xecjk/wogYj7IsXbsl0zGmPzU5N4Czi0JbZ0kBDaDgOv5dnbMZ01uNVmVp4t+zET7/HCT603WFqmvB9jxB5ssNFG0VipZ6wi4zm8f2cV/2+vqQeziIPv8UJP5JuuHsIv4YvR5O3Zfkz+a/IvJ80XaVK1jqPFVHwHXr/fbpT9gcovJYIGbv7Hv4CMcMxTJ8u/hK39rsqPJCpObTTbEzy+WYC22k4kufcGk2cRTKHgfCAvTnSYLTK4wSUef8138Wtykk6s/tff/btJt8prJgSYnm9wQO8beqgmBiiPgOvk+u9KvTD5hck+erjMI9P4Ek2+Z/JnJK5GuO1mKe+X5BOozduy/mbxpMt5kqEWo4jetCwiBIRBwu9jfjsNxPsrkvkHs4qTo+Y1dvL4Zu/A1gzXiOpPJUZ972+s6EwjX4yaKZEk9S0XAdecb1sF0k32i522cl/jz+Yf2He8J9Hi6MJ+3+LF8Dmn7J5OtTX5rcpgJ3OfvTN4w6SNhxRKsGyOFby/gjocyDP+eiBWD/HDU59n2eo7JJJPfy8AKQFqHlAsB90z2sg5/ZvI5k18P0PlQHk78FD92K/sQzxwvHe+faADv+3k85boR9SMEyohA3PH4H+v3r0wey+t/sFTfYMPwMpBj7AAWtw+ZPBgd/At7/YPJsXr+l3EWk9eV622b3fppJkSx8lu+3g7FW/x5/iPraAcTshi0bUx+aXKtyd+b9Ol3sQSLlAbs7fsmMEIiVoSC/yG6CJ1ebYIR/ofJ1032NPlJdPzO9soic4EJbJCF7HYTvJ7/ivqgz+dMiA7g2ageKwJGLxVHwI0HTwed/abJESYfj+nksmgUePE4GseZ4BGh93g0OAZ/bUJU69smvmhggB+L9J0I8HdM/sRko0kxhK3iIOgCQiAPAV90cAh+bEJE4GgTSkYgQjyrfxedg61AjLCLLU1YbG4yIW2Iw/KyCU70w9Hxc+31vSZfNPHrzLP3f2FCulFNCJSKgOsTukjG4RSTM03eY9Jl0mFCSppGUIesHLpH9NX+F2Gf7vJKSQf6epYJ+svzms9wlF+MDQ7HgIDRHJOSCNY1diK5ShYfQsW7mkC0MJjLTVpM8PghXfzNYDE+arYgUnjxHHuuyVUmLF6kHTFWNzgWqydMIGIsUFp8YjOotxVFIB7BusOutNyEehMI/9dMcAg+aEJdFn+fH32GzuLRk+5D16khZJF5twmR2ZfyRn2G/Y1dTDARwarolKrzMiAQj2Atsf4gVawB2AdOxu4mRKBYfHCqZ5mwiJGZeNRkTWQXpBW/YsK6gV28auJ9xyMHRI+xGdI1Q0UUynB76qJBEXDdQefgLl2RHhIkmmlCSg8+Q/uBCfwF5/cvTQj4UB7ycxPqq+A5nP/VPJ2kzIN1gagudbknmuBslJQipLYKrxtC5I1F5pMmh5iQ7mBxIdJFqAwmR7Ewi5LvEiGyxcX5/NMmFIZRd/WMD8peYYLcPDclghUDW28rioDrGosD4V4Wi/+MrkhBI4vFeSZEbCFQGCkLC94KGzu+a4LXQyNyS24eb+h+E4yd46g1PNUEj2mCiQhWRadUnZcBAV+oiDTxbGaRgWDRSJM8YgKpooQExwPnAbsggkUE90ITshE0osO/MWExwmY8i8I1qM1tM6F+l9QL9iaCFQGnl6IRcN1BJ8kYkCLEOaZBouApcBn0DG5DBAsyxmcUu3/ExFPhZDOIYPFcR09Hm+BQw2MIFuFkwHcyJps4SykpQlJ7hNz8ApArBgeJIly21ARDgzGyGPE5x7DDEALG53gwDOzI6EZ45SZpgILBEopWBCsCRS9VQcANg0JbQr0U2rIIYHgQo9tMSBGS0v6yCSFmjJbvKYonMsXOQxqOCM4GHhKLEnpN/xinCFZVplMXKRMCvlBR2It+f8qE57XbBdkJntnUubSZkKXAaSaChaPC4kRUisYCRWSAzR7YDNHfnui74+0VMsYiR+RY5KpME5jQblx/ptj9s3GO8g3fmLeLvScLx+fUU5HGRhd5rrO5CZ7C8ZR60HCoed7DZyBW3jiHZzq2gdOBIwGZ67t2OQgWxsZWdS5MKC1OsEgREtmCEXotFQOnFovFix2EDIrK+65oUKQR2aILG7zMBxq7Ib0VApVCYCiCxSJB+hqjzCdYpD+IuLIjloZHwwJCPRYLighWpWZN/VYagaEIFmlDIlU4GG0mTrB4lmMD1L3gMNNImzvBwma8b2yHCAF2BYkjXUN0V00IlIpAnGBB8qnp8x8E3c3e4/h6/XecYFH/R5kT2Qc4DY10NVFadtISXCKLQaQVvuO6Sj0hwSIiX7SwWILFTyeQPqETb1wIpkcRI54IHssiE1KEnSbcFIzQCRYpFPKWnvvEAIkKwBBpEK9/NGGQRA/kxcTA1tuKIuAECy8bPSafzk8q0IhI+SKCMaKv1BKS6sCT5/gZJp5ShGDhIbH9nNc4waLYksUEW1KKsKJTqs7LgIA/g0nvQY6ou4oTJhYZnA42eZA+RLdJEUKw0H0IlqcUJ9j7u014zkOwaHj8LGp8Rn2XmhAoBwLxGqyLrEMPAtE3QZ1LTUhFU8pBFo5oFPqLLsJf4DQro4Hwe1eesSCb8YAJug1B80Y9OVFb0o+0ogkWCwtMjjoqBjXBhPQfITg8d5gcoWLCZESfuAF+H4IolhOsRfaeCBY3QSMCxjZdUjIUE1P3QpSLPL7qryKQ9FIVBOILCXpMrQgLCsW7ECVS4NQN8htW5PXZDYtHjt4/aQJxwshoe0Tn41xQTxgnWNRfzTbhpxr0Mw1VmVpdZBgIxCNYbEbCNlhcSJ+w8DxjQskHP7hLmg/dxi7YOYsNsYOLNYOGU0F6kawGEQTqFrEbiozpm13kNH5DC+eda2gdiEDRS1EIuN5S0kRwCI6BjuEQt5mw8xsnmQYHoYyJna5wEmrDSfv5LsH4piZ+p40d4TgU2AEb+wgkEWiiSP4uk5JShFPtREJshM3wYlgcIFNdJhgBnWJcLCjspmJRwqBYiPxm2+w9BcMYlpMuftWa8DILFREDbo4mw4qA0EtVEHB9wwHA62YnKwZHWBiP5WITdkTR8G4wJqJW6DHRWhYJvHnau0xIc6PnOCP0jVADgHMBUaNmxetPotP0IgRqDgG3C2pnTzfBiSD1zQKEvl9igtNBIyKAbmMXpFKwAZ7nZCNo/NeOjAkO+NMm1OmyRhAVoCgee2ABZBchaUPfFp//r3Si7vQiBAZFwPW21Y5AIFj8xAiRVSKwFLl7g9sQwSK4Q+kSQSO4DESfxvOeCCt67c9sfqoB3oJTgJ5S7A7Z2sRbCk0RcoHNkZ2Bvoun9vz7+HH+fqAUoI9LRhXTAL2tCgJDkfqhdHOg8wfS+6rcjC4iBMqEQKXtokzDVDdCoB8CQ+ltPrcZjI/kc5HB+u33eTEEi4F4moOL8d7/3UH834HgzXshWTwt4nc92Gd8TqOv/H8vIp0RAtVEwKOx6LFHnlyn3dD8GP/fVui971DxsQ70WdyO9A/NqzmrutZwEaiEXWAj+YuV/y37GO6M6XwnUM5X4pmEOM+I84/8Z3u8j7hOOpdxffU1YhPqxRIsTZcQEAJCQAgIASEgBITAEAiIYElFhIAQEAJCQAgIASFQZgREsMoMqLoTAkJACAgBISAEhMD/A3Q3k+Y2CeeLAAAAAElFTkSuQmCC" } }, "cell_type": "markdown", "metadata": {}, "source": [ "### Add a Time of Day column\n", "\n", "Let's add a column that specifies the time of day (morning, afternoon, evening) when a sale was made.\n", "\n", "This can be done using the [cut](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.cut.html) function or by writing a custom function and using [apply](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html) to apply the function to each row of the Date_Time column.\n", "\n", "Here's how the `cut` function works, if we are specifying the bins and labels (there are a LOT of ways to use this function - see the docs):\n", "\n", "![image.png](attachment:b310b122-fed9-4fd8-8bc0-2f8104575b5a.png)\n", "\n", "`pd.cut(df[col], bins=[bin0,bin1,bin2,bin3], labels=['label0','label1','label2'])`\n", "\n", "*bins* is an iterable of the boundaries between labels. *labels* can be strings, numbers, etc. as you like. There must be 1 more label than there are bins. For 2 bins and 3 labels:\n", "* label0 is used for `bin0 <= df[col] <= bin1` \n", "* label1 is for `bin1 < df[col] <= bin2` \n", "* label2 is for `bin2 < df[col] <= bin3` \n", "\n", "An optional argument, `right=True` changes `<` to `<=` and `>=` to `>`. You can use values like negative or positive infinity for the ends of the bins if you're not sure what the min/max values are: `bins=[-np.inf,bin1,bin2,np.inf]`\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Pandas.cut()\n", "# 4 bin values,3 labels.\n", "# A Datetime column has a .dt attribute, which can be used to access datetime functions & attributes\n", "# Try: dir(df['Date_Time'].dt)\n", "df['Time_of_Day'] = pd.cut(df['Date_Time'].dt.hour,bins=[0,10, 16,24], labels=['morning','afternoon','evening'])\n", "df" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# What kind of data is Time_of_Day?\n", "print( df['Time_of_Day'].dtype)\n", "# https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.Categorical.html#pandas.Categorical" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get the categories\n", "# Categorical data picks up a .cat attribute \n", "print(df['Time_of_Day'].cat.categories)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "faCHhLX-vMtw" }, "outputs": [], "source": [ "# Two-way tables, let's see what kind of coffee is most populer at different times of the day.\n", "pd.crosstab(index=df['Time_of_Day'], columns=df['coffee_name'], margins=True\n", " )" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true, "id": "KzvZFygO0ibz", "jupyter": { "outputs_hidden": true } }, "source": [ "### Data slicing and grouping" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LE0RknOW0ib0" }, "outputs": [], "source": [ "#Extract a column by name \n", "df_new['money'].head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "6JrIilpE08Be" }, "outputs": [], "source": [ "# Calculate median transaction\n", "df_new['money'].median()" ] }, { "cell_type": "markdown", "metadata": { "id": "3gcURpQh0ib1" }, "source": [ "### Exercise - Working with a single column\n", "\n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "LXwRAzfQ0ib2" }, "outputs": [], "source": [ "# Calculate the descriptive statistics for only the money column in df\n", "# Get the money column as a Series\n", "df['money'].describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# OR get a single-column dataframe\n", "df[['money']].describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# OR - call describe() on the whole dataframe and extract just the \n", "# one you want. This is potentially VERY inefficient - this is a lot of computation \n", "# to run that gets ignored, and on large dataframes will be slow to execute. Use one of\n", "# the previous approaches.\n", "df.describe()[['money']]\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "wk7Er3ma0ib2" }, "outputs": [], "source": [ "# Get a count for each of the values in the Time_of_Day column in df\n", "pd.crosstab(index=df['Time_of_Day'], columns=\"count\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "08H11QC80ib2" }, "outputs": [], "source": [ "# Calculate the average (mean) money amount in df\n", "# Again, you can select either one column as a Series or a DataFrame\n", "df['money'].mean()" ] }, { "cell_type": "markdown", "metadata": { "id": "OR761ERr0ib2" }, "source": [ "### Grouping data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HxIkM-uT0ib3" }, "outputs": [], "source": [ "# Group data using Time_of_Day. observed=True is added because this is a Categorical column, it means\n", "# only deal with values that show up in that column.\n", "df_tod = df.groupby('Time_of_Day', observed=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "dX45khRv0ib3" }, "outputs": [], "source": [ "# Calculate mean of all numeric columns for the grouped object\n", "df_tod.mean(numeric_only=True)\n", "# What happens with df_rank.mean() ?\n", "# Most of the time, the \"grouping\" object is not stored, but is used as a step in getting a summary:\n", "#df.groupby('Time_of_Day', observed=True).mean([\"money\"])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "txqWe6sM0ib4" }, "outputs": [], "source": [ "# Another way - group_by, then refer to a single column, then get the mean\n", "df.groupby('Time_of_Day', observed=True)['money'].mean()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "TDRTnWXo0ib4" }, "outputs": [], "source": [ "# If we use double brackets Pandas will produce a DataFrame\n", "df.groupby('Time_of_Day', observed=True)[['money']].mean()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "nUMv2CM50ib5" }, "outputs": [], "source": [ "# Group using 2 variables - gender and rank:\n", "df.groupby(['Time_of_Day','coffee_name'], observed=True, sort=True)[['money']].sum()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Flights Data\n", "\n", "Next we'll load a new dataframe with airline flight data. This has more numerical data to play with." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bZCCJvtD0icD" }, "outputs": [], "source": [ "flights = pd.read_csv(\"http://rcs.bu.edu/examples/python/DataAnalysis/flights.csv\")\n", "flights.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "PcLvDYfOjNZv" }, "outputs": [], "source": [ "flights.info()\n" ] }, { "cell_type": "markdown", "metadata": { "id": "n_QMiWbN0ib5" }, "source": [ "### Exercise - Grouping data\n", "\n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bK_eeYF70ib5" }, "outputs": [], "source": [ "# Group data by the origin and carrier name and find the average distance and air_time for each\n", "x=flights.groupby(['origin','carrier'])[['distance','air_time']].mean()\n", "x" ] }, { "cell_type": "markdown", "metadata": { "id": "5M2fqDoL0ib6" }, "source": [ "### Filtering" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "bNCJvz590ib6" }, "outputs": [], "source": [ "# Find flights > 4000 miles\n", "# flights['distance']>4000 --> A Boolean Series which is used to filter the rows\n", "far = flights[flights['distance']>4000]\n", "far\n", "# Note the index column retains the original index value" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Oj51sat40ib6" }, "outputs": [], "source": [ "far.axes" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "xGZSM5EH0ib7" }, "outputs": [], "source": [ "# Select data for United Airlines\n", "ua = flights[flights['carrier'] == 'UA']\n", "ua.head()" ] }, { "cell_type": "markdown", "metadata": { "id": "iUOMOcM60ib8" }, "source": [ "### Exercise - Filtering data \n", "\n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0fCg9pRv0ib8" }, "outputs": [], "source": [ "# Using filtering, find the mean arrival delay for American Airlines (carrier name AA)\n", "# \n", "flights[flights['carrier']=='AA']['arr_delay'].mean()\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "DUHR3SlV0ib8" }, "outputs": [], "source": [ "# Challenge:\n", "# Count how many AA flights were over 2000 miles\n", "\n", "# To filter on two columns use the & operator, not the word \"and\", and don't\n", "# forget parentheses, ex. (expr1) & (expr2)\n", "flights[(flights['carrier']=='AA') & (flights['distance']>2000)].shape[0]" ] }, { "cell_type": "markdown", "metadata": { "id": "6Fir_77Y0ib8" }, "source": [ "### Slicing a dataframe" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "W4v86rQi0ib8", "scrolled": true }, "outputs": [], "source": [ "# Select the arr_delay column \n", "delay = flights['arr_delay']" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "sugc9L480ib-" }, "outputs": [], "source": [ "# Check data type of the result\n", "type(delay)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "XuISZ50f0ib-" }, "outputs": [], "source": [ "# Look at the first few elements of the output\n", "delay.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CKQS3knF0ib_" }, "outputs": [], "source": [ "# Select column salary and make the output to be a data frame\n", "delay = flights[['arr_delay']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gMPqOKc20ib_" }, "outputs": [], "source": [ "# Check the type\n", "type(delay)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4HBqgWTi0ib_" }, "outputs": [], "source": [ "# Select a subset of rows (based on their position):\n", "# Note 1: The location of the first row is 0\n", "# Note 2: The last value in the range is not included\n", "delay[0:10]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "4BEvyTpr0ib_" }, "outputs": [], "source": [ "# If we want to select both rows and columns we can use method .loc, try this on flights\n", "flights.loc[10:20, ['year', 'carrier','arr_delay']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Xg9Tuorn0icA" }, "outputs": [], "source": [ "# Unlike method .loc, method iloc selects rows (and columns) by absolute position:\n", "# iloc = integer location\n", "flights.iloc[10:20, [0,3,4,5]]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Exercise - Slicing a dataframe\n", "\n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "flights.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Create a new dataframe where you filter out flights below 1000 miles from flights\n", "# Call this data frame df_short\n", "df_short = flights[flights['distance'] >= 1000]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Extract rows 5:10 and columns ['carrier', 'tailnum'] of df_short using the .loc method\n", "# How many rows are in the output?\n", "df_short.loc[5:10, ['carrier', 'tailnum']]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Extract rows 5:10 and columns ['dep_time', 'arr_time'] from df_short using the iloc method\n", "# What are the values of the indices in the output?\n", "\n", "# You can hard-code the column numbers for the two columns, but let's find them instead\n", "# One way - the df_short.columns is a Pandas.Index type. Convert it to a Python list.\n", "cols = list(df_short.columns)\n", "# Now get the number from the list.index() method\n", "dep_time = cols.index('dep_time')\n", "arr_time = cols.index('arr_time')\n", "\n", "# now extract with iloc\n", "df_short.iloc[5:10,[dep_time,arr_time]]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Extract rows with index values [6, 12, 20, 22] and columns ['dep_time', 'arr_time'] from df_short\n", "# Hint: Use the loc method\n", "# This fails, because the filtering of flights resulted in the index in df_short \n", "# not having some index valiues - 12 and 22 aren't in the index.\n", "# One workaround is to re-index df_short.\n", "df_short = df_short.reset_index()\n", "df_short.loc[[6, 12, 20, 22], ['dep_time', 'arr_time']]\n" ] }, { "cell_type": "markdown", "metadata": { "id": "vzxWbXKF0icA" }, "source": [ "### Sorting the Data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "OXybOgop0icB" }, "outputs": [], "source": [ "# Sort the data frame df_new by service and create a new data frame\n", "df_sorted = flights.sort_values(by = 'distance')\n", "df_sorted.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Note the index of df_sorted retains the original position. Re-index.\n", "df_sorted = df_sorted.reset_index(drop=True)\n", "df_sorted.head()\n", "# What happens without drop=True?" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "mcrnORWR0icC" }, "outputs": [], "source": [ "# Restore the original order using the sort_index method\n", "df_sorted = flights.sort_values(by = 'distance')\n", "df_sorted = df_sorted.sort_index(axis=0, ascending = True)\n", "df_sorted.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "52sGV0z_0icD" }, "outputs": [], "source": [ "# Sort the data frame using 2 or more columns:\n", "df_sorted2 = flights.sort_values(by = ['distance', 'air_time'], ascending = [True,False])\n", "df_sorted2.head(15)" ] }, { "cell_type": "markdown", "metadata": { "id": "aBgfkBkK0icC" }, "source": [ "### Exercise - Sorting \n", "#### Solved" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Kh0qVebH0icD" }, "outputs": [], "source": [ "# Sort the data frame df by the dep_time (in descending order)\n", "# Store the output of this sorting in a dataframe called df_desc\n", "# Display the first 10 records of the output\n", "df_desc = flights.sort_values('dep_time', ascending = False)\n", "df_desc.head(10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Looping and DataFrames\n", "\n", "You can iterate over rows in a loop:\n", "```\n", "# use the iterrows() method\n", "sum_dist = 0\n", "for index, row in flights.iterrows():\n", " sum_dist += row['distance']\n", "```\n", "or using .loc():\n", "```\n", "sum_dist = 0\n", "for i in range(flights.shape[0]):\n", " sum_dist += flights.loc[i,'distance']\n", "```\n", "However, performance is generally _VERY POOR_, so this is to be *avoided* where a better alternative exists. The [apply](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.apply.html#pandas.DataFrame.apply) function performs much better if you must do an operation on every element in a column." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Numpy ndrrays\n", "Underlying each column of the data frame is a Numpy ndarray. Passing data from dataframes to functions (as found in scipy, your own functions, etc) sometimes requires that data frame columns or Pandas Series be converted to its ndarray representation. There's plenty of code that works with ndarrays as input that does not know anything about Pandas." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def my_func(x):\n", " a = np.empty_like(x)\n", " b = 0\n", " for i in range(len(x)):\n", " b += x[i]\n", " a[i] = b\n", " return a\n", "\n", "# Passing a Pandas Series is ok. This frequently works but\n", "# not always, for example if \"my_func\" was calling compiled C++ code\n", "# or is accelerated with the Numba library it likely won't handle \n", "# the Series correctly.\n", "vals = my_func(flights['distance'])\n", "\n", "# How about a DataFrame? Uncomment and this will fail.\n", "#vals2 = my_func(flights[['distance']])\n", "\n", "# Send the DataFrame as an ndarray\n", "vals3 = my_func(flights[['distance']].to_numpy())\n", "\n", "# Convert 2 columns to a 2-D ndarray\n", "numpy_vals = flights[['distance','air_time']].to_numpy()\n", "print(numpy_vals.shape)" ] }, { "cell_type": "markdown", "metadata": { "id": "XZjjbRia0icD" }, "source": [ "### Missing Values\n", "Now let's discuss the data in flights that is missing (\"NaN\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "IPfS620V0icE", "scrolled": true }, "outputs": [], "source": [ "# Select the rows that have at least one missing value\n", "flights[flights.isnull().any(axis=1)].head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AUI-q-wr0icE" }, "outputs": [], "source": [ "# Filter all the rows where arr_delay value is missing:\n", "flights1 = flights[flights['arr_delay'].notnull( )]\n", "flights1.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "oem6cDo90icF" }, "outputs": [], "source": [ "# Remove all the observations with missing values\n", "flights2 = flights.dropna()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "NoCw8dmx0icH" }, "outputs": [], "source": [ "# Fill missing values with zeros\n", "nomiss =flights['dep_delay'].fillna(0)\n", "nomiss.isnull().any()" ] }, { "cell_type": "markdown", "metadata": { "id": "d-Q3P5SO0icJ" }, "source": [ "### Exercise Count missing data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "uAfRQCPs0icL" }, "outputs": [], "source": [ "# Count how many missing pieces of data there are in the dep_delay and arr_delay columns\n", "# To look at just those two columns, subset the dataframe to just those two. Use isnull()\n", "# to create a dataframe with 2 Boolean columns.\n", "tmp = flights[['dep_delay','arr_delay']].isnull() \n", "# Apply tmp.any() across rows to make a single Boolean Series where the\n", "# value is True when it's True in either or both columns\n", "tmp2 = tmp.any(axis=1)\n", "# Then sum tmp2\n", "print(f'Step-by-step count of missing data: {tmp2.sum()}')\n", "\n", "# Better yet - do this all in one line.\n", "count = flights[['dep_delay','arr_delay']].isnull().any(axis=1).sum()\n", "print(f'One-liner count of missing data: {count}')" ] }, { "cell_type": "markdown", "metadata": { "id": "JGOFyTVm0icN" }, "source": [ "### Common Aggregation Functions:\n", "\n", "The following functions are commonly used functions to aggregate data.\n", "\n", "|Function|Description\n", "|-------|--------\n", "|min | minimum\n", "|max | maximum\n", "|count | number of non-null observations\n", "|sum | sum of values\n", "|mean | arithmetic mean of values\n", "|median | median\n", "|mad | mean absolute deviation\n", "|mode | mode\n", "|prod | product of values\n", "|std | standard deviation\n", "|var | unbiased variance\n", "\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "q5SgaHvG0icO" }, "outputs": [], "source": [ "# Find the number of non-missing values in each column\n", "flights.describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "gmZ5Dekd9N6g" }, "outputs": [], "source": [ "flights.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Get mean values\n", "flights.mean(numeric_only=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HrhCXQBi0icP" }, "outputs": [], "source": [ "# We can use agg() methods for aggregation:\n", "flights[['dep_delay','arr_delay']].agg(['min','mean','max'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# The value returned is a dataframe:\n", "agg_vals = flights[['dep_delay','arr_delay']].agg(['min','mean','max'])\n", "agg_vals.info()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AVsIj0La0icQ" }, "outputs": [], "source": [ "# An example of computing different statistics for different columns\n", "flights.agg({'dep_delay':['min','mean','max'], 'carrier':['nunique']})" ] }, { "cell_type": "markdown", "metadata": { "id": "sPGu6w9j0icR" }, "source": [ "## Exploring data using graphics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Graphics with the flights dataset" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "QJBJNupL0icR" }, "outputs": [], "source": [ "# Use matplotlib to draw a histogram of a salary data\n", "plt.hist(flights['distance'],bins=8, density=True)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "_x-c3hL20icS" }, "outputs": [], "source": [ "# Use seaborn package to draw a histogram\n", "sns.displot(flights['distance'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "CAwwtPMR0icS" }, "outputs": [], "source": [ "# Let's add a distance bin column. Use 10 bins.\n", "flights['distances_bins'] = pd.cut(flights['distance'],bins=10, labels=range(10))\n", "# Use regular matplotlib function to display a barplot\n", "flights.groupby(['carrier', 'distances_bins'], observed=True)[['air_time']].count().plot(kind='bar')" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Q4rEH0cx0icS" }, "outputs": [], "source": [ "# Use seaborn package to display a barplot\n", "sns.set_style(\"whitegrid\")\n", "# Randomly grab some rows to avoid overwhelming the plot functions\n", "sm_flights = flights.sample(n=300)\n", "ax = sns.barplot(x='carrier',y ='arr_delay', data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "AzBt0Vih0icU" }, "outputs": [], "source": [ "# Violinplot\n", "sns.violinplot(x = \"distance\", data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "1ieYE6KF0icU" }, "outputs": [], "source": [ "# Scatterplot in seaborn\n", "sns.jointplot(x='dep_delay', y='arr_delay', data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0u2HEEOp0icV" }, "outputs": [], "source": [ "# If we are interested in linear regression plot for 2 numeric variables we can use regplot\n", "sns.regplot(x='dep_delay', y='arr_delay', data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "0WD6YnSy0icV" }, "outputs": [], "source": [ "# Box plot\n", "sns.boxplot(x='arr_delay', y='carrier', data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "Dn-ibN7k0icW" }, "outputs": [], "source": [ "# Swarm plot\n", "sns.swarmplot(x='arr_delay', y='carrier', data=sm_flights)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "7aR8SY2r0ich" }, "outputs": [], "source": [ "# Factorplot\n", "sns.catplot(x='arr_delay', y='carrier', data=sm_flights, kind='bar')\n", "print(flights.groupby('carrier')[['arr_delay']].mean())" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "UfRCtVU20ici" }, "outputs": [], "source": [ "# Pairplot \n", "sns.pairplot(sm_flights[['arr_delay','dep_delay']])" ] }, { "cell_type": "markdown", "metadata": { "id": "8ABnDMpa0ici" }, "source": [ "### Exercise 7 Graphing data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "HP7rVYGp0ici" }, "outputs": [], "source": [ "# Use the seaborn package to explore the dependency of arr_delay on dep_time\n", "# in the flights dataset. You can use a scatterplot or regplot using sm_flights\n", "# " ] }, { "cell_type": "markdown", "metadata": { "id": "iWKUzQNP0icQ" }, "source": [ "## Descriptive statistics\n", "Statistics that are used to describe data. We have seen methods that calculate descriptive statistics before with the DataFrame describe() method. \n", "\n", "Descriptive statistics summarize attributes of a sample, such as the min/max values, and the mean (average) of the data. Below is a summary of some additional methods that calculate descriptive statistics." ] }, { "cell_type": "markdown", "metadata": { "id": "7qiHcIAJ0icQ" }, "source": [ "|Function|Description\n", "|-------|--------\n", "|min | minimum\n", "|max | maximum\n", "|mean | arithmetic mean of values\n", "|median | median\n", "|mad | mean absolute deviation\n", "|mode | mode\n", "|std | standard deviation\n", "|var | unbiased variance\n", "|sem | standard error of the mean\n", "|skew| sample skewness\n", "|kurt|kurtosis\n", "|quantile| value at %\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "y2izjRmU0icQ" }, "outputs": [], "source": [ "# Recall the describe() function which computes a subset of the above listed statistics\n", "flights.dep_delay.describe()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "yh4S78a80icR" }, "outputs": [], "source": [ "# find the index of the maximum or minimum value\n", "# if there are multiple values matching idxmin() and idxmax() will return the first match\n", "flights['dep_delay'].idxmin() #minimum value" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "id": "k9D_xUPK0icR" }, "outputs": [], "source": [ "# Count the number of records for each different value in a vector\n", "flights['carrier'].value_counts()" ] }, { "cell_type": "markdown", "metadata": { "id": "xM391jsk0icl" }, "source": [ "## Tutorial Evaluation\n", "After the tutorial please submit an evaluation by clicking on this link [Tutorial Evaluation](http://scv.bu.edu/survey/tutorial_evaluation.html)" ] } ], "metadata": { "colab": { "collapsed_sections": [ "M3FbA7um0ibR", "vYNDrGEL0ibY", "3f4HD5E90ibn", "PQprwXx40ibo", "tj85w0PMttNG", "KzvZFygO0ibz", "5M2fqDoL0ib6", "6Fir_77Y0ib8", "vzxWbXKF0icA", "XZjjbRia0icD", "JGOFyTVm0icN", "iWKUzQNP0icQ", "sPGu6w9j0icR", "a2XW__P90icj" ], "name": "dataScience.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.11" } }, "nbformat": 4, "nbformat_minor": 4 }