Dear all,
For Homework 5 you will need:
sd() function) and Python (using std() function from numpy package.std function in Python that returns the same result that sd() function in R.Below you will find detailed instructions, links, and examples that should help you to complete this homework. Please read instructions for each step very carefully. You should develop your code using SCC cluster. Once your code works interactively you can run it as a job.
You will need to submit your work to the Blackboard. You can either submit your Python script along with the output from the job you ran on the SCC or (if you decide to get an extra credit) a single pdf or html document that will be created as an output if you submit a job using your jupyter notebook.
There is a very detailed documentation for the Jupyter notebook .
You can find Markdown basics as well as many examples how to use LaTeX commands to include formulas if you would like to complete extra cretdit.
Open Rstudio and create a vector x:
x <- c( 21,7,5,3,8)
Calculate standard deviation:
sd(x)
[1] 7.085196
In Python environment (either in spyder or Jupyter notebook), load 3 packages that you will need for this homework:
import math
import numpy as np
import pandas as pd
Create a vector (called array in Python) using the same values:
x = [21,7,5,3,8]
Use std() function from package numpy to calculate standard deviation:
np.std(x)
Why the values are so different?
Your task is to write your own function that calculates a standard deviation value that is equal to the one you get in R.
You cannot use any Python functions other then len(), which returns the length of the vector.
To complete this homework you might need to use loops, conditional if statements and function defitions. Below I provide some simple examples for all three. Please remember that the body of if statement, loop and function must have indentation and add an empty line at the end to separate your function, loop or if from the following code.
# Example of if:
if len(x) > 0:
print("Length of the array is", len(x))
else:
print("Array has no elements")
y = [] # empty array
if len(y) > 0:
print("Length of the array is", len(y))
else:
print("Array has no elements")
# calculate sum of values in array:
sum = 0
for value in x:
sum = sum + value
print("Sum of elements in x is", sum)
#function definition
def my_mean(array):
sum = 0
for value in array:
sum = sum + value
return sum/len(array)
# Call function (check to make sure mean() function in R returns the same value:
my_mean(x)
# Let's check with Python function mean:
np.mean(x)
But my function does not work if I call it with an empty vector
y=[]
my_mean(y)
# Improved version of function mean:
#function definition
def my_mean(array):
if len(array) <1:
return np.nan
sum = 0
for value in array:
sum = sum + value
return sum/len(array)
# Apply function to an empty vectory
y=[]
my_mean(y)
# Check if it still works on a regular vector
my_mean(x)
# To read this file on the SCC, you should use /project/bs803 path:
# fhs = pd.read_sas("/project/bs803/fhs.sas7bdat")
# You can explore this dataset using head command:
fhs.head()
# Now let's calculate mean of column AGE using Pandas mean() method
fhs['AGE'].mean()
# We can also run mean() function from numpy package:
np.mean(fhs['AGE'])
# What about my own mean function:
my_mean(fhs['AGE'])
Looks like all 3 agree about mean value of column AGE. What about standard deviation?
# standard deviation method from pandas package
fhs['AGE'].std()
# standard deviation function from numpy package
np.std(fhs['AGE'])
As you can see the above 2 values differ and in R we get slightly different value then in the std() function from numpy... Why? Your task is to implement your own function my_std() that will compute standard deviation the way R and pandas package do, so you get the same value as sd() function in R.
First check it on a simple vector x that we defined earlier. Make sure your function works if you pass an empty vector too! When everything works fine, try to apply it to the column AGE in FHS dataset above.
Please make sure you add to your python script a detailed explanation of what you are doing. If you are creating a regular python file in spyder or other Python environment use comments. Single line comments in Python are similar to R, but if you want to have multi-line comments in Python, you can use the following syntax:
'''
This function calculates mean value of an array.
I calculate this value as a sum of all values in array divided by the length of the vector
'''
If however you are using a Jupyter notebook, you can place your text into a markdown cell.
Check your code running it interactively to make sure it runs without errors. Once it runs correctly, you can run it as a batch job on the SCC
If you want to submit a Python script (hw5.py) on the SCC, you need to create a batch script (job.qsub or similar) just like we did for SAS and R script, but at the bottom of your script you should put the following lines:
module load python3/3.7.7
python hw5.py
For the extra credit you can execute a Jupyter Notebook on the SCC instead of Python script. In this case you need to include the following lines into your job.qsub file:
module load python3/3.7.7
module load pandoc/2.5
module load texlive/2018
jupyter nbconvert --to notebook --execute hw5.ipynb
jupyter nbconvert hw5.nbconvert.ipynb --to pdf
Or if you want to save it into HTML format then:
module load python3/3.7.7
jupyter nbconvert --execute hw5.ipynb
Below are some helpful links if you want to learn more about Python:
Learn Python interactively
Link to the html page with many useful commands from pandas and seaborn packages that we went through during the class.