dar-f24-assignment3-template.Rmd

---
title: 'CTBench Eval Project Notebook:'
author: "Your Name Here"
date: "`r format(Sys.time(), '%d %B %Y')`"
output:
  html_document:
    toc: true
    number_sections: true
    df_print: paged
  pdf_document: 
    toc: true
    number_sections: true
    fig_caption: false
subtitle: DAR Assignment 3 (Fall 2024)
---
```{r setup, include=FALSE}

# Required R package installation; RUN THIS BLOCK BEFORE ATTEMPTING TO KNIT THIS NOTEBOOK!!!
# This section  install packages if they are not already installed. 
# This block will not be shown in the knit file.
knitr::opts_chunk$set(echo = TRUE)

# Set the default CRAN repository
local({r <- getOption("repos")
       r["CRAN"] <- "http://cran.r-project.org" 
       options(repos=r)
})

# Required packages for CTEval analysis
if (!require("rmarkdown")) {
  install.packages("rmarkdown")
  library(rmarkdown)
}


if (!require("knitr")) {
  install.packages("knitr")
  library(rmarkdown)
}
if (!require("tidyverse")) {
  install.packages("tidyverse")
  library(tidyverse)
}
if (!require("stringr")) {
  install.packages("stringr")
  library(stringr)
}

if (!require("ggbiplot")) {
  install.packages("ggbiplot")
  library(ggbiplot)
}

if (!require("pheatmap")) {
  install.packages("pheatmap")
  library(pheatmap)
}
if (!require("plotrix")) {
  install.packages("plotrix")
  library(plotrix)
}

```

# DAR ASSIGNMENT 3: Clinical Trials Benchmark Evaluation (CTBench) Research Overview

## General Overview: 

**CTBench** is a benchmark to assess language models (LMs) in aiding clinical study design. Given study-specific metadata, CTBench evaluates ability of different LMs to determine the baseline features of a clinical trial (CT), which include demographics and other relevant features collected at a trial's start from all participants. These baseline features, typically presented in CT publications as "Table 1," are crucial for characterizing study cohorts and validating results. Baseline features, including _confounders_ and _covariates_, are also necessary for accurate treatment effect estimation in studies involving observational data. 

CTBench consists of two datasets: `CT-Repo`, containing baseline features from 1,690 clinical trials sourced from this http URL, and `CT-Pub`, a 100-trial subset with more comprehensive baseline features gathered from relevant publications. Two LM-based evaluation methods are developed to compare the actual baseline feature lists against LM-generated responses. `ListMatch-LM` and `ListMatch-BERT` use `GPT-4o` and `BERT` scores (at various thresholds), respectively, for evaluation. To establish baseline results, advanced prompt engineering techniques using `LLaMa3-70B-Instruct` and `GPT-4o` in zero-shot and three-shot learning settings are applied to generate potential baseline features. 

The performance of `GPT-4o` as an evaluator is validated through _human-in-the-loop_ evaluations on the CT-Pub dataset, where clinical experts confirm matches between actual and LM-generated features. The results highlight a promising direction with significant potential for improvement, positioning CTBench as a useful tool for advancing research on AI in CT design and potentially enhancing the efficacy and robustness of CTs. 

Start by reading more details about CTBench in this paper:

https://arxiv.org/abs/2406.17888

## CTBenchEval Goals

The high-level goals of CTBenchEval project are to:

1. Imagine you are trying your own LLM approach and want to compare to the published CTBench results. 

* Can you conduct this experiment using the information provided in the CTBench paper and CTBench github?
* Can you perform the analysis of the results and obtain the appropriate precision, recall, and F1 statistics?

2. Provide a more comprehensive and deeper evaluation of CTBench results to gain further insights into the strengths and weaknesses of LLM solutions to CTBench to answer questions such as: 

* "Are there specific type of subgroups of trials, descriptors, or interventions that LLM seems to perform better or worse on?"  
* "Should we develop an app to do more comprehensive evaluation". 
* "Do baseline descriptor evolve through time and if so does this have impact on the LLM performance?" 
* "Can we evaluate if the descriptors proposed by LLM but not matched to clinicaltrial.gov data might actually be good choices?"

3. Create a demonstration tool to assist in baseline descriptor choices for clinical trial design  inspired by CTBench.  This tool could potentially have expanded capabilities such as explanation of why choices were made. 

4. What are opportunities to improve CTBench. Potential questions include: 

* "How can we make the  evaluation software provided effective and easy to use?" 
* "How can we make CTBench readily extensible to more clinical trials?" 

# DAR ASSIGNMENT 3 (Introduction): Introductory DAR Notebook

This notebook is broken into two main parts:

* **Part 1:** Preparing your local repo for **DAR Assignment 3**
* **Part 2:** Loading the CTBench Eval Datasets
* **Part 3:** Individual analysis of your team's dataset

**NOTE:** The RPI github repository for all the code and data required for this notebook may be found at:

* https://github.rpi.edu/DataINCITE/DAR-CTEval-F24

* **Part 4:** Be prepared to discuss your results in you team breakout. 

# DAR ASSIGNMENT 3 (Part 1): Preparing your local repo for Assignment 3

In this assignment you'll start by making a copy of the Assignment 3 template notebook, then you'll add to your copy with your original work. The instructions which follow explain how to accomplish this. 

## Cloning the `DAR-CTEval-F24` repository

* Access RStudio Server on the IDEA Cluster at http://lp01.idea.rpi.edu/rstudio-ose/
    * REMINDER: You must be on the RPI VPN if off campus!!
* Access the Linux shell on the IDEA Cluster by clicking the **Terminal** tab of RStudio Server (lower left panel). 
    * You now see the Linux shell on the IDEA Cluster
    * `cd` (change directory) to enter your home directory using: `cd ~`
    * Type `pwd` to confirm
    * NOTE: Advanced users may use `ssh` to directly access the Linux shell from a macOS or Linux command line
* Type `git clone https://github.rpi.edu/DataINCITE/DAR-CTEval-F24` from within your `home` directory
    * Enter your RCS ID and your saved personal access token when asked
    * This will create a new directory `DAR-CTEval-F24`
* In the Linux shell, `cd` to `DAR-CTEval-F24/StudentNotebooks/Assignment03`
    * Type `ls -al` to list the current contents
    * Don't be surprised if you see many files!
* In the Linux shell, type `git checkout -b dar-yourrcs` where `yourrcs` is your RCS id
    * For example, if your RCS is `erickj4`, your new branch should be `dar-erickj4`
    * It is _critical_ that you include your RCS id in your branch id!
* Back in the RStudio Server UI, navigate to the `DAR-CTEval-F24/StudentNotebooks/Assignment03` directory via the **Files** panel (lower right panel)
    * Under the **More** menu, set this to be your R working directory
    * Setting the correct working directory is essential for interactive R use!

You're now ready to start coding Assignment 3!

## Creating your copy of the Assignment 3 notebook

1. In RStudio, make a **copy** of `dar-f24-assignment3-template.Rmd` file using a *new, original, descriptive* filename that **includes your RCS ID!**
    * Open `dar-f24-assignment3-template.Rmd`
    * **Save As...** using a new filename that includes your RCS ID
    * Example filename for user `erickj4`: `erickj4-assignment2-f24.Rmd`
    * POINTS OFF IF:
       * You don't create a new filename!
       * You don't include your RCS ID!
       * You include `template` in your new filename!
2. Edit your new notebook using RStudio and save
    * Change the `title:` and `subtitle:` headers (at the top of the file)
    * Change the `author:` 
    * Don't bother changing the `date:`; it should update automagically...
    * **Save** your changes
3. Use the RStudio `Knit` command to create an HTML file; repeat as necessary
    * Use the down arrow next to the word `Knit` and select **Knit to HTML**
    * You may also knit to PDF...
4. In the Linux terminal, use `git add` to add each new file you want to add to the repository
    * Type: `git add yourfilename.Rmd` 
    * Type: `git add yourfilename.html` (created when you knitted)
    * Add your PDF if you also created one...
5. When you're ready, in Linux commit your changes: 
    * Type: `git commit -m "some comment"` where "some comment" is a useful comment describing your changes
    * This commits your changes to your local repo, and sets the stage for your next operation.
6. Finally, push your commits to the RPI github repo
    * Type: `git push origin dar-yourrcs` (where `dar-yourrcs` is the branch you've been working in)
    * Your changes are now safely on the RPI github.
7. **REQUIRED:** On the RPI github, **submit a pull request.**
    * In a web browser, navigate to https://github.rpi.edu/DataINCITE/DAR-CTEval-F24
    * In the branch selector drop-down (by default says **master**), select your branch
    * **Submit a pull request for your branch**
    * One of the DAR instructors will merge your branch, and your new files will be added to the master branch of the repo. _Do not merge your branch yourself!_

# DAR ASSIGNMENT 3 (Part 2): Loading the CTBench Eval Datasets

In this CTBench there are two sorts of data: clinical trial data used to generate the prompts and results data that shows the results.  For your conveniences, these dataset have been converted to R Rds files regardless of how they appear originally.   

* **Data Rds:** 

These are the datasets that describe each clinical trial and give it's baseline descriptors. Each row is a different clinical trial.

  * `CT_Repo_data.Rds` This contains selected clinical trials and their attributes from clinicaltrial.gov data repository for clinical trials. 
    
  * `CT_Pub_data.Rds` These contain a subset of clinical trials and their attributes. But the descriptors are taken from  the clinical trial publications.

    
* **Results Rds:** 

These include the results of various LLM on each trial.  

  * `trials.responses.Rds` contains the results of the LLM for each clinical trial and model combination. 
  * `trials.matches.Rds` contains the specific matches that the evaluation LLM made between the candidate and references descriptors.

NOTES: 

   * All of these datasets can be found in `/academics/MATP-4910-F24/DAR-CTEval-F24/Data`

## Load the CTBench Eval _Data_

Each of these datastructure has one row per trial. The features are given below. The size of each table is provided.  Comma separted lists are used for lists of features. 

  <br>    
  
<b>Table 1: Features of CT_Repo.df </b>
```
 Number Name                                    Notes
 ------ -----------------------------------     -----------------------------------------
 [1]    "NCTId"                                 Unique ID of each clinical trial                    
 [2]    "BriefTitle"                            Title of Trial                 
 [3]    "EligibilityCriteria"                   Eligibility Criteria           
 [4]    "BriefSummary"                          Summary of Trials                  
 [5]    "Conditions"                            Health Conditions the Trial addresses
 [6]    "Interventions"                         Intervention (a.k.a treatments)    
 [7]    "PrimaryOutcomes"                       Measure of success or failure of trial
 [8]    "BaselineMeasures"                      Original Features in ClinicalTrial.gov          
 [9]    "BaselineMeasures_Processed"            Cleaned up Features used in CTBench

```


<b>Table 2: Features of CT_Pub.df.</b>
```
 Number Name                                    Notes
 ------ -----------------------------------     -----------------------------------------
 [1]    "NCTId"                                 Unique ID of each clinical trial                    
 [2]    "BriefTitle"                            Title of Trial                 
 [3]    "EligibilityCriteria"                   Eligibility Criteria           
 [4]    "BriefSummary"                          Summary of Trials                  
 [5]    "Conditions"                            Health Conditions the Trial addresses
 [6]    "Interventions"                         Intervention (a.k.a treatments)    
 [7]    "PrimaryOutcomes"                       Measure of success or failure of trial 
 [8]    "BaselineMeasures"                      Original Features in ClinicalTrial.gov          
 [9]    "Paper_BaselineMeasures"                Original Features in trial paper        
 [10]   "Paper_BaselineMeasures_Processed"      Cleaned up Features used in CTBench

```

* For `CT_Repo.df`, the reference baseline descriptors used in the experiments are in a comma separated list in  `BaselineMeasures_Processed`. 
* For `CT_Pub.df`, the reference baseline decsriptors used in the experiments are in a comma separated list in  `Paper_BaselineMeasures_Processed`. 

First we load the data and see that `CT_Pub.df` contains 103 trials with 10 dimensions and that `CT_Repo.df` contains 1693 trials with 9 dimensions.   Note that 3 of the trials in each dataset are reserved to be used as examples to be included in the prompts, a process called three shot learning. 

```{r}
# Load the CT_Pub data
CT_Pub.df<- readRDS("../../Data/CT_Pub_data.Rds")

dim(CT_Pub.df)

# Load the CT_Repo data
CT_Repo.df<- readRDS("../../Data/CT_Repo_data.Rds")

dim(CT_Repo.df)


# Uncomment these if you want to look at a few examples of each dataset, but don't knit into your final notebook. 
#head(CT_Pub.df,5)

#head(CT_Repo.df,5)
```

## Load the CTBench Eval _Results_

This is the results file.  There is one  row given for each trial evaluated  x number of models.  We show you how to  prepare data  for CT_Pub. CT_Repo can be done similarly.  

These are the features in CT_Pub_responses.df.

<b>Table 3: Features of CT_Pub.responses.df.</b>

```
 Number Name                      Notes
 ------ ------------------------- -----------------------------------------
 [1]    "trial_id"                Unique trial id - same as NCTID     
 [2]    "trial_group"             trial address what group of disease.    
 [3]    "model"                   LLM model used   
 [4]    "gen_response"            Result generated  
 [5]    "processed_gen_response"  Cleaned up result generated
 [6]    "len_matches"             # of matching descriptors
 [7]    "len_reference"           # unmatched descriptors from reference
 [8]    "len_candidate"           # unmatched descriptors from LLM response
 [9]    "precision"               precision for this trial and model
[10]    "recall"                  recall for this trial and model
[11]    "f1"                      F1 score for this trial and model 
```

These are the features in CT_Pub_matches.df.  

<b>Table 4: Features of CT_Pub.matches.df</b>
```
 Number Name               Notes
 ------ -----------------  -----------------------------------------
 [1]    "trial_id"         Unique trial id - same as NCTID     
 [2]    "model"            LLM model used   
 [3]    "reference"        Matched reference feature (NA if none)   
 [4]    "candidate"        Matched candidate feature (NA if none) 

```

For each trial_id and model, an LLM is used to match the list of reference features with the list of candidate features proposed by the LLM.   

If the table has an entry such as 
   `trial_id_A model_id_B reference_C  candidate_D`, this means that for `trial_id A` using `model B`,  `reference descriptor C` was matched to `candidate descriptor D`.   
   
If the table has an entry such as 
   `trial_id_A model_id_B reference_C  NA`, this means that for `trial_id A` using `model B`,  `reference descriptor C` had _no match_ in the candidate list.
   
If the table has an entry such as 
   `trial_id_A model_id_B NA  candidate D`, this means that for `trial_id A` using `model B`,  `candidate descriptor D` had _no match_ in the reference list.   

```{r}
# Load the trials.responses
CT_Pub.responses.df<- readRDS("../../Data/trials.responses.Rds")

# convert model and type to factors
CT_Pub.responses.df$trial_group <- as.factor(CT_Pub.responses.df$trial_group)
CT_Pub.responses.df$model <- as.factor(CT_Pub.responses.df$model)

# check out the size
dim(CT_Pub.responses.df)

# Load the trials.matches
CT_Pub.matches.df<- readRDS("../../Data/trials.matches.Rds")

# convert model and type to factors
CT_Pub.matches.df$model <- as.factor(CT_Pub.matches.df$model)

dim(CT_Pub.matches.df)


# Look at some samples of each file
# Commment these out before you make pdf
#head(CT_Pub.matches.df,5)

#head(CT_Pub.responses.df,5)
```


The same process can be used for CT_Repo.  But be aware that variable names may be slightly different.  


# Analysis of Response Data 

For each clinical trial, the evaluation program (which calls two different LLM) calculates several measures of how good the candidate descriptor proposed for each LLM are as compared to the reference descriptors.  

The Bert score is a measure of similarity of the candidate and reference LLM in the the latent space of the Bert LLM.  This is a quick but very rough way to calculate similarity. You can read about Bert Scores here.  https://arxiv.org/abs/1904.09675

A more accurate evaluation is done by matching each candidate descriptor with at most 1 reference descriptor.   This is done using the LLM GPT-4o. Let matches.len = number of candidate descriptors matched with the reference descriptors.   Let  candidate.len = number of unmatched candidate descriptors and reference.len = number of unmatched candidate descriptor.   

Precision measures the proportion of candidate descriptors that were matched. Recall measure the proportions of the reference descriptor that were in the candidate descriptors.  F1 is a standard measure that combines precision and recall.  These calculations have already been done for each trial. 

For example, say reference = "Age, sex (OHM), race/ethnicity, cardiovascular disease, socio-economic status" and candidate = "age, gender, race, SBP, DBP, cholesterol ).

* There are three matches:  (Age, age), (sex(OHM),gender}, (race/ethnicity,race)  
* There are two unmatched reference  descriptors: socio-econmic status and blood pressure.
* There are three unmatched candidate descriptors:  SBP, DBP, cholesterol.

This is how precision, recall, and f1 are calculated: 

* Example:  

`precision <- matches.len / (matches.len + candidate.len)`
`precision =  3/(3+3) = .5`

* Example:

`recall <- matches.len / (matches.len + reference.len)`
`recall =  3/5 = .6`

* Example:

`f1 <- ifelse(precision == 0 | recall == 0, 0, 2 * (precision * recall) / (precision + recall))`
`f1 = 0.5454545`

**Note:** one could argue that the candidate descriptors are actually better than measured by these metrics since blood pressure and cholesterol are test used to determine  cardiovascular disease. What to do it about this is an open question. 

_The goal in this analysis is to see if different subgroups of data have different average precision, average recall, and average F1 scores._

## How do results differ by model on CT-pub?

We want to summarize the results across the trials.  So we calculate mean and standard error  of statistics for each trial for each model. Standard error is the standard deviation divided by the `sqrt()` of the number of total observations.   The dplyr package is being used to calculate the results in an easy way. Check out the cheatsheet on dplyr for more information on using dplyr https://rstudio.github.io/cheatsheets/html/data-transformation.html

\setcounter{table}{4}
```{r}

CT_Pub_model_results.df <- CT_Pub.responses.df %>% 
  group_by(model) %>% 
  summarize(meanPrecision=mean(precision),
            sePrecision=std.error(precision),
            meanRecall=mean(recall),
            seRecall=std.error(recall),
            meanF1=mean(f1),sef1=std.error(f1))

kable(CT_Pub_model_results.df, caption="Differences by Model on CT-Pub")

```

The models are `gpt4-omni` and `llama3-70`.  The ending `zs` means _zero shot_ and `ts` means _three shot_ prompts.  Here we can see  differences between the results for different models and prompts.  But this differences may not always be statistically significant.   No significance tests were done in the original CTBench paper. 

## How do results differ by model and trial type on CT-pub? 

Now we calculate calculate mean and standard error of response measures for different model and trial type combinations and display results in a table.  

```{r}

# Create the dataframe
CT_Pub_MT_results.df <- CT_Pub.responses.df %>% 
  group_by(model,trial_group) %>% 
  summarize(meanPrecision=mean(precision),
            sePrecision=std.error(precision),
            meanRecall=mean(recall),
            seRecall=std.error(recall),
            meanF1=mean(f1),
            sef1=std.error(f1))

kable(CT_Pub_MT_results.df, caption="Differences by Model and Subgroup on CT-pub")
```

Here we can see that predicting descriptors seems to be harder for some combinations of models and trial types.    But further analysis is needed to present the results in a more informative way and see if the differences are statistically significant. 

# Do we see differences in results for different candidate descriptors?  

This is a sample analysis of the matches data frame. The goal is to count the number of trials for the 'gpt4-onmi-zs' model results in which each reference term is accurately matched.   It does this by using tidyverse to find all the matches row with a value in reference and  `model="gpt4-omni-zs"`.  Then it groups by references and computes the number of trial the term occurs and the number of trials where match failed (i.e. how many have `NA` for candidate). Then it does a mutate to add the accuracy. Then it sorts the descriptors by the number of trials.   

```{r}
CT_Pub_reference_count.df <- CT_Pub.matches.df %>% 
  filter(model=="gpt4-omni-zs") %>%
  filter(!is.na(reference)) %>%   
  group_by(reference) %>% 
  dplyr::summarize(trials=n(),
                   wrong ={ sum(is.na(candidate))}) %>% 
  mutate(accuracy = 1- wrong/trials) %>% arrange(desc(trials))

# Number of different baseline descriptors
nrow(CT_Pub_reference_count.df)

# these are top 20 most common descriptors. 
kable(head(CT_Pub_reference_count.df,20),caption="Accuracy of Top 20 descriptors in CT_Pub")

```

Here we see that some descriptors are really easy like age but others like Hypertension may be more challenging, but the
sample sizes can be very low.    CT-Pub is really too small to do this type of analysis.  Ct-repo would yield more definitive results that may show significant different. In addition, different trials may use different terms to represent similar things.  For example, hypertension means a subject's high systolic or diastolic blood pressure exceeds desired thresholds.  So many trials may measure blood pressure but only a few of them may have a descriptor called "hypertension."  So some more thought needs to be put in this type of analysis. 

# Your Job 

You job is to do a more in-depth analysis of the results of the two models. Each member of your team can focus on a different question or model. You can use any analyses or visualizations in R that you like.  

Here are some ideas for questions to pursue, but feel free to make up your own. _Try to make up and answer at least two questions._  The additional questions can be a follow-up to previous questions.   Coordinate with your team so you look at different questions.  

Here are some ideas for questions to inspire you: 

1. Does the LLM models perform differently in terms of precision, recall, and F1 scores?   
2. Are prompts for some disease types e.g. (group_types) harder than others?  Does this difference hold across different models? Note we will refer to this as a subgroup analysis. 
3. How does the performance compare for CT_Pub and CT_Repo? 
4. Can you use multiple regression on the response to understand how different factors (e.g. model, group_type, and/or source(Repo or Pub)) affect the results controlling for the others? Note this can let you know the significance of any difference too. 
5. Are there difference in performance associated with different  baseline descriptors that are used frequently, are some more harder than others?  Here you will have to determine how often descriptors are used in a trial, and then find subgroup trials that use each descriptors, and then compare performance across these subgroups?   Probably you want to use CT_Repo for this analysis since it has more clinical trials. 
6. Are any  differences that you see statistically different?  You could use tests like t-test, paired t-test, and anova.
7. How can you make compelling visualization to illustrate your results effectively? Feel free to experiment with multiple ways. 
8. For those of you with NLP skills, can you cluster the baseline descriptors in some semantically meaningful way?  Can you characterize what the baseline descriptors are in each clusters?  Note you might want to do this task for one group_type like diabetes to make the clustering easier.  If you find the subgroups of trials associated with each cluster of group_types do they have different performance?   
9. Can you develop a measure of performance for  individual baseline descriptors (here you would probably want to stick to frequently used descriptors). Are some descriptors easy or hard?  Do you see differences in descriptor performance across models or group_types? 
10. If you observe any differences, can you develop any hypotheses on why this could by further examination of the data? Can you find examples that illustrate your point?  
11. Can you reproduce the results shown in the CTBench paper using the data we have?  Can you think of other ways to present those results?


## Analysis: Question 1 (Provide short name)

### Question being asked 

_Provide in natural language a statement of what question you're trying to answer_

### Data Preparation

_Provide in natural language a description of the data you are using for this analysis_

_Include a step-by-step description of how you prepare your data for analysis_

_If you're re-using dataframes prepared in another section, simply re-state what data you're using_

```{r, result01_data}
# Include all data processing code (if necessary), clearly commented

```

### Analysis: Methods and results

_Describe in natural language a statement of the analysis you're trying to do_

_Provide clearly commented analysis code; include code for tables and figures!_

```{r, result01_analysis}
# Include all analysis code, clearly commented
# If not possible, screen shots are acceptable. 
# If your contributions included things that are not done in an R-notebook, 
#   (e.g. researching, writing, and coding in Python), you still need to do 
#   this status notebook in R.  Describe what you did here and put any products 
#   that you created in github. If you are writing online documents (e.g. overleaf 
#   or google docs), you can include links to the documents in this notebook 
#   instead of actual text.

```

### Discussion of results

_Provide in natural language a clear discussion of your observations._


## Analysis: Question 2 (Provide short name)

### Question being asked 

_Provide in natural language a statement of what question you're trying to answer_

### Data Preparation

_Provide in natural language a description of the data you are using for this analysis_

_Include a step-by-step description of how you prepare your data for analysis_

_If you're re-using dataframes prepared in another section, simply re-state what data you're using_

```{r, result02_data}
# Include all data processing code (if necessary), clearly commented

```

### Analysis: Methods and Results  

_Describe in natural language a statement of the analysis you're trying to do_

_Provide clearly commented analysis code; include code for tables and figures!_

```{r, result02_analysis}
# Include all analysis code, clearly commented
# If not possible, screen shots are acceptable. 
# If your contributions included things that are not done in an R-notebook, 
#   (e.g. researching, writing, and coding in Python), you still need to do 
#   this status notebook in R.  Describe what you did here and put any products 
#   that you created in github (documents, jupytor notebooks, etc). If you are writing online documents (e.g. overleaf 
#   or google docs), you can include links to the documents in this notebook 
#   instead of actual text.

```

### Discussion of results

_Provide in natural language a clear discussion of your observations._




## Summary and next steps

_Provide in natural language a clear summary and your proposed next steps._


# When you're done: SAVE, COMMIT and PUSH YOUR CHANGES!

When you are satisfied with your edits and your notebook knits successfully, remember to push your changes to the repo using the following steps:

* `git branch`
   * To double-check that you are in your working branch
* `git add <your changed files>`
* `git commit -m "Some useful comments"`
* `git push origin <your branch name>`
*. Do a pull request
* Submit pdf to gradescope

# APPENDIX: Accessing RStudio Server on the IDEA Cluster

The IDEA Cluster provides seven compute nodes (4x 48 cores, 3x 80 cores, 1x storage server)

* The Cluster requires RCS credentials, enabled via registration in class
    * email John Erickson for problems `erickj4@rpi.edu`
* RStudio, Jupyter, MATLAB, GPUs (on two nodes); lots of storage and computes
* Access via RPI physical network or VPN only

# More info about Rstudio on our Cluster  

## RStudio GUI Access:

* Use: 
   * http://lp01.idea.rpi.edu/rstudio-ose/
   * http://lp01.idea.rpi.edu/rstudio-ose-3/ 
   * http://lp01.idea.rpi.edu/rstudio-ose-6/ 
   * http://lp01.idea.rpi.edu/rstudio-ose-7/ 
* Linux terminal accessible from within RStudio "Terminal" or via ssh (below)