Skip to content

Commit

Permalink
update rmd files
Browse files Browse the repository at this point in the history
  • Loading branch information
Conor Flynn committed Apr 19, 2023
1 parent ac05c03 commit ee83763
Show file tree
Hide file tree
Showing 3 changed files with 246 additions and 238 deletions.
158 changes: 33 additions & 125 deletions R-Code-Samples/DataEnginePrimaryFunctions.Rmd
View file
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
---
title: "DeFi Engine Use Example"
subtitle: "aave-protocol-dated function"
title: "Data Engine Primary Request Function"
subtitle: "request function definition"
author: "Conor Flynn"
date: "03/27/2023"
date: "04/18/2023"
output:
pdf_document: default
html_document:
Expand All @@ -12,50 +12,43 @@ output:
---
# Start by loading the proper libraries:
```{r setup, include=FALSE}
# Set the default CRAN repository
local({r <- getOption("repos")
r["CRAN"] <- "http://cran.r-project.org"
options(repos=r)
})
# Set code chunk defaults
knitr::opts_chunk$set(echo = TRUE)
library(ggplot2)
library(knitr)
library(plyr)
library(dplyr)
library(jsonlite)
library(RColorBrewer)
library(tidyverse)
library(beeswarm)
library(ggbeeswarm)
library(xts)
library(plotly)
library(lubridate)
library(survival)
library(survminer)
library(ranger)
library(ggfortify)
library(factoextra)
library(cluster)
library(fclust)
library(ppclust)
library(e1071)
library(randomNames)
# Load required packages; install if necessary
# CAUTION: DO NOT interrupt R as it installs packages!!
library(stringr)
library(tidyr)
```

We provide a function to request and parse data from our DeFi data engine living on the IDEA Cluster. This initializes a data stream from the Amber Data API, opens a socket, requests data, listens on the socket, and then parses the received data. The finished dataframe is as close as possible to the schema of the cold-storage data we currently use.
## Define request function:

# DESCRIPTION:
# This function serves as the primary request point for making internal requests to the defi data engine. Note that should a request be
# made outside the function, please ensure the necessary format depicted within it is used otherwise an internal error may occur.
# Note that you MUST be connected to the RPI network (such that a connection can be made to defi-de.idea.rpi.edu) otherwise this
# function will not work.

# INPUT:
# - protocol: The name of the protocol to receive data from.
# [optional] - properties: All properties used in the call (note typically REST API URL parameters).
# [optional] - headers: All headers used in the call (note typically REST API URL headers).
# [optional] - startdate: Starting date to retrieve data from. In format 'yyyy-MM-dd' i.e. 2023-04-01
# [optional] - enddate: Ending date to retrieve data from (non-inclusive). In format 'yyyy-MM-dd' i.e. 2023-04-01

# OUTPUT:
# list containing two elements:
# - $response: Contains all response information as listed below:
# - $response: Value denoting status of call TO the engine. Code 200 denotes connection was received by engine properly.
# - $code: Code returned by engine based on internal schema. Full list of codes can be found here
# (https://github.rpi.edu/DataINCITE/IDEA-DeFi-CRAFT/wiki/Response-Codes)
# - $message: Message response accompanying code should the response be irregular.
# - $data: Data returned by call should any be requested.
# - $df: Data frame containing all data parsed from the call.
```{r}
request_deprecated <- function(protocol, properties = "", headers = "", startdate = "", enddate = "") {
request <- function(protocol, properties = "", headers = "", startdate = "", enddate = "") {
suppressWarnings({
#Create socket and get destination which tells the engine where to put the data
socket <- socketConnection("defi-de.idea.rpi.edu", 61200, blocking=TRUE)
# socket <- socketConnection("localhost", 61200, blocking=TRUE)
destination <- readLines(socket, 1)
formatted_properties = ""
Expand Down Expand Up @@ -93,13 +86,12 @@ request_deprecated <- function(protocol, properties = "", headers = "", startdat
# define data frame
df = data.frame()
# temp.df = data.frame()
temp.df = data.frame()
#Now the engine will begin feeding us data
#We grab the first to initialize the data var and then we continue listening\
counter <- 0
response <- ""
json <- "{\"data\":["
while (TRUE) {
temp <- readLines(socket, 1)
Expand Down Expand Up @@ -130,97 +122,13 @@ request_deprecated <- function(protocol, properties = "", headers = "", startdat
print(paste("Processed", counter, "lines for", protocol))
# add data point line to data frame
# temp.df <- as.data.frame(fromJSON(temp))
# df <- rbind.fill(df, temp.df)
json <- paste(json, temp, ",", sep="")
temp.df <- as.data.frame(fromJSON(temp))
df <- rbind.fill(df, temp.df)
}
json <- substr(json, 0, str_length(json) - 1)
json <- paste(json, "]}")
df <- as.data.frame(fromJSON(json))
output <- list("response"=response, "df"=df)
close(socket)
return(output)
})
}
```

With this function, we can now get our data.
```{R}
#We make a sample call to the function which will return all transactions 2022 August 1st to August 3rd
# aave <- request("amberdata-aave-protocol", "", "x-api-key,UAK7ed69235426c360be22bfc2bde1809b6", "2022-08-01", "2022-08-03")
# aave.df <- aave$df
# aave.response <- aave$response
# aave.liquidations <- request("amberdata-aave-protocol", "action,LiquidationCall", "x-api-key,UAK7ed69235426c360be22bfc2bde1809b6", "2022-07-01", "2023-09-01")
# aave.liquidations.df <- aave.liquidations$df
# aave.liquidations.response <- aave.liquidations$response
# sushiswap <- request("amberdata-sushiswap-protocol", "", "x-api-key,UAK7ed69235426c360be22bfc2bde1809b6", "2022-08-01", "2022-08-03")
# sushiswap.df <- sushiswap$df
# sushiswap.response <- sushiswap$response
start_date <- "2022-01-01"
end_date <- "2022-01-10"
graph.borrows.df <- request("graph-aave-borrows", "", "", start_date, end_date)$df
graph.collaterals.df <- request("graph-aave-collaterals", "", "", start_date, end_date)$df
graph.deposits.df <- request("graph-aave-deposits", "", "", start_date, end_date)$df
graph.flashloans.df <- request("graph-aave-flash-loans", "", "", start_date, end_date)$df
graph.liquidations.df <- request("graph-aave-liquidations", "", "", start_date, end_date)$df
graph.pricehist.df <- request("graph-aave-price-history-items", "", "", start_date, end_date)$df
graph.redeems.df <- request("graph-aave-redeems", "", "", start_date, end_date)$df
graph.repays.df <- request("graph-aave-repays", "", "", start_date, end_date)$df
graph.reservehist.df <- request("graph-aave-reserve-params-hist-items", "", "", start_date, end_date)$df
graph.reserves.df <- request("graph-aave-reserves", "", "")$df
graph.swaps.df <- request("graph-aave-swaps", "", "", start_date, end_date)$df
graph.userreserves.df <- request("graph-aave-user-reserves", "", "", start_date, end_date)$df
graph.users.df <- request("graph-aave-users", "", "")$df
graph.users.df <- addAliases(graph.users.df, graph.users.df$id)
```

```{r}
amberdata.addresses.test <- request("amberdata-blockchain-addresses", "hash,0xfaf3af4f551f76af4cde17c3d3708c4f3a69d21e", "x-api-key,UAK7ed69235426c360be22bfc2bde1809b6")
```

```{r}
stablecoins <- request("llama-stablecoins", "", "")$df
```

```{r}
graph.users.sample <- graph.users.df %>%
select(-alias)
aliases = NULL
set.seed(69420)
while(length(aliases[,1]) < length(graph.users.sample$id)){
alias <- randomNames(1000, name.order = "first.last", name.sep = " ", sample.with.replacement = FALSE)
aliases <- aliases %>%
bind_rows(data.frame(alias)) %>%
distinct()
}
aliases <- aliases %>%
head(length(graph.users.sample$id))
userAliases <- bind_cols(graph.users.sample, aliases) %>%
mutate(version = "V2",
deployment = "Mainnet")
graph.users.sample <- userAliases
```
Loading

0 comments on commit ee83763

Please sign in to comment.