Push updated r bridge code

R-Code-Samples/data-engine-r-functions.Rmd

@@ -0,0 +1,134 @@
+---
+title: "DeFi Engine Use Example"
+subtitle: "aave-protocol-dated function"
+author: "Conor Flynn"
+date: "03/27/2023"
+output:
+  pdf_document: default
+  html_document:
+    toc: true
+    number_sections: true
+    df_print: paged
+---
+# Start by loading the proper libraries:
+```{r setup, include=FALSE}
+# Set the default CRAN repository
+local({r <- getOption("repos")
+       r["CRAN"] <- "http://cran.r-project.org" 
+       options(repos=r)
+})
+
+# Set code chunk defaults
+knitr::opts_chunk$set(echo = TRUE)
+
+# Load required packages; install if necessary
+# CAUTION: DO NOT interrupt R as it installs packages!!
+if (!require("ggplot2")) {
+  install.packages("ggplot2")
+  library(ggplot2)
+}
+
+if (!require("httr")) {
+  install.packages("httr")
+  library(httr)
+}
+if (!require("jsonlite")) {
+  install.packages("jsonlite")
+  library(jsonlite)
+}
+
+if (!require("lubridate")) {
+  install.packages("lubridate")
+  library(lubridate)
+}
+if(!require("dplyr")){
+  install.packages("dplyr")
+  library(dplyr)
+}
+if(!require("stringr")){
+  install.packages("stringr")
+  library(stringr)
+}
+if(!require("tidyr")){
+  install.packages("tidyr")
+  library(tidyr)
+}
+if(!require("knitr")){
+  install.packages("knitr")
+  library(knitr)
+}
+
+```
+
+We provide a function to request and parse data from our DeFi data engine living on the IDEA Cluster. This initializes a data stream from the Amber Data API, opens a socket, requests data, listens on the socket, and then parses the received data. The finished dataframe is as close as possible to the schema of the cold-storage data we currently use.
+```{R}
+
+request <- function(protocol, properties = "", headers = "", startdate = "", enddate = "") {
+  #Create socket and get destination which tells the engine where to put the data
+  socket <- socketConnection("localhost", 61200, blocking=TRUE)
+  destination <- readLines(socket, 1)
+  
+  formatted_properties = ""
+  if(properties != "")
+    formatted_properties = paste("properties", "&&&", properties, "&&&")
+  
+  formatted_headers = ""
+  if(headers != "")
+    formatted_headers = paste("headers", "&&&", headers, "&&&")
+  
+  formatted_startdate = ""
+  formatted_enddate = ""
+  if(startdate != "" && enddate != "") {
+    formatted_startdate = paste("start_date", "&&&", startdate, "&&&")
+    formatted_enddate = paste("end_date", "&&&", enddate, "&&&")
+  }
+  
+  #Build the request delimited by &&&
+  #Similar to a GET request in the way we handle parameters
+  request.raw <- paste(
+  				"SRC", "&&&", "RQST", "&&&",
+  				"type", "&&&", protocol, "&&&",
+  				formatted_properties,
+  				formatted_headers,
+  				formatted_startdate,
+  				formatted_enddate,
+  				"destination", "&&&", destination, "&&&",
+  				"\n", sep="")
+  
+  # remove all spaces from request
+  request.data <- str_replace_all(request.raw, " ", "")
+  
+  #Write this request back to the socket to tell engine what we want
+  writeLines(request.data, socket)
+  
+  # define data frame
+  df = data.frame()
+  temp.df = data.frame()
+  
+  #Now the engine will begin feeding us data
+  #We grab the first to initialize the data var and then we continue listening\
+  data <- readLines(socket, 1)
+  counter <- 0
+  while (TRUE) {
+    temp <- readLines(socket, 1)
+    counter <- counter + 1
+    if(counter %% 1000 == 0)
+      print(paste("Processed", counter, "lines"))
+    if (grepl("<<<end>>>", temp, fixed=TRUE)) {break}
+    temp.df <- as.data.frame(fromJSON(temp))
+    df <- rbind.fill(df, temp.df)
+  }
+  
+  return(df)
+}
+
+```
+
+With this function, we can now get our data.
+```{R}
+
+#We make a sample call to the function which will return all transactions 2022 August 1st to August 3rd
+df <- request("amberdata-aave-protocol", "", "x-api-key,UAK7ed69235426c360be22bfc2bde1809b6", "2022-08-02", "2022-08-03")
+kable(head(df), "simple")
+
+```