Skip to content

Api Handler

Jump to bottom
Flynn, Conor edited this page Feb 21, 2023 · 8 revisions
Clone this wiki locally

Introduction

The Api Handler is a subset of the DeFi Data Engine with the goal of retrieving data from external sources in an easy and code-less way. In this section we will cover how to set it up, how to extract the data it produces, and how to write properties files. Please note this page covers installation on Windows 10 so you may need to modify the instructions based on your OS.

System Requirements

To use the Api Handler you need the following:

Quickstart

Step 1

Start Docker Desktop, such that Docker can be used via the command line.

Step 2

We will now build the Docker Volume that the Api Handler will write the data to. This can be done through the following command:

$ docker volume create defi-data

If you navigate to Docker Desktop and go to the Volumes tab, it should now look like this: image

Step 3

Clone the repo to the desired directory. Note the repo is: https://github.rpi.edu/DataINCITE/IDEA-DeFi-CRAFT.

Step 4

Navigate to the root directory of the cloned repository, such that your directory is .../IDEA-DeFi-CRAFT/. From here, navigate to the Api-Handler folder by doing:

$ cd DeFi-Data-Engine
$ cd Api-Handler

Step 5

Once in the root directory of the Api Handler, we can now build the Docker container. To do this we run the following command

$ docker build -t api-handler .
$ docker run -p 8080:8080 -it -v defi-data:/data api-handler

Step 6

The Docker container should now be running and should look similar to below: image

Step 7

To terminate the application, you can go to Docker Desktop -> Containers and then stop the active container.

Accessing Data

To access data go to Docker Desktop. Once there, go to the Volumes tab and then select defi-data assuming you followed all the prior steps correctly. Once selected, go to the Data tab at the top and you will see a list of all .csv files generated. It should look similar to below: image

Note that one .csv will be generated per call, having the name of the call used. This file will be overridden should another call be made with the same name.

Properties Files

Properties files are how the Api Handler interprets new REST API calls. These files can be found in the requests directory: .../IDEA-DeFi-CRAFT/DeFi-Data-Engine/Api-Handler/src/main/resources/requests/. The template.properties folder serves as the README.txt for how these files should be formatted however that information is also reflected in this section. We will be using amberdata.io as an example for all of these parameters.

Properties

request.name

[REQUIRED]

This property is the unique naming of the REST API call. This MUST be unique otherwise the application will alert you of an error. It can be named anything so long as it follows standard conventions.

url.base

[REQUIRED]

This is the base url required to make the REST API call. This is typically found on the documentation of the data source. For example for amberdata-aave-protocol, the url can be found on the right hand side of the documentation here, returning the base url https://web3api.io/api/v2/defi/lending/aavev3/protocol:

image

Example definition:

url.base= https://web3api.io/api/v2/defi/lending/aavev3/protocol

url.base.path

[OPTIONAL]

This parameter is optional and only required when there are url PATH properties that are required for parsing of the url. For example, if the url is https://localhost:8080/api/v1/asset/value where value is dictated by the parameter 'key', then this variable will be set to 'key,.' Note all properties are in sequential order, such that asset must be defined before value. These properties are typically found under the section Path Params. Note that some parameters can be auto-defined in the url.base parameter rather than being required to be specified here. In the example image below, we define the path property protocolId=aavev3 already and then require the user to submit the property walletAddress on call by using the base url https://web3api.io/api/v2/defi/lending/aavev3/wallets/. This example is reflected in the example definition.

image

Example definition:

url.base.path= walletAddress,.

url.properties

[REQUIRED]

These are all the properties that are specified when calling the data source. These are properties that will be specified as <key>, <value> pairs in the url to generate requests similar to the following https://web3api.io/api/v2/defi/lending/aavev3/protocol?key1=value1&key2=value2. To find these properties, go to the documentation and scroll until you find Query Params:

image

From here you can see all of the properties listed. Although none of these are required, they can be optionally defined on the GET request. To make one defined, specify in the properties file the key-value pair. To make a parameter required and not pre-defined, set the value as a .. Note that the size or equivalent parameter which defines the number of data points for the call to return is REQUIRED, otherwise the recursive function will not work.

In the example definition, size is a predefined required property and action is a required property to be filled out on call.

Example definition:

url.properties= size,900,\
                action,.

url.headers

[REQUIRED]

This property details all headers to be passed on runtime when generating the request. Optional headers can be passed on runtime and are not required to be specified here. Note that headers can be given a default value or can be forced to be specified. The list is in <key>, <value> pairs, with each property having both a key and value specified. Default values can be placed in the <value> property. Key's with no default property that are required on runtime can be filled as .. In the example below, we find two headers in the documentation that are defined in the example definition. We require the key under x-api-key to be passed and predefine accept and x-amberdata-blockchain-id.

image

Example definition:

url.headers= accept,application/json,\
             x-amberdata-blockchain-id,polygon-mainnet,\
             x-api-key,.\

data.path

[REQUIRED]

This property defines where in the JSON response the data points exist. This should always point to a JSON array which contains the stored points. To define this property, just put the path of nested keys that point to the data. To find this property, make a sample API call to the desired point and the review the response. From here you can then find location of the data points. In the sample response, we find the data is stored in payload->data which we define in the example definition.

image

Example definition:

data.path= payload,data

recursion.type

[REQUIRED]

This property determines if the call is recursive, meaning that all data points required cannot be obtained in a single request. As of now, only the type parameterized is integrated however other calls can be integrated should they be required. This can be done by extending the org.stream.external.request.types.RequestFramework class and then putting the newly created class in the package org.stream.external.request.types. Use the RequestParameterized class as a template should you require a new implementation. Listed below are the primary functions of each request recursion type:

  • parameterized: Used when there is a specific parameter or response variable used to make the next REST API call. This can include a: URL, Property, or Static variable which will be described more in recursion.tags.

To define this variable, set it to whatever recursive type is in use. The example definition uses parameterized.

Example definition:

recursion.type= parameterized

recursion.tags

[REQUIRED]

The recursion tags are specific parameters that are referenced within the recursion.type java file to perform specific actions. These are defined in <tag>,<value> pairs. Listed below are all tag specific properties for each recursion type available.

  • parameterized
    • [required]:
      • -l: Required number of returned data points to trigger a recursive call. This should always match the value defined by the size parameter in the url.properties property.
      • -t: Type of recursive call. This can be url, incremental, static. Url refers to if there is a specific url pointer which points to the next call the system should make returned in the response. Incremental refers to there being a property that continuously increments with each call. Static refers to there being a property explicitly returned in the call that replaces a current property value. All of these settings will be described more in the recursive.location section.

Example definition:

recursion.tags= -l,900,\
                -t,url

recursion.location

This property sets the recurisve parameters location within the response. Should '-t' in recursion.tags be of type 'incremental', this property should be the property to be incremented in the url. Otherwise if it is of type 'url' or 'static', point to the exact location in the response which will retrieve this property. For example if the next url is contained in: { "payload": { "metadata": { "next": "https://..." } } } this property would be set to 'payload,metadata,next'. For properties with no recursive call (with recursion type 'single') this parameter can remain blank.

An example of a -t,url response is as follows with recursion.location= payload,metadata,next.

image

An example of -t,incremental can be found here where the call uses the page property to move to the next recursive call. This would set this property to recursive.location= page.

Example definition:

recursive.location= payload,metadata,next

date.valid

[REQUIRED]

This property determines if the call can be dated. This means that you can specify a date range and have the call return the data between those dates. If it can be, there are typically properties that determine these dates on the documentation. Below is an example call that can be dated based on the properties shown (startDate and endDate). This property should be set to a boolean (either true or false) to determine whether this call can be dated.

image

Example definition:

date.valid= true

date.location

[OPTIONAL]

  • Only required if date.valid=true

This property specifies the location of the date.start and date.end properties. This value can be set to either properties or headers which will include it into the respective category. Note that headers has not been implemented as it is a very rare edge case and can be implemented upon requirement.

Example definition:

date.location= properties

date.start

[OPTIONAL]

  • Only required if date.valid=true

This property specifies the name of the property that will be used to define the start date. In this example, date.start= startDate as shown in the documentation.

image

Example definition:

date.start= startDate

date.end

[OPTIONAL]

  • Only required if date.valid=true

This property specifies the name of the property that will be used to define the end date. In this example, date.end= endDate as shown in the documentation.

image

Example definition:

date.end= endDate

date.format

[OPTIONAL]

  • Only required if date.valid=true

This states the date formatting to be passed in the call. To specify this property use the SimpleDateFormat standard referenced here. In this example, we use the format yyyy-MM-dd (year-month-day).

image

Example definition:

date.format= yyyy-MM-dd