Api Handler
Introduction
The Api Handler is a subset of the DeFi Data Engine with the goal of retrieving data from external sources in an easy and code-less way. In this section we will cover how to set it up, how to extract the data it produces, and how to write properties files. Please note this page covers installation on Windows 10 so you may need to modify the instructions based on your OS.
System Requirements
To use the Api Handler you need the following:
- WSL
- Docker Desktop (with WSL 2.0 configured successfully)
- Git (Gitbash is highly recommended)
- Java SE17 or later
- Postman (Or any Api requesting application)
Quickstart
Step 1
Start Docker Desktop, such that Docker can be used via the command line.
Step 2
We will now build the Docker Volume that the Api Handler will write the data to. This can be done through the following command:
$ docker volume create defi-dataIf you navigate to Docker Desktop and go to the Volumes tab, it should now look like this:
Step 3
Clone the repo to the desired directory. Note the repo is: https://github.rpi.edu/DataINCITE/IDEA-DeFi-CRAFT.
Step 4
Navigate to the root directory of the cloned repository, such that your directory is .../IDEA-DeFi-CRAFT/. From here, navigate to the Api-Handler folder by doing:
$ cd DeFi-Data-Engine
$ cd Api-HandlerStep 5
Once in the root directory of the Api Handler, we can now build the Docker container. To do this we run the following command
$ docker build -t api-handler .
$ docker run -p 8080:8080 -it -v defi-data:/data api-handlerStep 6
The Docker container should now be running and should look similar to below:
Step 7
To terminate the application, you can go to Docker Desktop -> Containers and then stop the active container.
Accessing Data
To access data go to Docker Desktop. Once there, go to the Volumes tab and then select defi-data assuming you followed all the prior steps correctly. Once selected, go to the Data tab at the top and you will see a list of all .csv files generated. It should look similar to below:
Note that one .csv will be generated per call, having the name of the call used. This file will be overridden should another call be made with the same name.
Properties Files
Properties files are how the Api Handler interprets new REST API calls. These files can be found in the requests directory: .../IDEA-DeFi-CRAFT/DeFi-Data-Engine/Api-Handler/src/main/resources/requests/. The template.properties folder serves as the README.txt for how these files should be formatted however that information is also reflected in this section. We will be using amberdata.io as an example for all of these parameters.
Properties
request.name
[REQUIRED]
This property is the unique naming of the REST API call. This MUST be unique otherwise the application will alert you of an error. It can be named anything so long as it follows standard conventions.
url.base
[REQUIRED]
This is the base url required to make the REST API call. This is typically found on the documentation of the data source. For example for amberdata-aave-protocol, the url can be found on the right hand side of the documentation here, returning the base url https://web3api.io/api/v2/defi/lending/aavev3/protocol:
Example definition:
url.base= https://web3api.io/api/v2/defi/lending/aavev3/protocol
url.base.path
[OPTIONAL]
This parameter is optional and only required when there are url PATH properties that are required for parsing of the url. For example, if the url is https://localhost:8080/api/v1/asset/value where value is dictated by the parameter 'key', then this variable will be set to 'key,.' Note all properties are in sequential order, such that asset must be defined before value. These properties are typically found under the section Path Params. Note that some parameters can be auto-defined in the url.base parameter rather than being required to be specified here. In the example image below, we define the path property protocolId=aavev3 already and then require the user to submit the property walletAddress on call by using the base url https://web3api.io/api/v2/defi/lending/aavev3/wallets/. This example is reflected in the example definition.
Example definition:
url.base.path= walletAddress,.
url.properties
[REQUIRED]
These are all the properties that are specified when calling the data source. These are properties that will be specified as <key>, <value> pairs in the url to generate requests similar to the following https://web3api.io/api/v2/defi/lending/aavev3/protocol?key1=value1&key2=value2. To find these properties, go to the documentation and scroll until you find Query Params:
From here you can see all of the properties listed. Although none of these are required, they can be optionally defined on the GET request. To make one defined, specify in the properties file the key-value pair. To make a parameter required and not pre-defined, set the value as a .. Note that the size or equivalent parameter which defines the number of data points for the call to return is REQUIRED, otherwise the recursive function will not work.
In the example definition, size is a predefined required property and action is a required property to be filled out on call.
Example definition:
url.properties= size,900,\
action,.
url.headers
[REQUIRED]
This property details all headers to be passed on runtime when generating the request. Optional headers can be passed on runtime and are not required to be specified here. Note that headers can be given a default value or can be forced to be specified. The list is in <key>, <value> pairs, with each property having both a key and value specified. Default values can be placed in the <value> property. Key's with no default property that are required on runtime can be filled as .. In the example below, we find two headers in the documentation that are defined in the example definition. We require the key under x-api-key to be passed and predefine accept and x-amberdata-blockchain-id.
Example definition:
url.headers= accept,application/json,\
x-amberdata-blockchain-id,polygon-mainnet,\
x-api-key,.\
data.path
[REQUIRED]
This property defines where in the JSON response the data points exist. This should always point to a JSON array which contains the stored points. To define this property, just put the path of nested keys that point to the data. To find this property, make a sample API call to the desired point and the review the response. From here you can then find location of the data points. In the sample response, we find the data is stored in payload->data which we define in the example definition.
Example definition:
data.path= payload,data
recursion.type
[REQUIRED]
This property determines if the call is recursive, meaning that all data points required cannot be obtained in a single request. As of now, only the type parameterized is integrated however other calls can be integrated should they be required. This can be done by extending the org.stream.external.request.types.RequestFramework class and then putting the newly created class in the package org.stream.external.request.types. Use the RequestParameterized class as a template should you require a new implementation. Listed below are the primary functions of each request recursion type:
- parameterized: Used when there is a specific parameter or response variable used to make the next REST API call. This can include a: URL, Property, or Static variable which will be described more in
recursion.tags.
To define this variable, set it to whatever recursive type is in use. The example definition uses parameterized.
Example definition:
recursion.type= parameterized
recursion.tags
[REQUIRED]
The recursion tags are specific parameters that are referenced within the recursion.type java file to perform specific actions. These are defined in <tag>,<value> pairs. Listed below are all tag specific properties for each recursion type available.
- parameterized
- [required]:
- -l: Required number of returned data points to trigger a recursive call. This should always match the value defined by the
sizeparameter in theurl.propertiesproperty. - -t: Type of recursive call. This can be
url,incremental,static. Url refers to if there is a specific url pointer which points to the next call the system should make returned in the response. Incremental refers to there being a property that continuously increments with each call. Static refers to there being a property explicitly returned in the call that replaces a current property value. All of these settings will be described more in therecursive.locationsection.
- -l: Required number of returned data points to trigger a recursive call. This should always match the value defined by the
- [required]:
Example definition:
recursion.tags= -l,900,\
-t,url
recursion.location
This property sets the recurisve parameters location within the response. Should '-t' in recursion.tags be of type 'incremental', this property should be the property to be incremented in the url. Otherwise if it is of type 'url' or 'static', point to the exact location in the response which will retrieve this property. For example if the next url is contained in:
{
"payload": {
"metadata": {
"next": "https://..."
}
}
}
this property would be set to 'payload,metadata,next'. For properties with no recursive call (with recursion type 'single') this parameter can remain blank.
An example of a -t,url response is as follows with recursion.location= payload,metadata,next.
An example of -t,incremental can be found here where the call uses the page property to move to the next recursive call. This would set this property to recursive.location= page.