Api Handler
Introduction
The Api Handler is a subset of the DeFi Data Engine with the goal of retrieving data from external sources in an easy and code-less way. In this section we will cover how to set it up, how to extract the data it produces, and how to write properties files. Please note this page covers installation on Windows 10 so you may need to modify the instructions based on your OS.
System Requirements
To use the Api Handler you need the following:
- WSL
- Docker Desktop (with WSL 2.0 configured successfully)
- Git (Gitbash is highly recommended)
- Java SE17 or later
- Postman (Or any Api requesting application)
Quickstart
Step 1
Start Docker Desktop, such that Docker can be used via the command line.
Step 2
We will now build the Docker Volume that the Api Handler will write the data to. This can be done through the following command:
$ docker volume create defi-dataIf you navigate to Docker Desktop and go to the Volumes tab, it should now look like this:
Step 3
Clone the repo to the desired directory. Note the repo is: https://github.rpi.edu/DataINCITE/IDEA-DeFi-CRAFT.
Step 4
Navigate to the root directory of the cloned repository, such that your directory is .../IDEA-DeFi-CRAFT/. From here, navigate to the Api-Handler folder by doing:
$ cd DeFi-Data-Engine
$ cd Api-HandlerStep 5
Once in the root directory of the Api Handler, we can now build the Docker container. To do this we run the following command
$ docker build -t api-handler .
$ docker run -p 8080:8080 -it -v defi-data:/data api-handlerStep 6
The Docker container should now be running and should look similar to below:
Step 7
To terminate the application, you can go to Docker Desktop -> Containers and then stop the active container.
Accessing Data
To access data go to Docker Desktop. Once there, go to the Volumes tab and then select defi-data assuming you followed all the prior steps correctly. Once selected, go to the Data tab at the top and you will see a list of all .csv files generated. It should look similar to below:
Note that one .csv will be generated per call, having the name of the call used. This file will be overridden should another call be made with the same name.
Properties Files
Properties files are how the Api Handler interprets new REST API calls. These files can be found in the requests directory: .../IDEA-DeFi-CRAFT/DeFi-Data-Engine/Api-Handler/src/main/resources/requests/. The template.properties folder serves as the README.txt for how these files should be formatted however that information is also reflected in this section. We will be using amberdata.io as an example for all of these parameters.
Properties
request.name
[REQUIRED]
This property is the unique naming of the REST API call. This MUST be unique otherwise the application will alert you of an error. It can be named anything so long as it follows standard conventions.
url.base
[REQUIRED]
This is the base url required to make the REST API call. This is typically found on the documentation of the data source. For example for amberdata-aave-protocol, the url can be found on the right hand side of the documentation here, returning the base url https://web3api.io/api/v2/defi/lending/aavev3/protocol:
Example definition:
url.base= https://web3api.io/api/v2/defi/lending/aavev3/protocol
url.base.path
[OPTIONAL]
This parameter is optional and only required when there are url PATH properties that are required for parsing of the url. For example, if the url is https://localhost:8080/api/v1/asset/value where value is dictated by the parameter 'key', then this variable will be set to 'key,.' Note all properties are in sequential order, such that asset must be defined before value. These properties are typically found under the section Path Params. Note that some parameters can be auto-defined in the url.base parameter rather than being required to be specified here. In the example image below, we define the path property protocolId=aavev3 already and then require the user to submit the property walletAddress on call by using the base url https://web3api.io/api/v2/defi/lending/aavev3/wallets/. This example is reflected in the example definition.
Example definition:
url.base.path= walletAddress,.
url.properties
[REQUIRED]
These are all the properties that are specified when calling the data source. These are properties that will be specified as <key>, <value> pairs in the url to generate requests similar to the following https://web3api.io/api/v2/defi/lending/aavev3/protocol?key1=value1&key2=value2. To find these properties, go to the documentation and scroll until you find Query Params:
From here you can see all of the properties listed. Although none of these are required, they can be optionally defined on the GET request. To make one defined, specify in the properties file the key-value pair. To make a parameter required and not pre-defined, set the value as a .. Note that the size or equivalent parameter which defines the number of data points for the call to return is REQUIRED, otherwise the recursive function will not work.
In the example definition, size is a predefined required property and action is a required property to be filled out on call.
Example definition:
url.properties= size,900,\
action,.
url.headers
[REQUIRED]
This property details all headers to be passed on runtime when generating the request. Optional headers can be passed on runtime and are not required to be specified here. Note that headers can be given a default value or can be forced to be specified. The list is in <key>, <value> pairs, with each property having both a key and value specified. Default values can be placed in the <value> property. Key's with no default property that are required on runtime can be filled as '.'. In the example below, we find two headers in the documentation that are defined in the example definition. We require the key under x-api-key to be passed and predefine accept and x-amberdata-blockchain-id.
Example definition:
url.headers= accept,application/json,\
x-amberdata-blockchain-id,polygon-mainnet,\
x-api-key,.\