Architecture

Introduction

This section covers the general overview of the DeFi-Data-Engine's architecture including but not limited to:

• General system overview
• Internal routing architecture explanation
• Routing architecture current implementation
• Implemented processes overview
• Implemented processes subroutines
• Data transfer system
• System time series analysis
• REST API overview
• REST API connection architecture

General System Overview

The DeFi-Data-Engine (DFDE) is a native Java application designed to support the transferring, processing, and storage of high frequency data streams. To accomplish this, the system has been designed with multiple considerations such as: expansion into other sources, multi-user threading capabilities, and language agnostic understanding.

There are three main component types to the DFDE: the Java Applications, the External (R) Applications, and the External Data Sources. The Java Applications are the two primary applications consisting of the main engine architecture and the REST API. The External Applications are any external programs that utilize the engine to extract data. Finally External Data Sources are implemented sources from which the engine can extract data and parse it into the DFDE ecosystem. Below is a high level overview of the DFDE and all of the primary components that interact with the engine.

The primary focus of this section will be on the "Engine Processes" component which houses all major internal communication within the engine. This will be further expanded upon in later sections, breaking down the individual processes, how they are implemented, and how they can be expanded upon. Also note that R is not the only supported language, however it is the primary language utilized by the lab and therefore will be the most heavily documented. Please note that any language that supports Socket connections and REST API calls can utilize the DFDE.

Internal Routing Architecture

Routers

Introduction

To mediate and standardize all communication across different processes within the engine, a process similar to a network has been implemented (hence the use of "routers" and "packets"). All routers are connected through a "manager" which stores and groups all routers into a single object which allows for communication between all of them. Each router has a unique id (uuid) and a tag which are used to identify it. For example, the Core process has the uuid core and the tag 'COR'. Each tag is denoted by a three character string (although not required it is standard practice to save memory).

Framework

The abstract Router class is listed in the org.framework.router package and holds the framework for implementing a new process. Below are all core methods and their descriptions used within the class. Please note all methods not listed with abstract are implemented and just not shown in this example.

// base constructor
public Router(String uuid, String tag) {}

// constructor for auto-connecting the Router to an existing Manager
public Router(String uuid, String tag, Manager manager) {}

// returns uuid of the Router
public final String getUUID() {}

// returns tag of the Router
public final String getTag() {}

// returns the Manager the Router is connected to
public final Manager getManager() {}

// connects the Router to a new Manager
public final void setManager(Manager manager) {}

// connects list of Routers to this Router
public final void connect(Router... routers) {}

// checks to see if a Router with the given tag is connected to this Router
public final boolean isConnected(String tag) {}

// checks to see if a Router is connected to this Router
public final boolean isConnected(Router router) {}

// returns all connected Routers to this Router
public final Collection<String> connectedTags() {}

// sends a Packet with the given data to a connected Router
// returns a Response object
public final Response send(String tag, String sub_tag, HashMap<String, String> data) {}

// sends a Packet with the given data to a connected Router
// returns a Response object
// note all data is in the format ["key_1", "value_1", "key_2", "value_2",..., "key_n", "value_n"]
public final Response send(String tag, String sub_tag, String... data) {}

// function that receives an incoming Packet
public final Response receive(Packet packet) {}

// function for adding a sub-process to the Router
public final void addProcess(String subtag, Method method) {}

// function used for handling incoming processes
// users do NOT need to interact with this function
private final Response process(Packet packet) {}

// function which auto adds any function labeled "process----"
// to the Router which can then be accessed
// for example the function "processSUBS" will be added to the function
// under the sub_tag "SUBS" that can be called from other Routers
private final void defineProcesses() throws NoSuchMethodException, SecurityException {}

Examples

Definitions

We define the two Router objects.

public class Router1 extends Router {
  public Router1() {
    super("router1", "RT1");
  }
}

public class Router2 extends Router {
  public Router2() {
    super("router2", "RT2");
  }

  public Response processEXPL(Packet packet) {
    System.out.println(packet.getData("sample"));
    return ResponseFactory.response200();
  }
}

Connecting the Routers

Router1 rt1 = new Router1();
Router2 rt2 = new Router2();
  
rt1.connect(rt2);
rt1.isConnected("RT2"); // true
}

Sending a Packet

// this call will print the data stored under "sample"
// and then return a 200 response
rt1.send("RT2", "EXPL", "sample", "this-is-printed").code(); // 200

Packets

Introduction

Each process has sub-processes as well denoted by four character strings. These sub-processes carry out actions within the engine and are the standard way processes communicate with one another.