Architecture

Introduction

This section covers the general overview of the DeFi-Data-Engine's architecture including but not limited to:

• General system overview
• Internal routing architecture explanation
• Routing architecture implementation
• Implemented processes overview
• Implemented processes subroutines
• Data transfer system
• System time series analysis
• REST API overview
• REST API connection architecture

General System Overview

The DeFi-Data-Engine (DFDE) is a native Java application designed to support the transferring, processing, and storage of high frequency data streams. To accomplish this, the system has been designed with multiple considerations such as: expansion into other sources, multi-user threading capabilities, and language agnostic understanding.

There are three main component types to the DFDE: the Java Applications, the External (R) Applications, and the External Data Sources. The Java Applications are the two primary applications consisting of the main engine architecture and the REST API. The External Applications are any external programs that utilize the engine to extract data. Finally External Data Sources are implemented sources from which the engine can extract data and parse it into the DFDE ecosystem. Below is a high level overview of the DFDE and all of the primary components that interact with the engine.

The primary focus of this section will be on the "Engine Processes" component which houses all major internal communication within the engine. This will be further expanded upon in later sections, breaking down the individual processes, how they are implemented, and how they can be expanded upon. Also note that R is not the only supported language, however it is the primary language utilized by the lab and therefore will be the most heavily documented. Please note that any language that supports Socket connections and REST API calls can utilize the DFDE.

Internal Routing Architecture

Routers

Overview

To mediate and standardize all communication across different processes within the engine, a process similar to a network has been implemented (hence the use of "routers" and "packets"). All routers are connected through a "manager" which stores and groups all routers into a single object which allows for communication between all of them. Each router has a unique id (uuid) and a tag which are used to identify it. For example, the Core process has the uuid core and the tag 'COR'. Each tag is denoted by a three character string (although not required it is standard practice to save memory).

Each process has sub-processes as well denoted by four character strings. These sub-processes carry out actions within the engine and are the standard way processes communicate with one another. Below we explore their implementation and how they can be created.

Framework

The abstract Router class is listed in the org.framework.router package and holds the framework for implementing a new process. Below are all core methods and their descriptions used within the class. Please note all methods not listed with abstract are implemented and just not shown in this example.

// base constructor
public Router(String uuid, String tag) {}

// constructor for auto-connecting the Router to an existing Manager
public Router(String uuid, String tag, Manager manager) {}

// returns uuid of the Router
public final String getUUID() {}

// returns tag of the Router
public final String getTag() {}

// returns the Manager the Router is connected to
public final Manager getManager() {}

// connects the Router to a new Manager
public final void setManager(Manager manager) {}

// connects list of Routers to this Router
public final void connect(Router... routers) {}

// checks to see if a Router with the given tag is connected to this Router
public final boolean isConnected(String tag) {}

// checks to see if a Router is connected to this Router
public final boolean isConnected(Router router) {}

// returns all connected Routers to this Router
public final Collection<String> connectedTags() {}

// sends a Packet with the given data to a connected Router
// returns a Response object
public final Response send(String tag, String sub_tag, HashMap<String, String> data) {}

// sends a Packet with the given data to a connected Router
// returns a Response object
// note all data is in the format ["key_1", "value_1", "key_2", "value_2",..., "key_n", "value_n"]
public final Response send(String tag, String sub_tag, String... data) {}

// function that receives an incoming Packet
public final Response receive(Packet packet) {}

// function for adding a sub-process to the Router
public final void addProcess(String subtag, Method method) {}

// function used for handling incoming processes
// users do NOT need to interact with this function
private final Response process(Packet packet) {}

// function which auto adds any function labeled "process----"
// to the Router which can then be accessed
// for example the function "processSUBS" will be added to the function
// under the sub_tag "SUBS" that can be called from other Routers
private final void defineProcesses() throws NoSuchMethodException, SecurityException {}

Examples

Defining Router Objects

public class Router1 extends Router {
  public Router1() {
    super("router1", "RT1");
  }
}

public class Router2 extends Router {
  public Router2() {
    super("router2", "RT2");
  }

  public Response processEXPL(Packet packet) {
    System.out.println(packet.getData("sample"));
    return ResponseFactory.response200();
  }
}

Connecting the Routers

Router1 rt1 = new Router1();
Router2 rt2 = new Router2();
  
rt1.connect(rt2);
rt1.isConnected("RT2"); // true
}

Sending a Packet

// this call will print the data stored under "sample"
// and then return a 200 response
rt1.send("RT2", "EXPL", "sample", "this-is-printed").code(); // 200

Packets

Overview

Packets are the standard communication source used by the engine. Every Router object is able to easily handle and process a Packet object, allowing for easy communication between processes and expansion of the engine. Packets contain four data components: sender, tag, sub_tag, and data`.

The sender object refers to the Router that sent the Packet. This can be used for verification purposes (say for example an important process can only receive packets from a verified process) to make sure no rouge packets interfere with various communications. The tag is used to identify the destination Router that the packet is sent to. The sub_tag denotes the sub-process the Packet is to be sent to and interact with. Finally the data object stores all data the packet is transmitting to the sub-process.

Packets are created through the Router.send(String tag, String sub_tag, HashMap<String, String> data) and Router.send(String tag, String sub_tag, String... data) functions from which they are automatically sent to the appropriate locations.

Framework

The Packet class is listed in the org.framework.router class and can only be created through the given factory methods packet(Router router, String tag, String sub_tag, HashMap<String, String> data) and packet(Router router, String tag, String sub_tag, String... data). Below are all functions defined within the Packet class.

// private constructor
private Packet(Router router, String tag, String sub_tag, HashMap<String, String> data) {}

// returns sending Router's tag
public final String getSender() {}

// returns destination Router's tag
public final String getTag() {}

// checks if data object contains given key
public final boolean containsKey(String key) {}

// returns HashMap containing all data
public final HashMap<String, String> getData() {}

// returns data point under given key
public final String getData(String key) {}

// validates that all passed keys are contained in the packet
// if not returns a string of the missing key
public final String validate(String... keys) {}

// factory method for creating packet with pre-defined HashMap data object
public static Packet packet(Router router, String tag, String sub_tag, HashMap<String, String> data) {}

// factory method for creating packet from list of strings
public static Packet packet(Router router, String tag, String sub_tag, String... data)

Routing Architecture Implementation

This subsection covers all currently implemented packet communications between all processes. This includes requesters, destinations (tag), sub-processes (sub_tag), passed variables, the return, the possible response codes, and a description. Please note that all processes and their tags will be explained in a later section.

Requesters	Tag	Sub_Tag	Variables	Return	Description	Response Codes
CRL, OUT	SRC	EXSR	source: String	exists: boolean	Determines if external stream source exists.	200, 500
CRL, OUT	SRC	EXST	key: String	exists: boolean	Determines if an external stream with the given tag exists	200, 500
OUT	SRC	INIT	source: String, auth_data: String[]	key: String	Initializes a new stream with the given data and authorizes it.	200, 420, 422, 500
CRL, OUT	SRC	IATH	key: String	is_authorized: boolean	Determines if a stream with the given key is authorized.	200, 421, 500
CRL, OUT	SRC	IATV	key: String	key: String	is_active: boolean	Determines if a stream is currently active.
OUT	SRC	EXEC	key: String	success: boolean	Executes the stream with the given key.	200, 421, 423, 424, 500
CRL, OUT	SRC	KILL	key: String	success: boolean	Kills the stream with the given key.	200, 421, 425, 500
OUT	SRC	SUBS	key: String, subscription: String	n/a	Subscribes to a new type of data embedded within the data stream.	200, 421, 425, 426, 427
OUT	SRC	RQST	key: String, request: String, start_date: String, end_date: String, query: String[]	data: String[]	200, 421, 423, 428, 429, 441, 445, 446, 447, 449