Skip to content

Architecture

Jump to bottom
Flynn, Conor edited this page Dec 7, 2022 · 6 revisions
Clone this wiki locally

Introduction

This section covers the general overview of the DeFi-Data-Engine's architecture including but not limited to:

  • General system overview
  • Internal routing architecture explanation
  • Routing architecture current implementation
  • Implemented processes overview
  • Implemented processes subroutines
  • Data transfer system
  • System time series analysis
  • REST API overview
  • REST API connection architecture

General System Overview

The DeFi-Data-Engine (DFDE) is a native Java application designed to support the transferring, processing, and storage of high frequency data streams. To accomplish this, the system has been designed with multiple considerations such as: expansion into other sources, multi-user threading capabilities, and language agnostic understanding.

There are three main component types to the DFDE: the Java Applications, the External (R) Applications, and the External Data Sources. The Java Applications are the two primary applications consisting of the main engine architecture and the REST API. The External Applications are any external programs that utilize the engine to extract data. Finally External Data Sources are implemented sources from which the engine can extract data and parse it into the DFDE ecosystem. Below is a high level overview of the DFDE and all of the primary components that interact with the engine. image

The primary focus of this section will be on the "Engine Processes" component which houses all major internal communication within the engine. This will be further expanded upon in later sections, breaking down the individual processes, how they are implemented, and how they can be expanded upon. Also note that R is not the only supported language, however it is the primary language utilized by the lab and therefore will be the most heavily documented. Please note that any language that supports Socket connections and REST API calls can utilize the DFDE.

Internal Routing Architecture

Routers

Introduction

To mediate and standardize all communication across different processes within the engine, a process similar to a network has been implemented (hence the use of "routers" and "packets"). All routers are connected through a "manager" which stores and groups all routers into a single object which allows for communication between all of them. Each router has a unique id (uuid) which is used to identify it. For example, the Core process has the uuid COR. Each uuid is denoted by a three character string (although not required it is standard practice to save memory) and is denoted by the Tag parameter in packets.

Framework

The abstract Router class is listed in the org.framework.router package and holds the framework for implementing a new process. Below are all core methods and their descriptions used within the class.

Each process has sub-processes as well denoted by four character strings. These sub-processes carry out actions within the engine and are the standard way processes communicate with one another.