Data transformation (computing)

In computing, data transformation is the process of converting data from one format or structure into another format or structure. It is a fundamental aspect of most data integration and data management tasks such as data wrangling, data warehousing, data integration and application integration. Data transformation can be simple or complex based on the required changes to the data between the source (initial) data and the target (final) data. Data transformation is typically performed via a mixture of manual and automated steps. Tools and technologies used for data transformation can vary widely based on the format, structure, complexity, and volume of the data being transformed. A master data recast is another form of data transformation where the entire database of data values is transformed or recast without extracting the data from the database. All data in a well designed database is directly or indirectly related to a limited set of master database tables by a network of for

Optimus Prime: Accelerating Data Transformation in Servers

Mario Paulo Drumond Lages De Oliveira, Babak Falsafi, Siddharth Gupta, Hussein Kassir, Christoph Koch, Arash Pourhabibi Zarandi, Mark Johnathon Sutherland, Zilu Tian

Modern online services are shifting away from monolithic applications to loosely-coupled microservices because of their improved scalability, reliability, programmability and development velocity. Microservices communicating over the datacenter network require data transformation (DT) to convert messages back and forth between their internal formats. This work identifies DT as a bottleneck due to reductions in latency of the surrounding system components, namely application runtimes, protocol stacks, and network hardware. We therefore propose Optimus Prime (OP), a programmable DT accelerator that uses a novel abstraction, an in-memory schema, to represent DT operations. The schema is compatible with today's DT frameworks and enables any compliant accelerator to perform the transformations comprising a request in parallel. Our evaluation shows that OP's DT throughput matches the line rate of today's NICs and has 60x higher throughput compared to software, at a tiny fraction of the CPU's silicon area and power. We also evaluate a set of microservices running on Thrift, and show up to 30% reduction in service latency.

ACM2020

SANDS: Structured Algebraic Net Development System

SANDS is a development tool based on structured algebraic nets. The environment is composed of the following features: - Algebraic nets allowing the definition and use of structured tokens by means of algebraic specifications. Apart from reducing the size of a model these nets are accurate in describing data transformations during the evolution of the system being modelled; - Structured nets necessary for large system developments. The latter two components form together the language used and understood by the environment. The development environment provides the user with an application simulation tool. This component provides immediate views of any modifications brought to the system being modelled. The environment also permits modifications to the automatically generated code through manual insertions or deletions of C or Prolog instructions. Model semantics possess a well formalized underlying theory. Connections with other theories are numerous and offer a large range of possibilities for describing and proving properties about systems. We can mention amongst other rewriting systems, bisimulation (behaviour comparison between two system models), refinement of objects, algebraic specifications and proof of temporal logic properties used as a more abstract specification language. The user builds and structures his model as a set of algebraic nets called 'objects' representing either instances or classes depending on the context of the modelling. These objects are related to each other by synchronization links which determine and limit the way the objects can evolve concurrently. Algebraic specifications and objects can be used in three different ways: they can be generic, 'ground' or instantiated. The underlying model, namely CO-OPN (Concurrent Object-Oriented Petri Nets), is described in more depth in: 'CO-OPN: A Concurrent Object- Oriented Petri Nets' International Conference on Application and Theory of Petri Nets, Aarhus June 1991. SANDS has two main components: the CO-OPN graphical editor and the simulation tool. Both parts are independent in the following sense: the user can develop his model inside the graphical editor and call the simulation tool from the editor, on the other hand he can use the CO-OPN specification language to describe his model in a textual way and then use the simulation tool after having verified the validity of the specification with the compiler. Moreover these possibilities can be mixed. The user can also modify the specifications by editing the file generated by the graphical editor (written in the CO-OPN language). The graphical tool can be avoided, if desired, through hand written specifications describing the model; a specification made in the CO-OPN language can also be loaded and then modified in the graphical environment. The specifications obtained can then be simulated once the compilation and linking phases have been successfully achieved. The compiling and linking can be performed through the grap The algebraic specification of data can be defined outside of the graphical editor environment. When this has been done the latter specifications can be imported in the graphical editor where their signatures are graphically represented. This set of algebraic specifications are used as a basis which support object specifications. There is no constraint on the way the user builds his specifications. He can adopt a top-down or a bottom-up methodology. Therefore the user can set the overall objects distribution in order to structure his model and then define the behaviour of each object. On the othe hand he can build his application on a set of objects already defined. However, these approaches can also be mixed.

1993

Translating Data into Images: The Design Process for the Creation of Visualizations

Dario Rodighiero

The design process is a series of endeavors aimed to solve a problem, which is extensively defined as a need, a task, a situation, etc. Visualizations are objects that result from a design process that solves a problem of unreadable data. They are technical mediators that transform data in graphics through a precise authorship that defines their social and political context. One way to understand visualizations is to analyze the process of design: this approach, applied to a case study called Affinity Map, unveils the complexity of an object and reveals the reasons of its creation.

2018

SANDS: Structured Algebraic Net Development System

1993