Latency optimizations on the system level

Identifying and optimizing hot spots in the code may boost the performance of your system, but the improvement is unlikely to be drastic unless the code was poorly written initially, a condition known as premature pessimization. Conversely, a single system-level optimization, if successful, can remove unnecessary network calls and file access operations from the hot path, sometimes both improving latency by an order of magnitude and lowering the load on the main system.

Shortcutting#

The most obvious way to speed up answering a request or reacting to an event is to skip some of the involved system components.

In an early response one of the first components to receive the input may immediately generate an answer or block the input from reaching the rest of the system:

• A Firewall or Rate Limiter blocks disallowed requests. It may send a pre-generated response or ignore the input.
• A Response Cache may answer a client request without consulting the underlying business logic if it has recently seen an identical request and remembered the response.
• The first component of a Pipeline may immediately notify the client that its request has been accepted and then push the request down the pipeline for actual processing.

It’s hard to imagine anything that responds faster than such systems.

It is also possible to bypass the main system, either completely or for a rapid provisional response:

• A hierarchical control system often relies on its low-level nodes to react rapidly to incoming events while its high-level business logic analyzes the situation and chooses the best long-term strategy.
• Various kinds of Hexagonal Architecture and Microkernel tend to bypass their core components as a performance optimization.
  • This is why there is the communication path between the view and controller in Model-View-Controller.
  • Low latency systems often rely on DPDK which maps network packets directly into the application’s memory.

Finally, one of the system components may be omitted from interactions in some scenarios or once it has fulfilled its role:

• A Half-Proxy connects the client to a matching server and then steps out of the way.
• An open layer (or open orchestrator) may be used in processing some requests but omitted for others. Likewise, Layered Orchestrator relies on bypassing the complex orchestration logic for the most common use cases.
• Telephony servers and Middleware exist in general to help their clients find each other and establish a direct connection.

Co-locating#

Each network hop causes delay, therefore we should co-locate as many components as possible:

• The most common example is the frontend tier which runs on a user device and thus provides for seamless user interaction.
• An Ambassador Proxy is co-located with a client application but acts on the system’s behalf. An example is a Sharding Proxy which lets the client software connect directly to the shard which contains the client’s data.
• A Sidecar co-locates generic code with a system’s service. Sidecars are used in Service Mesh and Space-Based Architecture to reduce latency between the business logic and the distributed communication infrastructure.
• Distributed Microkernels invest in co-location optimizations:
  • Actor frameworks tend to move the actors between hosts to ensure that intensely interacting actors run on the same host.
  • In AUTOSAR it is preferable to run an application and the services which it uses on the same chip.
• Monolith is the pattern in which everything is co-located, therefore it can have very low latency if properly implemented.

Preloading and preprocessing#

If you have all the data you need in the right place and right format, you can use it right away:

• Actors and Space-Based Architecture keep the data in operating memory next to the code which uses it. They are blazingly fast.
• Most implementations of Response Cache keep the data in memory as well.
• Various kinds of Polyglot Persistence stream changes to a derived database specialized in queries or analytics:
  • CQRS often relies on a derived OLAP database to answer queries. A Reporting Database plays the same role for analytical reports.
  • A Memory Image or Materialized View collects state changes from event sourcing into an in-memory state snapshot.
  • A CQRS View Database or Query Service collects events from the system’s services to aggregate the data which its clients find useful.
  • A Front Controller keeps track of the status of requests which are being processed by a Pipeline.
  • An External Search Index supports efficient search in a large collection of documents or data records.
• An Interpreter may compile user scripts which it runs for faster execution.

All at once#

If your system does not allow for any one of the individual optimizations listed above, there is still a chance to apply them all at once, though not without a certain planning and effort. In this approach one component injects a part of its business logic, together with preprocessed data to base decisions on, into another component which handles incoming events. This allows the second component to process certain kinds of incoming messages without any external help or network communication:

• In high frequency trading it is a common practice to encode simple precomputed trading rules into an FPGA chip or a programmable network card. That removes the delay caused by the host’s operating system and results in nearly instantaneous trading decision and response when the incoming price notification matches one of the preprogrammed trading rules.
• A service which is originally dependent on other services may instead publish an interface for the other services’ teams to write Ambassador Plugins which it would call as strategies during request processing. As plugins tend to be co-located with the host process, the interservice calls disappear, making request processing much faster and more stable. However, in many cases there will be a need for the service on which behalf the plugin acts to feed an event stream to its plugin to supply the plugin’s decision-making logic with data, further complicating the practical use of Ambassador Plugins.

Non-blocking#

When requests access shared resources, they may block or slow each other down. Therefore the fastest systems either avoid blocking anything during request processing, or avoid sharing resources between their clients:

• A component that processes every event with a non-blocking callback is known as Proactor. It grants the component real-time properties at the cost of hard-to-read code.
• Shards dedicate database instances to subsets of the system’s clients, thus lowering the chance for one client’s requests to negatively affect others.
• Actor systems use both of the foregoing latency optimizations: they dedicate one non-blocking actor to each client.
• A Space-Based Architecture (shown above) creates multiple in-memory replicas of its data, each serving client requests. The replicas synchronize their states in background, but there is still a chance of a write conflict when multiple clients simultaneously edit the same data.

Miscellaneous optimizations#

There are also several optimizations which don’t fit into wider categories:

• An Orchestrator may split a request into subrequests to several services or shards for parallel processing, as shown on the diagram above.
• Request hedging is sending an incoming request to multiple copies of the system in parallel and accepting the first response available. It greatly improves tail latency and system stability.
• Fine-tuning the persistence layer often drastically improves latency. The techniques range from using a pair of SQL + NoSQL databases to moving the historical data from the main database to a backup storage.
• Shared memory provides for the fastest interprocess communication.

Summary#

The common system-level latency optimization techniques for request processing include bypassing some of the system’s components, co-locating components with each other or with the client, and preloading or preprocessing the data needed to answer requests. Though each of these optimizations is powerful on its own, they can be used together to instantly answer a subset of client requests. Furthermore, there are specialized optimizations such as request hedging or non-blocking which improve latency of a single subsystem.