Latency is a big issue for trading systems, but monitoring and managing it is not easy.

Considering today's competitive and regulatory pressures, trading firms need a network and an infrastructure that is always performing at a required latency level. Who can afford to spend minutes, hours, days and weeks figuring out what is wrong when those levels are not being reached? In the past, the technology to monitor, measure and optimise latency simply was not available, but those days are over. In fact, financial institutions are almost spoiled for choice. But that is not to say there are no challenges left in the arms race against latency.

Monitoring, measuring and optimising latency requires an understanding of time and synchronisation. Endace, a New Zealand-based solutions provider, uses Data Acquisition and Generation (DAG) cards to monitor network traffic without interrupting it and then time-stamp it. The cards are synchronised with Coordinated Universal Time (UTC), which is referenced to a collection of about 300 atomic clocks worldwide that produce an average time. Endace feeds GPS time receivers into the DAG cards to synchronise clocks worldwide. That enables it to measure the latency and time flight for a packet in the network between any computer systems.

The challenge lies in getting a GPS signal at every point the bank wants to measure. "If you take your TomTom into the basement of a financial trading house's building, you're not going to get much of a GPS signal," Steve Gleave, vice president of global marketing at Endace, points out.

These problems are solvable using a bit of ingenuity. For instance, Endace has an installation in a computing cluster where the timing of packets is being measured at 48 separate points and at other points outside. Several time feeds go into that. Endace installed a GPS receiver on the roof, and the timing is distributed across the LAN using category 5 cabling. It can now terminate a GPS signal 1, distribute that across a LAN and time all of the probes in a given building from that same universal time source.

Endace recently teamed up with Reuters and Trading Metrics to develop the Reuters Latency Monitor for RMDS. RLM measures the latency of a message leaving the exchange through to the Reuters Integrated Data network and arriving at a customer's site. Then it measures latency as it goes through the feed handler on the site, the market data hub, the P2PS (Peer-to-Peer Simplified), and then to the algorithmic trader.

In addition, Endace is helping customers reduce latency using InfiniBand, an interconnect standard with links that go up to 40 gigabits a second. InfiniBand was designed to get information up the physical infrastructure and into the processing applications faster and cheaper than Ethernet. However, Ethernet has one advantage over InfiniBand: it can be passively monitored in the optical interface by taking a little bit of light and splitting it off. "With InfiniBand, you don't have that luxury because it's an electrical interface, and so you can't just do what's called a light budget switch the way you can with optics," Gleave points out.

Endace built a high-density set of taps that can be put into a computing cluster of 24 or more machines and the trading firm can select which links they want to monitor. The taps are fail-safe so if the power equipment fails, the customer does not lose data. Endace also built a new DAG card for InfiniBand.

New York-based CodeStreet designed its Market Data Works product specifically to make it easier for developers, testers and operators to monitor, manage and optimise market data latency. Market Data Works simulates an entire market data platform so developers can change the content update rates on the platform. They can build market data scenarios and then apply them repeatedly to optimise the system or do automated regression testing. In addition, they can download wizard tools from CodeStreet's web site and a few minutes later be replaying, transforming and analysing data coming from exchanges and across the infrastructure.

Firms use Market Data Works to stress test the infrastructure and monitor the latency of a stressed infrastructure. According to Howard Pein, chief excutive of CodeStreet, until now setting up these kinds of tests was a complex task, whereas it only takes about 15 seconds to set up a test in Market Data Works. Moreover, operators can set up alerts so they get an email when latency exceeds a given threshold. "The operator is armed at all times with information," says Pein. "He never runs into a situation where the phone rings and it's the trading desk yelling at him saying we're not getting Reuters data or Reuters data is slow. He now knows this proactively."

Chicago-based 29West's products provide measurements on one-way latency and determine the output at different points through the messaging layer. Its Latency Busters Messaging product is designed so that message routing is done primarily by the network infrastructure, not by higher-level message routing software. The company's Ultra Messaging for the Enterprise offers guaranteed message delivery (persistence), so if a machine fails, the customer does not lose data.

According to Mark Mahowald, chief excutive and founder of 29West, the solution is unique because it is highly configurable. "When we started out, I assumed the people that would be willing to take a chance on a new messaging product would be people that would be considering writing it themselves," he says. "So we approached it with the idea: let's give people a level of control and tune-ability they would have if they built it themselves."

Users can make many adjustments such as holding messages until they get to a certain packet size. Mahowald stresses, however, that firms need to make tradeoffs to achieve an optimal latency. But once they decide how they want to handle certain conditions, they have more control. Another bonus: 29West gives customers the source code so they can run tests to measure the performance in their environment and compare different products.

To compete in today's trading environment, firms are striving to push latency to ever lower levels. They understand that being able to measure latency is not enough. They need to be able to configure their infrastructure in order to optimise it. It is imperative to have a network infrastructure that is always up for the task and rarely falls out of compliance. Fortunately, advancements in technology are turning that vision into reality. BT

Case study: Credit Suise

Having been through many mergers and acquisitions, Credit Suisse had a very complex, costly environment comprising many point solutions and aging technologies. The functionality available to the business varied significantly between regions. The US tended to have the best technology while operations in the rest of the world were like poor cousins.

To address this problem, Credit Suisse decided the best course of action was to outsource its network. The bank set out looking for a service provider that could deliver a transformation program globally and help it obtain competitive advantage by leveraging trading systems, market data access and high-speed frequency transactions. Most important, it wanted to partner with a service provider that was prepared to get involved in the business and work within the organisation.

After going through a lengthy RFP process, Credit Suisse decided to outsource its network to BT in a contract worth $1.4 billion. "They chose BT because they felt we had all of those qualities," says Correy Voo, Global CTO, BT Atlas, Credit Suisse Network Outsource. "They also recognised intrinsic value from organisations and capabilities within our portfolio such as Radianz Trading Systems, the ability to manage mobile services and those kinds of things."

After holding a multi-day, intensive workshop, BT selected some technologies and solutions that would improve latency in Credit Suisse's network. It selected CorvilNet from Dublin-based Corvil because the solution offers microsecond granularity in its ability to capture and analyse data particularly around how packets and network applications behave over a given infrastructure. The analysis and correlation capabilities make it easy to extrapolate the effects of potential improvements in the network. In addition, CorvilNet can identify hot spots or bottlenecks in the network that could affect an application's performance end-to-end.

Teaming up with Corvil enabled BT to identify points in the infrastructure that could be altered, changed, improved or swapped out. This could be a device, architecture in the network, or circuit connectivity.

Overall, Credit Suisse reduced latency between three and five milliseconds end-to-end in two high-frequency trading applications and a fixed income application that draws market data from the networks. "The bank was able to reach its destinations - the exchanges and liquidity pools - quicker than they were able to do before," says Voo. "We're continuing to use that same technology now across the globe to do the same type of functionality."

Traders are getting information quicker, but like other banks, Credit Suisse does not reveal how much additional revenue they earn by improving latency. However, the bank is continuing to work with BT and Corvil to optimise other elements on the infrastructure to stay ahead of the competition.

-Sherree DeCovny