Why Big Data & Intelligent Middleware will revolutionize Financial trading…(part 1 of 3)

This article is the first in a 3 part series that talks about the business issues faced by large trading desks in Capital Markets space and the agile architectures that can be put in place using Big Data techniques to help clients gain competitive advantage.

There are very few industries that are as data-centric as the financial services industry. Every interaction that a client or partner system has with a banking institution produces actionable data that has potential business value associated with it.

Even within the large spectrum of client facing domains that make up financial services – capital markets occupies the pride of place in terms of being forward looking as far as adoption of new age technologies goes.

While traditional domains like retail banking, wealth management, consumer banking  have historically had multiple sources and silos of data across the front-, back- and mid-offices – capital markets always have had a data challenge especially around areas like the trade lifecycle.

In 2015, it is no secret that the capital markets have seen better times from a business standpoint, namely because –

1.Falling volumes across a range of asset classes e.g. equities from the days of 2008-2009. The below graphic from the Tabb group perfectly captures this dynamic.Average daily trading volume, tallied by month, was just 5.8 billion shares in May 2015, less than half of the peak of 12.3 billion shares during the financial crisis. (Ref – www.marketwatch.com)

2.Decline in profits across previously lucrative areas like High Frequency Trading as volumes fall. An important point to note here is that profitability of a trading desk is driven by trading volumes that the desk attracts from their various institutional clients.

3. Overall decreased enterprise profitability owing to both #1 and #2

4.Provide the head of capital markets & risk managers with a 360 degree view of the customers that their entire range of desks do business with to not just optimize enterprise profits but to also manage risk

A definition of high frequency trading is in order for  those new to the space –

High-frequency trading HFT) is essentially algorithmic trading in finance by using of sophisticated technological tools and computer algorithms to rapidly trade securities. HFT uses proprietary trading strategies carried out by computers to move in and out of positions in seconds or fractions of a second.

About two years ago, I had the good fortune of being part of a panel discussion around the use of Big Data in a range of trading use-cases (High and low latency trading) covering both hardware based platforms as well as hybrid platforms. An example discussed at length was by using  Big data technologies, coupled with flash memory to improve performance, a firm could develop a strategy that includes weather data, social data or geolocation in real time.

In fact, at that time some financial services firms were already starting to experiment with advanced analytics, coupled with low latency technology, to develop smarter or intelligent trading decisions.

(Ref – http://www.wallstreetandtech.com/latency/big-data-for-intelligent-trading/d/d-id/1268561? )

Now, in the last two years, the above initial trickle has transformed into a gushing waterfall with firms rushing to re-architect their shrink wrapped legacy applications with a view to –

1.Re-tooling their trading infrastructures so that they are more integrated yet loosely coupled and efficient

2.Automating complex trading strategies that are quantitative in nature across a range of asset classes like equities, forex,ETFs and commodities etc

3.Needing to incorporate newer & faster sources of data (social media, sensor data, clickstream date) and not just the conventional sources (market data, position data, M&A data, transaction data etc). Pure speed can only get a firm so far

4.Retrofitting existing trade systems to be able to accommodate a range of mobile clients who have a vested interest in deriving analytics. e.g marry tick data with market structure information to understand why certain securities dip or spike at certain points and the reasons for the same (e.g. institutional selling or equity linked trades with derivatives)

5.Helping traders create algorithms as well as customizing these to be able to generate constant  competitive advantage

In fact the moniker given to this new architectural paradigm is “Smart Data” or the need to generate a holistic view of market state based on hitherto unharness-able data streams. Legacy and shrink wrapped systems are simply unable to meet the above five business requirements as we have seen this across a range of clients.

The key here is the word “integrated” as that signifies a platform that captures all and any kind of structured, unstructured data while allowing traders (and more importantly quants) to develop strategies that can be, tested, simulated and validated –  by first leveraging a classical rules/CEP engine based approach. And secondly, by augmenting this approach by using the tens of analytical packages found in a programming language like R. These packages provide sophisticated time series analysis, financial network analysis and risk modeling etc. The lifecycle of developing such smart strategies that take into account not just streaming/real time data but also combine it with existing silos will be covered in a future post. 

Having thus set the business stage, the next post in this three part series will focus on the architecture of a real world integrated trading platform built around the Apache Hadoop  as well as using other enterprise open source components – enterprise messaging, distributed caching, rules/workflow and CEP engines.

We will then round out the discussion in the third post by focusing on what are the key areas within trading that require a hardware based platform and which areas benefit from software based platforms where Smart Data based techniques have a significant role to play.