Capital Markets Surveillance using Big Data and Artificial Intelligence

Recently, The Securities and Exchange Board of India ordered RIL to give up the Rs 447cr of gains made through the network on trades.  It also directed RIL to pay an additional penal interest of 12 per cent per annum from November 29, 2007.  RIL and the 12 front entities were also banned from accessing the equity derivatives market for a year.


Is this a case of Market Manipulation?   The regulators of Capital Markets, SEBI in the case of India, have been empowered to carry out regulatory functions, including Surveillance of Markets.  SEBI in turn also collaborates with Market Infrastructure providers, the Exchanges, and related players, to ensure effective Surveillance.

Purpose of Capital Market Surveillance

Below are some of the main reasons and aim of Capital Markets Surveillance:-

  1. Create level playing field for all Market players : Investors, Hedgers, Traders
  2. Prevent insider trading, market manipulation, abating unfair prices
  3. Aid Exchanges towards Regulatory compliance

You will notice that the RIL case happened in Nov 2007.  What could the Regulator have done to catch this case earlier?  Also, the advent of automated & algorithmic trading have made the job of the Regulator more difficult because of proliferation of Data.  What can be done to prevent drowning in Data?

In this Blog post, usage of Machine Learning to detect and flag all such incidents, and that too on at a quicker pace is being showcased.  This Use Case and solution for using Machine Learning was demonstrated in NSE Fintech Hackathon 2018.

Factual Analysis of the case using Market Data

Analysis was done on the historical equities and futures data for RPL equity for the entire Calendar Year 2007.  In graph below, the blue line shows the Price of Reliance Power shares for the entire Calendar Year 2007.  Notice on Left hand y-axis the price ranges from Rs. 70 to Rs 270.  The second line, with Red color, shows the Daily futures turnover over the same period, with the Right hand y-axis showing the range.


From the chart above a clear pattern with malicious intent can be detected.  In Oct-Nov, it seems that participants with insider information have bought low and sold high.  These transactions have been spread over two separate counters, viz., Equities and Futures Markets.

For the regulator, such type of malicious intents need lot of manual work to catch.  However such patterns can be flagged effortlessly using Machine Learning Algorithms.  Additionally, on a ongoing basis Machine Learning can analyze such patterns for all securities in the traded universe.

Spoiler Alert!

This section is Technical in nature and some Machine Learning know-how is a pre-requisite.  This section can be skimmed/skipped by general audience.

Features of Machine Learning Used

This scenario being elaborated here is case for Machine Learning of Supervised Learning type.  Neural networks usage in this case is not a proper fit as Neural logic favors cases of pattern detection where logic is not known to humans or very complex one, e.g., face recognition, voice recognition, etc.  Whereas in this case, domain knowledge/logic for such pattern detection exists and we know features to use.  This is also specifically because Regulatory aspects themselves provide the detection features to be used.  Gaussian Kernel algorithm is preferred because of non-linear boundaries, small n (features), medium m (training set).  We need to perform feature scaling before using Gaussian Kernel.  Alternative functions like Logistic & sigmoid, are not well suited for this case.  Also, this being, Convex Optimization problem, regression will be able to find global minima.  C value for Gaussian Kernel, which represents penalty for mis-classified training examples, can be optimized using training with different parameters.

The graphs below represent process of Classification of Normal and Abnormal cases using Machine Learning.  The pluses (+) represents Flagged points.   The left side graphs represents Linear Kernel with C (slack of 1).  Notice that one left most plus (+) has been missed.  This can be optimized by using C = 1000 as shown in right side graph.


Using Gaussian Kernel, along with boundary prediction on real data will give the actual analysis as explained in next section.  The model can be made more sturdy.  Also more features added in case more data points are available.

Output/Intelligence through the Machine Learning Process

The graphs below represent Classification of Normal and Abnormal cases using Machine Learning.  Below parameters are being used:

  1. MWPL (Market wise position limit) Daily Data in % terms on x-axis
  2. Rollover of Monthly Futures and Options positions on y-axis


In graph above, notice how some cases have been demarcated by a boundary and marked as plus (+).  These Flagged points are cases which the Exchange and Regulatory Compliance Teams can review further and take appropriate action.  This review can be part of EOD processing.  With huge number of transactions, EOD closure is getting elongated.  This is where Machine Learning can help cut-down detection time.


Artificial Intelligence, along with it sub-domains, including Machine Learning, have a critical role to play in Capital Market Surveillance.  The advent of Internet based trading and the proliferation of algorithmic training has lead to data drown.  Time has come for the regulators and market players, including exchanges, to ask for “help” from Technologists.  Artificial Intelligence has been in existence since some time.  However with more wide spread availability of:

  1. computational power
  2. data storage and usage through Big Data systems for  distributed data including unstructured one,
  3. Logic for formulating predictors/features/factors that have economic value in short, medium, long term (Machine Learning)

not utilizing these as part of Business as usual processes will lead to competitive disadvantage.  Also in growing economies, fair play will lead to fair markets and thus widespread financial inclusion.  Time has come for regulators and exchanges to embrace Machine Learning in their day-to-day IT Operations.

Options Conversion Strategy

A Conversion is an arbitrage strategy in options trading that can be performed when options are overpriced relative to the underlying stock.  This strategy falls in the arbitrage category; thus, if it is executed well then you can get riskless profit.

In this post I will explain with example how this strategy can be carried out.  More examples of Conversion strategies are on this page.  To do a conversion, the trader buys the underlying stock and offset it with an equivalent synthetic short stock (long put + short call) position.  The underlying stock in this case will be under priced, or, will be at price less than the synthetic short.

The Underlying

On 22 Nov 2016, Marico was trading at 251.55.  Refer below screenshot for details.  Also note the price for the 260 call and put options.


Strategy Positions

To implement the strategy, 260 call option is sold for 1.2.  Put option is bought at 7.75 and the stock is bought at 251.55.  Picture below shows more details about the quantity which depends on lot size, Margin, and other details.


Strategy Performance

The graph below depicts how this strategy will fare for different closing price of the underlying on expiry.  You will notice that irrespective of the value of the underlying, at expiry the strategy gives a positive return.


Strategy Summary

The strategy provides a return of 1.12% on the Total Investment


Given that the investments are generally over shorter horizon, the annualized returns are greater.  In this case the 1 Month return is 17.22%.  This is further amplified in case the strategy is executed nearer to expiry.

Examples of Conversion and Reversion Strategies

In one of my previous blog, I had explained about Conversion Strategy.  At FinArbitrage, we do lot of analysis on a day-to-day basis to search for executable examples of Conversion Reversion Strategies.

As on 29 November 2016, below are the available instances of such strategies.

Points of Caution

  1. These strategies are arbitrage strategies, and as such gets snatched as soon as available.  There are many human as well as automated programs being run by specialized investment desk in search of these strategies.
  2. It is paramount that all the multiple legs of a particular strategy be executed in one go.  Only if all legs are available at an instance, the particular instance of these Conversion or Reversion needs to be entered into.  Thus no leg of the strategy can be entered into in isolation.
  3. Once a particular strategy is executed, it needs to be kept intact till expiry so as to realize the planned gain.  Thus available margin on an ongoing basis need to be monitored and any additional margin as required will have to be made available.

Riskless Profit Idea for Retail Investors

Infosys has announced November 1 as record date for its Rs. 13,000-crore buyback.  In this article, the returns that retails investors, and specifically, small shareholders stand to make, will be analyzed.

As per Sebi regulations, 15% of the offer is to be reserved for small shareholders holding shares upto a value of Rs. 2,00,000 as on the record date.  Thus these investors can buy shares till 27-October and below is the chart showing returns on Rs. 2,00,000.

The columns denotes the buying price for the shares.  Since there is time available till 27-October, investors can keep track of buying price and buy depending on availability of funds.  As the table denotes, lower the buying price, higher the return.


The returns are calculated assuming all small shareholders opt for buy back.  In that case buyback will be oversubscribed, and only part of the shares will be bought back.  At June 2017 share holding pattern of various categories of shareholders, for Small Shareholders (below Rs. 2 lakhs) the acceptance ratio works out to 59% . The remaining shares will have to be sold by the investors on their own.  The rows shows profit on selling the ~41% shares remaining after buyback basis the then ongoing price, i.e., post buy back completion.

“Buy Low and Sell High” is the mantra everyone wants to follow.  But the market offers very few opportunities for investors to actually act on such mantra.  The Infosys buyback is one such opportunity and Investors, especially, Small Shareholders need to make the most of it.

There are 2 added things that will spice up the returns.  One is the Acceptance Ratio, which is likely to be higher as not all investors have complete knowledge, time or inclination to study and subscribe for buyback.  Secondly, Infosys is likely to offer dividend along with the upcoming results on 1 November, which is likely to increase the returns.

How can FinArbitrage help

  1. If you are buying Infosys shares with the intention of benefiting from buyback, ensure that all procedures and mechanism are in place so that you can actually tender your shares in the buyback.  If you are not able to tender in the buyback, your returns will be drastically lower or even negative.  For more details please leave a comments below or Contact Us for more details.
  2. Depending on your holding period, the profit will be categorized as Long Term or Short Term.  If holding period is less than one year, then Short Term tax at your Marginal Rate will be applicable.  In order to optimize your returns and taxes, please leave a comments below or Contact Us for more details.
  3. Brokerage and applicable taxes have been factored to arrive at estimated profits in table above.  To know exact returns you are likely to make, please leave a comments below or Contact Us for more details.
  4. If value of your holdings keeps rising, and goes above Rs. 2,00,000 on Record day, then you may not be counted as Small Shareholder. Thus buy requisite quantity, monitor your holdings value, and, offload shares prior to record date as applicable.  To understand more you may Contact Us or leave comments below.

Computational Investing

Computational Investing Course on Coursera is a good one for beginners. You will learn how to get started creating algorithms that hedge funds and investment professionals use. You will also learn how to develop the platform for machine learning models in python to solve real life Equity investing problems.

Go to this URL to check out this course.  Do review this course and put your feedback in the comments.