Capital Markets Surveillance using Big Data and Artificial Intelligence

Recently, The Securities and Exchange Board of India ordered RIL to give up the Rs 447cr of gains made through the network on trades.  It also directed RIL to pay an additional penal interest of 12 per cent per annum from November 29, 2007.  RIL and the 12 front entities were also banned from accessing the equity derivatives market for a year.


Is this a case of Market Manipulation?   The regulators of Capital Markets, SEBI in the case of India, have been empowered to carry out regulatory functions, including Surveillance of Markets.  SEBI in turn also collaborates with Market Infrastructure providers, the Exchanges, and related players, to ensure effective Surveillance.

Purpose of Capital Market Surveillance

Below are some of the main reasons and aim of Capital Markets Surveillance:-

  1. Create level playing field for all Market players : Investors, Hedgers, Traders
  2. Prevent insider trading, market manipulation, abating unfair prices
  3. Aid Exchanges towards Regulatory compliance

You will notice that the RIL case happened in Nov 2007.  What could the Regulator have done to catch this case earlier?  Also, the advent of automated & algorithmic trading have made the job of the Regulator more difficult because of proliferation of Data.  What can be done to prevent drowning in Data?

In this Blog post, usage of Machine Learning to detect and flag all such incidents, and that too on at a quicker pace is being showcased.  This Use Case and solution for using Machine Learning was demonstrated in NSE Fintech Hackathon 2018.

Factual Analysis of the case using Market Data

Analysis was done on the historical equities and futures data for RPL equity for the entire Calendar Year 2007.  In graph below, the blue line shows the Price of Reliance Power shares for the entire Calendar Year 2007.  Notice on Left hand y-axis the price ranges from Rs. 70 to Rs 270.  The second line, with Red color, shows the Daily futures turnover over the same period, with the Right hand y-axis showing the range.


From the chart above a clear pattern with malicious intent can be detected.  In Oct-Nov, it seems that participants with insider information have bought low and sold high.  These transactions have been spread over two separate counters, viz., Equities and Futures Markets.

For the regulator, such type of malicious intents need lot of manual work to catch.  However such patterns can be flagged effortlessly using Machine Learning Algorithms.  Additionally, on a ongoing basis Machine Learning can analyze such patterns for all securities in the traded universe.

Spoiler Alert!

This section is Technical in nature and some Machine Learning know-how is a pre-requisite.  This section can be skimmed/skipped by general audience.

Features of Machine Learning Used

This scenario being elaborated here is case for Machine Learning of Supervised Learning type.  Neural networks usage in this case is not a proper fit as Neural logic favors cases of pattern detection where logic is not known to humans or very complex one, e.g., face recognition, voice recognition, etc.  Whereas in this case, domain knowledge/logic for such pattern detection exists and we know features to use.  This is also specifically because Regulatory aspects themselves provide the detection features to be used.  Gaussian Kernel algorithm is preferred because of non-linear boundaries, small n (features), medium m (training set).  We need to perform feature scaling before using Gaussian Kernel.  Alternative functions like Logistic & sigmoid, are not well suited for this case.  Also, this being, Convex Optimization problem, regression will be able to find global minima.  C value for Gaussian Kernel, which represents penalty for mis-classified training examples, can be optimized using training with different parameters.

The graphs below represent process of Classification of Normal and Abnormal cases using Machine Learning.  The pluses (+) represents Flagged points.   The left side graphs represents Linear Kernel with C (slack of 1).  Notice that one left most plus (+) has been missed.  This can be optimized by using C = 1000 as shown in right side graph.


Using Gaussian Kernel, along with boundary prediction on real data will give the actual analysis as explained in next section.  The model can be made more sturdy.  Also more features added in case more data points are available.

Output/Intelligence through the Machine Learning Process

The graphs below represent Classification of Normal and Abnormal cases using Machine Learning.  Below parameters are being used:

  1. MWPL (Market wise position limit) Daily Data in % terms on x-axis
  2. Rollover of Monthly Futures and Options positions on y-axis


In graph above, notice how some cases have been demarcated by a boundary and marked as plus (+).  These Flagged points are cases which the Exchange and Regulatory Compliance Teams can review further and take appropriate action.  This review can be part of EOD processing.  With huge number of transactions, EOD closure is getting elongated.  This is where Machine Learning can help cut-down detection time.


Artificial Intelligence, along with it sub-domains, including Machine Learning, have a critical role to play in Capital Market Surveillance.  The advent of Internet based trading and the proliferation of algorithmic training has lead to data drown.  Time has come for the regulators and market players, including exchanges, to ask for “help” from Technologists.  Artificial Intelligence has been in existence since some time.  However with more wide spread availability of:

  1. computational power
  2. data storage and usage through Big Data systems for  distributed data including unstructured one,
  3. Logic for formulating predictors/features/factors that have economic value in short, medium, long term (Machine Learning)

not utilizing these as part of Business as usual processes will lead to competitive disadvantage.  Also in growing economies, fair play will lead to fair markets and thus widespread financial inclusion.  Time has come for regulators and exchanges to embrace Machine Learning in their day-to-day IT Operations.

Blockchain COE Setup

Globally we are seeing 3 trends emerging in Technology space, viz., Open Source Software, Sharing Economy and Decentralized Decision Making. Blockchain is at the intersection of all three of these and as a technology is capable of tremendous industry disruption and growth.  Gone is time frame when Blockchain as emerging technology had just theoretical applications; currently we are all set for adoption of Blockchain in a big way.  Below graphic showcases the current stage and future prospects of Blockchain technology.


Way forward towards Adoption of Blockchain

There already is available a broad scope of unique Blockchain platforms, viz. Ethereum, Hyperledger, Multichain, HydraChain, Open Chain, IBM Bluemix Blockchain, Chain, IOTA, Ripple, and many others.

And going forward adoption of Blockchain is going to be driven by three broad set of supply drivers:

  1. Challenger disruptions  developed outside of the core business and/or Technolgy markets ecosystem. These can expected in the next 12 to 18 months
  2. Collaborative efforts to shift the existing value chain to blockchains. While such efforts are already starting, with potentially massive benefits, it is likely to take more than ten years to reach core parts of the system.
  3. Mandated policy where supervisors direct the industry to introduce new market infrastructure, so that costs are reduced or that operational or systemic risk is lessened

Challenges and Opportunities for incumbent IT Services providers

Blockchain presents an extremely complex implementation challenge.  In terms of Technology, Blockchain is a confluence of many latest technologies – Data Science, Big Data, Cloud, Analytic, Encryption, Cryptocurrencies.  From a business angle, Blockchain adoption needs comprehensive knowledge and awareness about the fit of Blockchain towards Revenue driving measures as well as Cost and Operational efficiencies.

Given that most IT organizations are mandated to focus on their current work as usual, Blockchain present a big opportunity to mid-size IT companies to build “New” capability in a big way and be poised to capture a big chunk of future market and revenue.

How a Blockchain COE can help organisations 

Being nimble and agile, Mid to small size IT Companies in niche/focused verticals providing end-to-end services across Business Consulting, Technology and Digital, can adopt Blockchain technology in a big way by building real use case-driven applications using Blockchain. With this in mind, they need to establish Blockchain Center of Excellence (COE) with a foremost aim to define the use cases that would help solve real business challenges while simultaneously building in-house solutions and leverage existing platforms.  This could ultimately reshape the industries they serve and also help their own top-line.  Basically this become an investment area for them with huge potential ROI.

Mid to small size IT Companies can set out to achieve this by establishing a COE to work on Blockchain proofs of concepts (POCs) across each of the leading open source blockchain platforms, initially focusing on Ethereum, Hyperledger Fabric, and Ripple. To build the COE, they need to assemble business analysts (BAs) from across section of their customer-facing teams/verticals to identify problems facing the industry as a whole. Next, the COE should insource a diverse set of talent across consulting, technology and digital. Based on the use cases provided by the BAs, the team can develop Blockchain Accelerators – prototype applications – addressing industry challenges such as, but not limited to, Identity management (Security risks), KYC, insurance claims, mortgage financing, trade finance, payments, and capital market post-trade processing. With each platform requiring different skillsets, coding languages, consensus mechanisms, and more, they can establish dedicated teams for each platform, and expand to other platforms like, Corda and Quorum based on client and industry demand.

The Blockchain ecosystem encompasses large number of players – such as financial institutions, and technology companies, service providers, consortia, start-ups, regulatory institutions and government.  To further expand its presence within the Blockchain ecosystem, the COE can establish strategic partnerships with various industry Blockchain forums.  Eg. the Enterprise Ethereum Alliance, Microsoft Azure Blockchain Council, the Hyperledger forum, etc.

Below graphic show priority focus area for the Blockchain COE.


The COE can work on POCs/project and take learnings from these for fundamental business requirements and modify the application architecture per the learnings.  Depending on the learnings, the team can be expanded from an initial 10-15 person core team to 50+ dedicated business consultants, technologists and digital experts. The COE can pivot from lab-driven development to client-driven pilots and advise clients on project scoping, building business cases with a high probability of entering production. Additionally, Blockchain COEs can advise clients on how to run parallel production for a seamless future transition to Blockchain. They can also work with clients to advise them on how to set up their own Blockchain COEs that align with their respective business goals.  In the long run each of these initiatives will lead to projects and corresponding revenue, as well as build capability and solutions and IPs.

Evolving landscape for Blockchain and Digital Tech adoption in India 

India’s central bank, the RBI, came out with a detailed white paper  detailing the technical aspects of Blockchain and how it is beneficial for financial services sector in India in January 2017 itself.  Apart from these below are few big tailwinds for Blockchain adoption in India.

  1. India Demonetization Or Currency Withdrawal
  2. Progressive Central Bank in Reserve Bank of India (RBI)
  3. Availability of “IndiaStack” – Technology for 1.2 Billion Indians.  IndiaStack is a set of APIs that allows governments, businesses, startups and developers to utilise an unique digital Infrastructure to solve India’s hard problems towards presence-less, paperless, and cashless service delivery.

COE Ideas for India specific needs

As the penetration of internet in India grows and as people get more comfortable with the idea of transacting digitally, digitisation of assets will increase as well. In fact, digitisation of land records scheduled to be completed by 2021 is one of primary aims of the Indian Government under the Digital India Campaign.  Below are some steps which the Blockchain COE can take specific to India market.

  1. Integrating with Adhaar for verification
  2. Run Community Events and Encourage Innovation
  3. Roadshows at Incubators and Accelerators
  4. Approaching Corporates and Businesses through Partners and Affiliates
  5. Work with various players and government bodies as the Regulatory landscape evolves
  6. Work with Banks and Financial Institutions on BankChain an intra-bank Blockchain
  7. Work on use cases like Fraud Prevention


We are all set for a stage where Blockchain adoption would require a fundamental shift in Supplier Ecosystem, which includes the current incumbent IT Services providers.  Blockchain presents a big opportunity to mid-size IT companies to build capability in a big way and be prepared to capture a big chunk of upcoming market and revenue. Setting up Blockchain COE is the foremost step they can take to get started on this journey.  This article presented a broad framework and focus areas for such a Blockchain COE.  Following these steps will accelerate the Blockchain journey.

Options Conversion Strategy

A Conversion is an arbitrage strategy in options trading that can be performed when options are overpriced relative to the underlying stock.  This strategy falls in the arbitrage category; thus, if it is executed well then you can get riskless profit.

In this post I will explain with example how this strategy can be carried out.  More examples of Conversion strategies are on this page.  To do a conversion, the trader buys the underlying stock and offset it with an equivalent synthetic short stock (long put + short call) position.  The underlying stock in this case will be under priced, or, will be at price less than the synthetic short.

The Underlying

On 22 Nov 2016, Marico was trading at 251.55.  Refer below screenshot for details.  Also note the price for the 260 call and put options.


Strategy Positions

To implement the strategy, 260 call option is sold for 1.2.  Put option is bought at 7.75 and the stock is bought at 251.55.  Picture below shows more details about the quantity which depends on lot size, Margin, and other details.


Strategy Performance

The graph below depicts how this strategy will fare for different closing price of the underlying on expiry.  You will notice that irrespective of the value of the underlying, at expiry the strategy gives a positive return.


Strategy Summary

The strategy provides a return of 1.12% on the Total Investment


Given that the investments are generally over shorter horizon, the annualized returns are greater.  In this case the 1 Month return is 17.22%.  This is further amplified in case the strategy is executed nearer to expiry.

Examples of Conversion and Reversion Strategies

In one of my previous blog, I had explained about Conversion Strategy.  At FinArbitrage, we do lot of analysis on a day-to-day basis to search for executable examples of Conversion Reversion Strategies.

As on 29 November 2016, below are the available instances of such strategies.

Points of Caution

  1. These strategies are arbitrage strategies, and as such gets snatched as soon as available.  There are many human as well as automated programs being run by specialized investment desk in search of these strategies.
  2. It is paramount that all the multiple legs of a particular strategy be executed in one go.  Only if all legs are available at an instance, the particular instance of these Conversion or Reversion needs to be entered into.  Thus no leg of the strategy can be entered into in isolation.
  3. Once a particular strategy is executed, it needs to be kept intact till expiry so as to realize the planned gain.  Thus available margin on an ongoing basis need to be monitored and any additional margin as required will have to be made available.

Riskless Profit Idea for Retail Investors

Infosys has announced November 1 as record date for its Rs. 13,000-crore buyback.  In this article, the returns that retails investors, and specifically, small shareholders stand to make, will be analyzed.

As per Sebi regulations, 15% of the offer is to be reserved for small shareholders holding shares upto a value of Rs. 2,00,000 as on the record date.  Thus these investors can buy shares till 27-October and below is the chart showing returns on Rs. 2,00,000.

The columns denotes the buying price for the shares.  Since there is time available till 27-October, investors can keep track of buying price and buy depending on availability of funds.  As the table denotes, lower the buying price, higher the return.


The returns are calculated assuming all small shareholders opt for buy back.  In that case buyback will be oversubscribed, and only part of the shares will be bought back.  At June 2017 share holding pattern of various categories of shareholders, for Small Shareholders (below Rs. 2 lakhs) the acceptance ratio works out to 59% . The remaining shares will have to be sold by the investors on their own.  The rows shows profit on selling the ~41% shares remaining after buyback basis the then ongoing price, i.e., post buy back completion.

“Buy Low and Sell High” is the mantra everyone wants to follow.  But the market offers very few opportunities for investors to actually act on such mantra.  The Infosys buyback is one such opportunity and Investors, especially, Small Shareholders need to make the most of it.

There are 2 added things that will spice up the returns.  One is the Acceptance Ratio, which is likely to be higher as not all investors have complete knowledge, time or inclination to study and subscribe for buyback.  Secondly, Infosys is likely to offer dividend along with the upcoming results on 1 November, which is likely to increase the returns.

How can FinArbitrage help

  1. If you are buying Infosys shares with the intention of benefiting from buyback, ensure that all procedures and mechanism are in place so that you can actually tender your shares in the buyback.  If you are not able to tender in the buyback, your returns will be drastically lower or even negative.  For more details please leave a comments below or Contact Us for more details.
  2. Depending on your holding period, the profit will be categorized as Long Term or Short Term.  If holding period is less than one year, then Short Term tax at your Marginal Rate will be applicable.  In order to optimize your returns and taxes, please leave a comments below or Contact Us for more details.
  3. Brokerage and applicable taxes have been factored to arrive at estimated profits in table above.  To know exact returns you are likely to make, please leave a comments below or Contact Us for more details.
  4. If value of your holdings keeps rising, and goes above Rs. 2,00,000 on Record day, then you may not be counted as Small Shareholder. Thus buy requisite quantity, monitor your holdings value, and, offload shares prior to record date as applicable.  To understand more you may Contact Us or leave comments below.

Computational Investing

Computational Investing Course on Coursera is a good one for beginners. You will learn how to get started creating algorithms that hedge funds and investment professionals use. You will also learn how to develop the platform for machine learning models in python to solve real life Equity investing problems.

Go to this URL to check out this course.  Do review this course and put your feedback in the comments.