BSE Becomes ECH Agile

Business Requirement

At BSE, use of Big data has been the focal point in our Technology stack to harness its benefits within the organization. Data is growing exponentially every year in Bombay Stock Exchange with
a growth of 100 TB annually. Business demands complex analytic reporting, real-time analytics, fraud detection, data products.

Statutory and legal compliance mandates data must be available for providing ad-hoc and need base queries and must meet security requirements. Detection of frauds in near real-time, rumour verification by applying machine learning algorithm on unstructured data collecting from various social media and provide data products to the various vendor of BSE.

Solution Deployment

Following USE Cases implemented:
• Enterprise Data Warehouse offload from IBM ISAS D5600
• Social Media Analytics on Textual Data, Fraud Analytics
• Real-Time analytics, Streaming Analytics, Near Real Time Machine learning
• Big Data security and Audit compliance
• Big Data Disaster Site implementation at the remote place
• Operational Analytics

Objectives were to have Enterprise Data Hub with structure and unstructured data with scalable, highly available, Lambda architecture with lowest the total cost of ownership. The Big Data Lake was implemented using Cloudera Enterprise distribution of HADOOP of 500 TB capacity with a multi-rack design in production site as well as remote disaster site.

Driven by highly competitive market conditions and the need to conduct fraud analysis, BSE has formulated an extensive approach to develop an automated social media monitoring solution. As a part of this process, BSE monitor’s content on BSE listed companies and looks for material information as well as possible rumours on social media sites like Twitter, Facebook
and also on news websites. This solution is tuned to run on a specified frequency and provides an interface to BSE’s dashboard. This solution eliminates manual process of keeping a manual watch on social media.

Solution Benefits

In terms of implementation, overall the project was implemented phased-wise. BSE adopted agile methodologies to work in sprints of three weeks and the output was visible after each sprint. A brief on phased-wise the approach is as below.

As an furtherance to the above-implemented initiative, BSE has also started working to include live streaming input from TV channel and analyze the text streaming in the TV news to detect any kind of rumour about our listed companies, using voice to text mining, initially in English language and later in regional languages.

BSE also implemented Machine translation for regional language transformation from Hindi to English and subsequently implemented text mining algorithm to find the rumour about its listed companies.