Author(s: Raviraj Shinde Natural Language Processing Investing on the Stock Market can be a good way to combat inflation. The rise in prices for most items and services that are common or daily use such as food, clothing (clothes), housing, transportation, staples, etc. is called inflation. Basically, with 100 rupees you won’t be able to buy as many vada pavs (wadapavs) as you could last year. In the pandemic-struck financial year of 2020-2021 a whopping 142 lakh new investors have started trading in the stock market. To make smart investments in stock markets, you must be able to analyze financial and business news. What company is expanding its business sectors? Or is it heading for bankruptcy? It is important to stay informed about every deal or fallout in the markets. For those new to financial news, it can be difficult to grasp the details of financial news. A tool will be created that pulls together the most recent headlines from every Indian stock market. It then runs the sentiment analysisr, which is specialized in financial news. This creates an aggregate sentiment to help stock investors understand the stock news. The Real-Time Stock News Analysis: Urllib Numpy Pandas BeautifulSoup HuggingFace Transformers Library Getting the Data. Let’s start with financial news web-scraping from trusted sources. I discovered that tickertape separates its stock alphabetically. It also uses a tick (keyword) in the URL to link to specific stocks. This interface is very useful in web-scraping. Let’s first create a list all ticks in tickertape. https://medium.com/media/96d44ff10ea473a0e420ee310eb13565/href I have created a get_ticks function to extract all the ticks in one go from tickertape/stocks. Every stock’s name is linked to the screening page. The link is of the format “/stocks/stock-name-tick”. To navigate seamlessly to the stock, we need to get this tick. BeautifulSoup, urllib and the following were used to extract the tick and the name of each stock. They are stored in an alphabetical Pandas DataFrame. https://medium.com/media/9d53280bac7af7524ccc4d95b4b4d1b2/hrefhttps://medium.com/media/110e192e4078594bed34813f259d469c/href The output looks like this: Now moving on to the news part. We use the same pattern as get_ticks to do this. News related to each stock is located at “tickertape/stock-tick/news?checklist=basic&ref=stock-overview_overview-sections&type=news”. This news can be extracted again using BeautifulSoup or urllib. https://medium.com/media/530f30521725df5af4275f1c19465db1/href Running a for loop where we replace the stock-tick part with the elements in the tick column of our DataFrame we can extract the news of each stock and store it in a list called news. https://medium.com/media/c212436d3f22067ee5f98987c54c405e/href Now that we have the news in the format that we need we can move on to the sentiment analysis part. The Sentiment Analysis for Financial News I will use FinBERT, a pre-trained model. FinBERT, a pre-trained model that is specifically designed for financial news and text analysis. This model is built on BERT architecture. FinBERT is easy to use when combined with HuggingFace’s transformers library. For more details on BERT and FinBERT you can refer to my blog: Financial News Sentiment Analysis using FinBERT https://medium.com/media/235d1fbaca13018804a619a386b23336/href Now that FinBERT has been loaded we start the process of analysis. FinBERT’s output is in numeric form so we will need to convert it to something more human-friendly. We will use labels to create a Python dictionary. https://medium.com/media/1c39c6ecbf2fd3cd55fcaa2caa78cf78/href Now the good part. This is the good part. We use the tokenizer object in order to preprocess text using NLP best practices. The output of the tokenizer is passed to finbert to perform sentiment analysis. https://medium.com/media/4abc8c3bfe734f416975b420565b7590/href We do this for every stock in our list. To make it easier to group the sentiments for every headline, we store them in a list. The output of the function is stored in a number of lists, called tot_val. Because the website doesn’t contain news, some lists may appear empty. These stocks are returned neutral by me as their sentiment. Then we can create an aggregate of all the sentiments. This is done by adding the sentiments from each stock’s headline. To do this, I add +1 to my agg variable for headlines that are positive or -1 if they’re negative. Based on the value of the last agg variable, I assign the stock a positive, negative or neutral. https://medium.com/media/b0f42fab537e0ab55a539a25555ee22e/href Finally, we pass our list of sentiments through the get_sent and obtain the aggregate sentiments. These sentiments are stored in a file which is then assigned to a column sentiment from our tick_df DataFrame. https://medium.com/media/f9d53a2cc597e75cc6d3539b9f7743ac/href The output looks like this: Thus we have successfully managed to create a sentiment analyzer for the Indian stock market based on financial news. This project should be used only for education purposes and not for investment decision making. Also, this blog is in no way a sponsorship for tickertape or any other entity mentioned in the blog The entire code is available on my git profile: GitHub – Raviraj2000/Realtime-Stock-News-Sentiment-Analyzer Thanks for reading! Real-Time Stock News Sentiment Analysiszer originally appeared in on Medium. People are responding and highlighting this story. Published via
