Twitter Sentiment Analysis

Our client needed a product to simplify the recruitment of new job applicants by analyzing their psychological portrait through social networks, i.e. Twitter. Our team had to create an algorithm that would analyze a person’s profile and posts on Twitter and determine the degree of compliance of the author’s tweets with certain crucial requirements for job applicants.

Choosing a social network for further psychological and sentiment analysis of text

Most social networks have privacy restrictions that make it impossible to parse information from third-party pages. Therefore, it is impossible to use such social networks as a data source for a machine learning model.

Having analyzed the most suitable social platforms, we decided on Twitter as a platform for our future MVP, as far as it allows us to parse information about users’ tweets.

data understanding and updating

To train our model for sentiment analysis of tweets, we needed to process a lot of data, including pre-trained cases that contained information on which kind of phrases are considered to have a positive or negative meaning.

This challenge comprises:

• the possibility that some phrases could be understood the wrong way

• the information itself could appear to be outdated

To avoid any mistakes related to outdated information, we chose to constantly update these data cases. By doing so, we continued retraining the system to keep information relevant.

positive and negative data dividing

To start our model training, we had to divide data by those with positive and negative meanings.

We used the SKLearn library. This tool contains the models of regression, classification, forecasting, prediction, etc.

Therefore, the classification comprised of:

1. Data preparation
2. Transferring these data inputs to the model
3. The model analyzes the data and decides whether it is positive or negative

Other solutions

During the process of Twitter sentiment analysis, we applied natural language processing methods, such as NLTK, StanfordNERTagger, spaCy, and Word2Vec.

Solution # 1: NLTK for tokenizing and cleaning of the tweets

Solution # 2: SpaCy for tokens lemmatization

Solution # 3: StanfordNERTagger to define entities and keywords

Solution # 4: Word2Vec to create models for assessing the sentiment of tweets, constructing a field of word proximity, and studying the context of the message.

1. Initial text data preparing

For a personal tweet analysis, we needed an accurate model for assessing the sentiment of tweets.

2. Tweet cleaning

• Removing of punctuation, URL links, and other technical tweet components
• Tokenization and lemmatization of the pre-cleared text

This graph shows the statistics on the length of the tweets. Based on this data, we can conclude that Twitter users are predominantly sharing tweets that are about 5 words long.

3. Model training

After data cleaning, we trained a model based on the cleaned tweets. Based on a statistical analysis of tweets, we proposed a hypothesis that the quality of the tweet sentiment analysis is directly affected by tweets’ length.

Therefore, further training and model validation was conducted on the basis of tweets with a limited length of 5 words and an unlimited amount of processed data.

January 2019