ShareChat Data Processing Pipeline

This repository contains a data processing pipeline for scraping, text extraction, translation, sentiment analysis, and sentiment labeling of ShareChat posts. The pipeline is designed to process multilingual posts and generate aggregate sentiment scores for each post.

Usage

1. Scraping:

   • Run the main.py script located in the sharechat_scraper directory using the command python3 main.py.
   • Modify the constants.py file to include leader tags. Leader tags are identifiers for specific ShareChat users. Add leader tags to the LEADER_TAGS list in constants.py.

2. Image to Text Conversion:

   • Execute the tesseract.sh script to extract text from images. Use the command ./tesseract.sh.

3. Text Translation:

   • Run the translation.py script using Python 3 to translate multilingual text to English and perform preprocessing on the English text.
   • Execute python3 translation.py.
   • Preprocessed English texts are saved in translations.pickle.

4. Sentiment Score Calculation for Each Post:

   • Run the sentiment_analysis.py script to calculate sentiment scores for each post.
   • Execute python3 sentiment_analysis.py.
   • All sentiment scores for each post are saved in sentiments.pickle.

5. Aggregate Sentiment Score Calculation and Labeling:

   • Execute the sentiment_labeller.py script to calculate aggregate sentiment scores and label each post.
   • Use the command python3 sentiment_labeller.py.
   • Labelled results are stored in results.pickle.

Result Implementation

User interface (UI) for visualizing sentiment analysis results for each leader. run index.html in frontend

Project Demonstration

https://youtu.be/nNgTlCs4e5U