Data Exploration

---

The original dataset used for analysis is The Pushshift Reddit Dataset. Our given data is from January 2021 to August 2022.

About the Raw Reddit Dataset

The format of the dataset is parquet and made up of two components: "submissions" and "comments". The submissions file contains 643,255,528 rows and 68 columns, and the comment file contains 4,473,556,762 rows and 21 columns.

The column that we looked into first was the 'subreddit' column. A subreddit is a smaller community in social media Reddit that contains posts associated with the topic. It is the starting point for our Data Selection.

There are some columns that we potentially will use in our project, including numerical columns such as "num_comments"(Submission), "score"(Comment), text columns such as "title"(Submission), "selftext"(Submission), "body"(Comment), and boolean columns such as "over_18", "disable_comments".

Subreddit Selection

We decide to replicate the data collection process from this paper. We select our data from the 12 subreddits in TABLE 1. The "RU Conflict" category directly focuses on the Russia-Ukraine conflict, hence we use all the data from these subreddits. The "Military Related" category contains general discussion on war and military-related topics. To make all the submissions and comments relevant to our topic, we further filter the content and title columns of the "Military Related" subreddits to ensure they contain keywords like ukraine, russia, Putin, etc.

Data Engineering

Our source data from the above subreddits is built in this notebook.

Because our selected reddit posts about the Russia-Ukraine conflict can contain multiple languages, we used John Snow's language detection model, detect_language_375, to create "English", "Russian", and "Ukrainian" language percentage columns on the submission data. Notebook is for creating the three language columns.

We also used Regex to find Market and Energy related reddit posts in different languages, using this notebook.

Exploratory Analysis

FIGURE 1 counts the number of unique submission IDs in the selected subreddits. The resulting subset from the original dataset indicates that the submissions are more active after the Russia-Ukraine conflict became a global issue. This corroborates that we are collecting the right data.

Our initial steps of EDA are to check data size, missing values, etc. TABLE 2 gives an overview about the basic information of the filtered reddit data and our collected external commodity price data.

Stats	Reddit Submissions	Reddit Comments	Commodity Prices
Size	0.05G	1.4G	166KB
Columns	69	22	15
Rows	384,972	9,993,790	1,455
Missing Values	54,375[delete]; 49,239[removed]	367,528[delete]; 741,394[removed]	2,927
Startdate	2022-01-01	2022-01-01	2018-01-02
Enddate	2022-08-31	2022-08-31	2022-08-31

TABLE 2: Basic Information of the Selected Reddit and External Commodity Data. Note:[deleted] is deleted by author; [removed] is removed by moderator.Comments Code and Submissions Code.

Russia-Ukraine Conflict was declared on Feb 24th 2022. We collected reddit and commodity price data before that time in order to compare the trends before and after the event.

Subreddit	# of Submissions	# of Comments	Active Users	Contains Foreign Langauge
CombatFootage	107	270539	50383	36.45%
CredibleDefense	169	64817	4828	18.34%
RussiaUkraineWar2022	25513	481116	48934	27.33%
UkraineConflict	13394	72528	10933	29.79%
UkraineInvasionVideos	14360	116823	16474	30.88%
UkraineWarReports	7746	78552	14175	30.03%
UkraineWarVideoReport	45260	1724675	157469	29.08%
UkrainianConflict	86925	1987549	129635	25.03%
geopolitics	60	23085	5526	13.33%
ukraina	17975	158422	25841	50.05%
ukraine	172818	4990804	262567	27.01%
war	645	24880	7485	21.86%

TABLE 3:Summary Statistics of the Selected Subreddits. Comments Code and Submissions Code.

TABLE 3 shows statistics of each collected subreddit. As shown, some subreddits have many more users and posts than the others. "ukraine", "UkraineWarVideoReport", and "UkrainianConflict" are the top three active subreddits. By using John Snow's language detection model, detect_language_375, we find that there is a significant number of foreign languages in the reddit posts, containing Ukrainian and Russian.

The volumes of Reddit submissions in these three languages from selected subreddits changed similarly from January 2022 to August 2022, as shown in FIGURE 2:. The peak for Ukrainian submissions arrived in February, earlier than the other two languages in March. The submission count in English started decreasing ever since, while for Russian and Ukrainian submissions, the number started to rise again around July 2022.

To categorize the submission records by language, we used Spark NLP pre-trained language detection model detect_language_375 and obtained the prediction confidences toward these three languages for each submission content. With the confidence values, we can define dummy variables for three languages by any desired bar. For this plot, the variables are set to be True by confidence over 90%. Please note that since there are posts with prediction confidence lower than 90% for all three designated languages, the total number of submissions in this plot will not add up to the length of our dataset.

The above charts show the distribution of the number of words. Overall, most comments contain less than 100 words. From the first part, it can be observed that submitters tend to have longer comments than commenters. From the second part, comments above the average score tend to be longer. These findings are intuitively expected.