Last week as I attended the Feminist X-Phi workshop out of Cuny, during a lecture, Jordan Wylie mentioned this project called eMFD. eMFD stands for extra moral foundations dictionary and what it is is a “crowd based approach to extracting moral intuitions from texts” (see (Hopp et al., 2021)). What Hopp et al have done, is created a moral intuitions dictionary where words in a given document are compared to words in a dictionary. The words in the dictionary have been given a score where the score is the probability that that word corresponds to each of five moral foundations and each of five sentiment scores. This results in a 10 item vector for each word where each of the ten vectors is an average of all of the individual word scores per moral foundation and sentiment score for each occurrence of that word. The moral foundations and sentiments are: care/harm, fairness/cheating, loyalty/betrayal, authority/subversion and sanctity/degradation (missing reference).

This might be the one. Thus far, I have been looking to determine the moral content algorithmically by comparing the sentiments expressed in a given song as understood according to various sentiment analysis software with the contents in those songs. For instance, the song “Ballin” (missing reference) expresses “joy” in the following sentences:

## [1] "Now a nigga cannot answer calls because I am making lots of money."
## [2] "Because I am getting money now, I know you heard that."            
## [3] "Young nigga on the corner, bitch, I had to serve crack."           
## [4] "We came up on dirty money, I gave it a birdbath."                  
## [5] "Now a nigga cannot answer calls because I am making lots of money."
## [6] "I been waking up to get the money, woah, woah."                    
## [7] "Grab my baby mama that new Bentayga."                              
## [8] "Now a nigga cannot answer calls because I am making lots of money."
## [9] "I have been making lots of money, little nigga."

Which I then thought about comparing the ways in which the sentiments where expressed with other genres, for instance country music. Here is the sentiment “joy” as it is expressed in the song “More Hearts Than Mine” (More Hearts Than Mine by Ingrid Andress | Billboard The Hot 100 Chart, 2019)

## [1] "Pack a shirt for church because we will go." 
## [2] "She falls in love a little faster than."     
## [3] "She falls in love a little faster than I do."

This is controversial, but my question had to do with whether we might think the expression of “joy” in the first example is synonymous with virtue, a virtuous life, a good life, well-being, etc. What about in comparison with the second example? Does one express virtue to a greater extent than the other?

We might think that in line with materialism, that yes, maybe the expressions of joy in the first example are expressions of joy which we might think unproblematic. But what about from other kinds of ethical frameworks? For instance, according to an ethic of care, would the expression of “joy” in the above contexts be synonymous with virtue, a virtuous life?

While I think that this approach might be useful for discovering whether popular music encourages virtuous or vicious behavior, it is not clear whether we might truly understand the contexts for the expression of joy here. For instance, according to a care ethic, we might associate virtue with behaviors and practices that facilitate loving relationships (missing reference). Does the first line in the first example express the belief that earning income is more important than maintaining relationships with others? It is not clear that it does. What about the lines in the second example?

eMFD

According to Jonathan Haidt and others, there are five moral foundations (E.g. (missing reference)). What if we could measure texts according to whether they correspond with these five moral foundations? There have been a few attempts in this direction (see (Graham et al., 2011)), but there have also been problems with these attempts (see (Hopp et al., 2021)).

According to the authors, eMFD is said to improve on previous attempts at measuring texts according to moral foundations theory in a number of ways. First, rather than relying on moral words selection by experts, eMFD uses crowd sourced annotation. 854 annotators where drawn from the general American population using the crowd sourcing platform Prolific Academic.

First data frame will utilize emfd dictionary with the “all” probability mapping, “box of words” scoring method and sentiment out metrics. Sentiment metrics are the average sentiment score for each moral foundation (virtue): care, fairness, loyalty, authority, sanctity; and for (vice) harm, cheating, betrayal, subversion, and degradation.

From the tutorial:

Sentiment Scores: Each word is assigned 5 sentiment scores that denote the average sentiment of the foundation context in which this word appeared. For example, the word “kill” has an average “care_sent” of -0.69, meaning that all “care-harm” highlights in which “kill” appeared had an average, negative sentiment of -0.69.

There are two ways in which we can assign sentiment probability:

  1. use all probabilities assuming that each word matched with the dictionary is used as an indicator of multiple foundations with the probabilities as weights.

  2. Assign a single probability to each word in the dictionary according to the foundation with the highest probability.

I compare the songs “Life is Good,” “Ballin,” “More Hearts than Mine,” and “Bones.”

The data frame below df utilizes the first method while df4 the second.

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'all'
SCORE_METHOD = 'bow'
OUT_METRICS = 'sentiment'
OUT_CSV_PATH = 'emfd-all-bow-sent.csv'

df = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 1  20% |❤❤❤             | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df.to_csv(OUT_CSV_PATH, index=False)

col_list = ["care_p", "fairness_p", "loyalty_p", "authority_p", "sanctity_p", "care_sent", "fairness_sent", "loyalty_sent", "authority_sent", "sanctity_sent", "moral_nonmoral_ratio", "f_var", "sent_var"]
df_csv1 = pd.read_csv("emfd-all-bow-sent.csv", usecols=col_list)
pd.set_option("display.max_columns", None)
df_csv1
##      care_p  fairness_p  loyalty_p  authority_p  sanctity_p  care_sent  \
## 0  0.106867    0.087836   0.096660     0.084637    0.094600  -0.084095   
## 1  0.095360    0.082747   0.084345     0.085235    0.090488  -0.091297   
## 2  0.000000    0.000000   0.000000     0.000000    0.000000   0.000000   
## 3  0.099883    0.090181   0.095119     0.078157    0.091662  -0.073710   
## 4  0.112436    0.090973   0.089485     0.091214    0.084529  -0.084626   
## 
##    fairness_sent  loyalty_sent  authority_sent  sanctity_sent  \
## 0      -0.022420      0.036468        0.047830      -0.017547   
## 1      -0.025913      0.013727       -0.062388      -0.045111   
## 2       0.000000      0.000000        0.000000       0.000000   
## 3      -0.027324     -0.008354        0.011478      -0.029747   
## 4      -0.021881      0.015219       -0.013411      -0.011573   
## 
##    moral_nonmoral_ratio     f_var  sent_var  
## 0              0.700000  0.000075  0.002796  
## 1              0.525773  0.000027  0.001555  
## 2              0.000000  0.000000  0.000000  
## 3              0.661972  0.000065  0.001002  
## 4              0.435484  0.000117  0.001370

What I currently like about eMFD, is that from it, users can use its scores to “assess whether an entity primarily engages in actions that uphold or violate certain moral conduct and, likewise, whether an entity is the target of primarily moral or immoral actions” (Hopp et al., 2021, p. 10). Exactly how we might assess these scores in the way mentioned by Hopp et al is not yet clear to me.

Second data frame df1 will use emfd dictionary type, all probability mapping, named entity recognition pat scoring method and sentiment OUT_METRICS.

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'all'
SCORE_METHOD = 'bow'
OUT_METRICS = 'vice-virtue'
OUT_CSV_PATH = 'emfd-all-bow-vv.csv'

df1 = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 3  60% |❤❤❤❤❤❤❤❤❤       | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df1.to_csv(OUT_CSV_PATH, index=False)

col_list = ["care.virtue", "fairness.virtue", "loyalty.virtue", "authority.virtue", "sanctity.virtue", "care.vice", "fairness.vice", "loyalty.vice", "authority.vice", "sanctity.vice", "moral_nonmoral_ratio", "f_var"]
df_csv1 = pd.read_csv("emfd-all-bow-vv.csv", usecols=col_list)
pd.set_option("display.max_columns", None)
df_csv1
##    care.virtue  fairness.virtue  loyalty.virtue  authority.virtue  \
## 0     0.037599         0.035883        0.054402          0.031528   
## 1     0.029243         0.034106        0.054111          0.027794   
## 2     0.000000         0.000000        0.000000          0.000000   
## 3     0.025817         0.029216        0.031973          0.038363   
## 4     0.032688         0.028323        0.047205          0.038865   
## 
##    sanctity.virtue  care.vice  fairness.vice  loyalty.vice  authority.vice  \
## 0         0.031907   0.069268       0.051953      0.042258        0.053108   
## 1         0.033932   0.065505       0.047217      0.030234        0.056288   
## 2         0.000000   0.000000       0.000000      0.000000        0.000000   
## 3         0.057681   0.074067       0.057183      0.063146        0.039794   
## 4         0.037552   0.079748       0.061117      0.042280        0.051771   
## 
##    sanctity.vice  moral_nonmoral_ratio     f_var  
## 0       0.061719              0.700000  0.000170  
## 1       0.056250              0.525773  0.000194  
## 2       0.000000              0.000000  0.000000  
## 3       0.033981              0.661972  0.000274  
## 4       0.045817              0.435484  0.000225

Third data frame df2 will use emfd dictionary type, all probability mapping, bow scoring, and virtue-vice OUT_METRICS.

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'all'
SCORE_METHOD = 'bow'
OUT_METRICS = 'vice-virtue'
OUT_CSV_PATH = 'all-pat-vir-vice.csv'

df2 = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 3  60% |❤❤❤❤❤❤❤❤❤       | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df2.to_csv(OUT_CSV_PATH, index=False)

This is a copy of the first data frame with weighted averages according to the foundation with highest probability:

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'single'
SCORE_METHOD = 'bow'
OUT_METRICS = 'sentiment'
OUT_CSV_PATH = 'emfd-single-bow-sent.csv'

df4 = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 2  40% |❤❤❤❤❤❤          | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df4.to_csv(OUT_CSV_PATH, index=False)

col_list = ["care_p", "fairness_p", "loyalty_p", "authority_p", "sanctity_p", "care_sent", "fairness_sent", "loyalty_sent", "authority_sent", "sanctity_sent", "moral_nonmoral_ratio", "f_var", "sent_var"]
df_csv1 = pd.read_csv("emfd-single-bow-sent.csv", usecols=col_list)
pd.set_option("display.max_columns", None)
df_csv1
##      care_p  fairness_p  loyalty_p  authority_p  sanctity_p  care_sent  \
## 0  0.044354    0.024016   0.028571     0.010511    0.034242  -0.027560   
## 1  0.039825    0.021815   0.027860     0.022583    0.020579  -0.047014   
## 2  0.000000    0.000000   0.000000     0.000000    0.000000   0.000000   
## 3  0.049724    0.018616   0.020174     0.001620    0.043426  -0.030096   
## 4  0.064743    0.024445   0.017069     0.022588    0.007375  -0.068176   
## 
##    fairness_sent  loyalty_sent  authority_sent  sanctity_sent  \
## 0      -0.004559      0.028911       -0.002667      -0.014070   
## 1      -0.013808      0.028244       -0.010553      -0.027475   
## 2       0.000000      0.000000        0.000000       0.000000   
## 3       0.004943      0.030486       -0.001227      -0.019184   
## 4      -0.008798      0.024374        0.000978       0.008186   
## 
##    moral_nonmoral_ratio     f_var  sent_var  
## 0              0.700000  0.000157  0.000435  
## 1              0.525773  0.000063  0.000767  
## 2              0.000000  0.000000  0.000000  
## 3              0.661972  0.000387  0.000546  
## 4              0.435484  0.000484  0.001253

And with virtue-vice metrics:

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'all'
SCORE_METHOD = 'bow'
OUT_METRICS = 'vice-virtue'
OUT_CSV_PATH = 'emfd-all-bow-vv.csv'

df5 = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 3  60% |❤❤❤❤❤❤❤❤❤       | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df5.to_csv(OUT_CSV_PATH, index=False)

col_list = ["care.virtue", "fairness.virtue", "loyalty.virtue", "authority.virtue", "sanctity.virtue", "care.vice", "fairness.vice", "loyalty.vice", "authority.vice", "sanctity.vice", "moral_nonmoral_ratio", "f_var"]
df_csv3 = pd.read_csv("emfd-all-bow-vv.csv", usecols=col_list)
pd.set_option("display.max_columns", None)
df_csv3
##    care.virtue  fairness.virtue  loyalty.virtue  authority.virtue  \
## 0     0.037599         0.035883        0.054402          0.031528   
## 1     0.029243         0.034106        0.054111          0.027794   
## 2     0.000000         0.000000        0.000000          0.000000   
## 3     0.025817         0.029216        0.031973          0.038363   
## 4     0.032688         0.028323        0.047205          0.038865   
## 
##    sanctity.virtue  care.vice  fairness.vice  loyalty.vice  authority.vice  \
## 0         0.031907   0.069268       0.051953      0.042258        0.053108   
## 1         0.033932   0.065505       0.047217      0.030234        0.056288   
## 2         0.000000   0.000000       0.000000      0.000000        0.000000   
## 3         0.057681   0.074067       0.057183      0.063146        0.039794   
## 4         0.037552   0.079748       0.061117      0.042280        0.051771   
## 
##    sanctity.vice  moral_nonmoral_ratio     f_var  
## 0       0.061719              0.700000  0.000170  
## 1       0.056250              0.525773  0.000194  
## 2       0.000000              0.000000  0.000000  
## 3       0.033981              0.661972  0.000274  
## 4       0.045817              0.435484  0.000225

Virtue-Vice with single PROB_MAP.

import os
import pandas as pd 
import numpy as np
# import seaborn as sns
# from matplotlib import pyplot as plt
from emfdscore.scoring import score_docs

template_input = pd.read_csv('/home/redapemusic35/VimWiki2/Learning/Research/Sentiment-Topic-Modeling/lyrics/comp.csv', header=None)
template_input.head()
##                                                    0
## 0  'I can''t wait to show you where I grew up Wal...
## 1  'I don''t really wanna go. I don''t really wan...
## 2                                                  '
## 3  'Dancing in the kitchen. You singing my favori...
## 4  'Workin'' on the weekend like usual. Way off i...
num_docs = len(template_input)

DICT_TYPE = 'emfd'
PROB_MAP = 'single'
SCORE_METHOD = 'bow'
OUT_METRICS = 'vice-virtue'
OUT_CSV_PATH = 'emfd-single-bow-vv.csv'

df6 = score_docs(template_input,DICT_TYPE,PROB_MAP,SCORE_METHOD,OUT_METRICS,num_docs)
## Processed: 0 N/A% |                | Elapsed Time: 0:00:00 ETA:  --:--:--
## Processed: 3  60% |❤❤❤❤❤❤❤❤❤       | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 4  80% |❤❤❤❤❤❤❤❤❤❤❤❤    | Elapsed Time: 0:00:00 ETA:   0:00:00
## Processed: 5 100% |❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤❤| Elapsed Time: 0:00:00 Time:  0:00:00
df6.to_csv(OUT_CSV_PATH, index=False)

col_list = ["care.virtue", "fairness.virtue", "loyalty.virtue", "authority.virtue", "sanctity.virtue", "care.vice", "fairness.vice", "loyalty.vice", "authority.vice", "sanctity.vice", "moral_nonmoral_ratio", "f_var"]
df_csv4 = pd.read_csv("emfd-single-bow-vv.csv", usecols=col_list)
pd.set_option("display.max_columns", None)
df_csv4
##    care.virtue  fairness.virtue  loyalty.virtue  authority.virtue  \
## 0     0.018672         0.004774        0.015820          0.002336   
## 1     0.015829         0.003833        0.019765          0.006664   
## 2     0.000000         0.000000        0.000000          0.000000   
## 3     0.004741         0.003152        0.013682          0.000000   
## 4     0.012133         0.007430        0.007145          0.016696   
## 
##    sanctity.virtue  care.vice  fairness.vice  loyalty.vice  authority.vice  \
## 0         0.013799   0.025682       0.019243      0.012750        0.008175   
## 1         0.002877   0.023995       0.017982      0.008095        0.015919   
## 2         0.000000   0.000000       0.000000      0.000000        0.000000   
## 3         0.028080   0.044983       0.015464      0.006491        0.001620   
## 4         0.006217   0.052610       0.017015      0.009924        0.005892   
## 
##    sanctity.vice  moral_nonmoral_ratio     f_var  
## 0       0.020443              0.700000  0.000054  
## 1       0.017701              0.525773  0.000053  
## 2       0.000000              0.000000  0.000000  
## 3       0.015346              0.661972  0.000196  
## 4       0.001157              0.435484  0.000212

Works Cited

  1. More Hearts Than Mine by Ingrid Andress | Billboard The Hot 100 Chart. (2019). /charts/hot-100/2020-03-14
  2. Sliwa, P. (2012). In Defense of Moral Testimony. Philosophical Studies, 158(2), 175–195. https://doi.org/10.1007/s11098-012-9887-6
  3. Kiritchenko, S., & Mohammad, S. M. Happy Accident: A Sentiment Composition Lexicon for Opposing Polarity Phrases. 8.
  4. Kiritchenko, S., Zhu, X., & Mohammad, S. M. (2014). Sentiment Analysis of Short Informal Texts. Journal of Artificial Intelligence Research (JAIR), 50, 723–762.
  5. Kiritchenko, S., Zhu, X., & Mohammad, S. M. (2014). Sentiment Analysis of Short Informal Texts. Journal of Artificial Intelligence Research, 50, 723–762. https://doi.org/10.1613/jair.4272
  6. Kiritchenko, S., & Mohammad, S. (2016). The Effect of Negators, Modals, and Degree Adverbs on Sentiment Composition. Proceedings of the 7th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, 43–52. https://doi.org/10.18653/v1/W16-0410
  7. Kiritchenko, S., & Mohammad, S. M. (2016). Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling. Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL).
  8. Kiritchenko, S., & Mohammad, S. M. (2016). The Effect of Negators, Modals, and Degree Adverbs on Sentiment Composition. Proceedings of the Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis (WASSA).
  9. Kiritchenko, S., & Mohammad, S. M. (2016). Capturing Reliable Fine-Grained Sentiment Associations by Crowdsourcing and Best–Worst Scaling. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 811–817. https://doi.org/10.18653/v1/N16-1095
  10. Kiritchenko, S., Mohammad, S. M., & Salameh, M. (2016, June). Semeval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Proceedings of the International Workshop on Semantic Evaluation.
  11. Kiritchenko, S., Mohammad, S., & Salameh, M. (2016). SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 42–51. https://doi.org/10.18653/v1/S16-1004
  12. Kiritchenko, S., & Mohammad, S. M. (2016). Sentiment Composition of Words with Opposing Polarities. Proceedings of the 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), San Diego, California.
  13. Kiritchenko, S., & Mohammad, S. M. (2016). Sentiment Composition of Words with Opposing Polarities. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 1102–1108. https://doi.org/10.18653/v1/N16-1128
  14. Kiritchenko, S., & Mohammad, S. M. (2016). Happy Accident: A Sentiment Composition Lexicon for Opposing Polarity Phrases. Proceedings of 10th Edition of the the Language Resources and Evaluation Conference (LREC).
  15. Kiritchenko, S., Mohammad, S. M., & Salameh, M. (2016, June). Semeval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Proceedings of the International Workshop on Semantic Evaluation.
  16. Kiritchenko, S., Mohammad, S., & Salameh, M. (2016). SemEval-2016 Task 7: Determining Sentiment Intensity of English and Arabic Phrases. Proceedings of the 10th International Workshop on Semantic Evaluation (SemEval-2016), 42–51. https://doi.org/10.18653/v1/S16-1004
  17. Mohammad, S. M., & Turney, P. D. Emotions Evoked by Common Words and Phrases: Using Mechanical Turk to Create an Emotion Lexicon. 9.
  18. Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a Word-Emotion Association Lexicon. Computational Intelligence, 29(3), 436–465.
  19. Mohammad, S. M. (2018). Word Affect Intensities. Proceedings of the 11th Edition of the Language Resources and Evaluation Conference (LREC-2018).
  20. Mohammad, S. (2018). Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 174–184. https://doi.org/10.18653/v1/P18-1017
  21. Mohammad, S. M. (2018). Obtaining Reliable Human Ratings of Valence, Arousal, and Dominance for 20,000 English Words. Proceedings of the Annual Conference of the Association for Computational Linguistics (ACL).
  22. Mohammad, S. M. (2021). Sentiment Analysis: Automatically Detecting Valence, Emotions, and Other Affectual States from Text. http://arxiv.org/abs/2005.11882
  23. Rinker, T., Dame, U. of N., Technologies, D. of K., Unicode, Inc, Higgins, J., Ward, G., Possel, H., Mechura, M. B., Liu, B., Hu, M., Mohammad, S. M., Turney, P., Cambria, E., Poria, S., Bajpai, R., Schuller, B., SentiWordNet, Wu, L., … Malaescu, I. (2019). Lexicon: Lexicons for Text Analysis (Version 1.2.1). https://CRAN.R-project.org/package=lexicon
  24. Rosenthal, S., Nakov, P., Kiritchenko, S., Mohammad, S. M., Ritter, A., & Stoyanov, V. SemEval-2015 Task 10: Sentiment Analysis in Twitter. 13.
  25. Rosenthal, S., Nakov, P., Kiritchenko, S., Mohammad, S., Ritter, A., & Stoyanov, V. (2015). SemEval-2015 Task 10: Sentiment Analysis in Twitter. Proceedings of the 9th International Workshop on Semantic Evaluation (SemEval 2015), 451–463. http://www.aclweb.org/anthology/S15-2078
  26. Hopp, F. R., Fisher, J. T., Cornell, D., Huskey, R., & Weber, R. (2021). The Extended Moral Foundations Dictionary (eMFD): Development and Applications of a Crowd-Sourced Approach to Extracting Moral Intuitions from Text. Behavior Research Methods, 53(1), 232–246. https://doi.org/10.3758/s13428-020-01433-0
  27. Graham, J., Nosek, B. A., Haidt, J., Iyer, R., Koleva, S., & Ditto, P. H. (2011). Mapping the Moral Domain. Journal of Personality and Social Psychology, 101(2), 366–385. https://doi.org/10.1037/a0021847