% sage_latex_guidelines.tex V1.20, 14 January 2017

%\documentclass[Afour,sageh,times]{sagej}

\documentclass[sageh,times]{sagej}


%\usepackage{natbib}
\usepackage{moreverb,url}
\usepackage{bibentry}


\usepackage{amsmath} 

\usepackage[colorlinks,bookmarksopen,bookmarksnumbered,citecolor=red,urlcolor=red]{hyperref}

\usepackage{graphicx}% Include figure files

\usepackage{dcolumn}% Align table columns on decimal point

\usepackage{bm}% bold math

\usepackage{xcolor}

\newcommand{\sw}[1]{\textbf{\color{blue} #1}} 
\newcommand{\kj}[1]{\textbf{\color{green} #1}} 
\newcommand{\ls}[1]{\textbf{\color{purple} #1}} 


\newcommand\BibTeX{{\rmfamily B\kern-.05em \textsc{i\kern-.025em b}\kern-.08em
T\kern-.1667em\lower.7ex\hbox{E}\kern-.125emX}}

\def\volumeyear{2019}

\begin{document}

\runninghead{Willoughby, Johnson and Sterman}

\author
{S. D. Willoughby\affilnum{1} K. Johnson\affilnum{1} and L. Sterman\affilnum{2}}

\affiliation{\affilnum{1}Department of Physics, Montana State University \\
\affilnum{2}Renne Library, Montana State University}

\corrauth{Shannon Willoughby,
Department of Physics,
264 Barnard Hall
Montana State University,
Bozeman, MT}

\email{shannon.willoughby@montana.edu}


\title{Quantifying Scientific Jargon}% Force line breaks with \\a

\date{\today}% It is always \today, today,
             %  but any date may be explicitly specified
% Double-space the manuscript.

\baselineskip24pt


\maketitle
\tableofcontents
 
\newpage

\section{Supplemental Material}

\section{\label{sec:methods}Methods}

\subsection{\label{sec:benchtext}{Benchmarking Texts}}

In order to create a baseline for jargonness values, we downloaded a large number of transcripts and texts from a variety of sources. From the Gutenberg project, we downloaded 20 classic texts, including \textit{Dracula}, \textit{Alice in Wonderland}, \textit{Pride and Prejudice}, and others (see full list below).  All were chosen using random number generation from the list of the top 100 downloads in Project Gutenberg in June 2019. One author, a subject expert in Children's literature, complied 20 of the most widely known Children's books from Project Gutenberg. For popular podcasts, we obtained transcripts from most recent 20 episodes (as of June 2019) of \textit{Hidden Brain}, a podcast which focuses on scientific topics and is intended for a general audience. All 95 Supreme Court decisions from 2010 were downloaded. Transcripts of the top 36 most-viewed TED talks were downloaded from their host website \citep{TEDtalks}.  For a sample of transcripts we suspect contain jargon, we downloaded 30 Material Safety Data Sheets. These were also chosen randomly.  All science-ebooks available from NASA, 24 in total, were obtained from their host website \citep{NasaEbooks}.  Finally, we chose 540 ArXiv publications spanning from 1999 to 2019. 


Texts used in bench marking: \\
\underline{Classic texts}
\begin{enumerate}
\item A Christmas Carol
\item Alice in Wonderland
\item Being Ernest
\item Doll's House
\item Dracula
\item Frankenstein
\item Hard Times
\item The Heart of Darkness
\item Meditations
\item Moby Dick
\item A Modest Proposal
\item Northanger
\item Pride and Prejudice
\item The Hounds of Baskerville
\item The Illiad
\item The Prophet
\item The Call of the Wild
\item A Tale of Two Cities
\item Ulysses
\item War of the Worlds

\end{enumerate}


\underline{Children's Literature}
\begin{enumerate}
\item A Little Princess
\item Anne of Green Gables
\item Black Beauty
\item Box Car Children
\item A Christmas Carol
\item Fairy Tales of Hans Christian Anderson
\item Fairy Tales
\item The Adventures of Huckleberry Finn
\item Little Women
\item Through the Looking Glass
\item Mother Goose
\item Peter Rabbit
\item Poems Every Child should Know
\item The Secret Garden
\item The Jungle Book
\item The Red Fairy
\item Tom Sawyer
\item Treasure Island
\item The Wind in the Willows
\item The Wizard of Oz
\end{enumerate}

\underline{Materials Safety Data Sheets}
\begin{enumerate}
\item Yttrium
\item Acetolyne
\item Acetone
\item Atrazine
\item Benzene
\item Bisphenol- A
\item Bromine
\item Chlorine
\item Copper
\item Dioxins
\item Fenchone
\item Fluorine
\item Hydrogen gas
\item Iron
\item Lead
\item Litium
\item Magnesium
\item Mercury
\item Octanol
\item Perchlorate
\item PFE
\item Phthaltes
\item Sulfur
\item Silver iodide
\item Sulfur dioxide
\item S1 Nuclease
\item T282
\item Thiazol
\item Bonide Sulfur
\item Molten Sulfur
\end{enumerate}





\subsection{\label{sec:calc}{Calculating Jargonness}}
In order to calculate the jargonness of a text file, the following algorithm is employed:

\begin{enumerate}
	\item Choose a set of texts to create a scientific corpus.
	\item Choose a set of texts to create a contemporary American English corpus.
	\item Create both corpora, with stop words, punctuation, numbers, and hyphens removed.
	\item Prepare the text file for which the jargon is to be calculated, remove punctuation, numbers, and hyphens. 
	\item Determine how many times each word in the cleaned text file occurs in each corpus.
	\item Calculate the jargonness value for each word.
	\item Calculate the mean jargonness per word for the text as a whole.
\end{enumerate}

The scientific corpus is generated from a collection of scientific journal articles. We chose a 5 year compilation of all journal articles published by Elsevier from year 2011 to year 2015 in the areas of the physical, life, health, and social sciences \citep{sci_corpus}. These articles have been converted to plain text, with the references and figures removed.  The full corpus consists of 5,686,428 words collected from 895 papers.  This corpus can easily be updated or made larger by the inclusion of ArXiv manuscripts.  Large volumes of these manuscripts can be downloaded free of charge, and either for specific domains, or by a range of dates they were posted. 

The American English corpus utilized in this study, the Open American National Corpus \citep{eng_corpus}, is a freely available and continually growing corpus of contemporary English from 1990 on.  The full corpus includes over 15 million words from travel guides, switchboard conversations, government websites, fiction, online news websites, and other audio and written sources. The English corpus serves an important role in the analysis because it is being used to represent words with which the average American is familiar. For sampling consistency, we use a randomly chosen subset of this corpus in our calculations so that the number of words in this corpus has the same order of magnitude as the words in the full scientific corpus.  The subset of words was chosen such that we sampled across the word lists available, and includes switchboard conversations, works of fiction, articles from Slate, travel books, letters, 911 reports, and government reports.  

In order to generate relevant information regarding each corpus (word counts/frequencies), we used a package in R called Quanteda \citep{quanteda}.  This package was created by Kenneth Benoit in order to analyze texts within R using natural language processing. Text files of various formats may be read into Quanteda (.txt, .pdf, .doc, etc.) and filtered for undesirable content (removal of stop words, punctuation, numbers, and hyphens).  Following the filtering process, a document frequency matrix (dfm) is generated, which contains the word counts for every word within the input text files.  Both corpora were individually read into Quanteda and a dfm was generated for each, with the aforementioned filtering criteria applied.  After this work was complete, the scientific corpus contained 1,868,785 words, and the American English corpus contained 1,460,998 words.  Next, the relevant text file for which jargon is to be calculated is read in.  Filtering criteria are again applied, and a dfm for that text file is generated.

The filtering criteria of the corpora included removing punctuation, numbers, hyphens, and stop words.  As raw numbers (e.g.\ 1, 7.36, etc.) and punctuation do not necessarily contain any form of jargon, they were removed. Hyphenated words such as ``self-contained" are broken into their respective segments (e.g. ``self" and ``contained") because we assume that interpretation of the compound word as a whole requires understanding of the components of that word.  Stop words are very commonly used words that are typically removed before text analysis is performed, such as `we', `him', `which', and `this'. The removal of these words reduces the overall dimensionality of the text without losing the overall meaning.  We assert that stop words should have a jargonness value of zero, and are thus filtered out of the scientific and English corpora.  

The stop words removed from the analysis may be found within R using the command \textit{stopwords (source = ``smart")}  with more information available on the Quanteda website \citep{quanteda}.  We also use a second source of stop words to ensure that an exhaustive list of stop words are included  \citep{12dict}.  Note that when calculating the jargonness of a particular input file (e.g. ``Moby Dick"), all the above criteria are still employed except the filtering of stop words, since this would impact metrics such as the mean jargonness per word in the file of interest. 

Using the list of words associated with the text file dfm, the frequency of occurrence for the $i$th unique word is obtained both within the text file ($f_i$) and within each corpus ($f_{i,sci}$ \& $f_{i,eng}$).  The formula is taken directly from \citep{jargon}. To calculate the jargonness value $j_i$ for the $i$th word, and the jargonness $J$ for the entire document, we use the following expressions:

$$ j_i = 
\begin{cases}
\log\left(\frac{f_{i,\text{sci}}}{f_{i,\text{eng}}}\right),& 0< f_{i,\text{sci}}<f_{i,\text{eng}}\\
3,              & f_{i,\text{eng}} = 0$ and $f_{i,\text{sci}}>0
\end{cases} $$

$$J = \sum_{i}{j_i \times f_i}$$

\noindent $J$ is then divided by the total number of unique words in the text file, which yields a mean jargonness per word for the file as a whole. 


\subsection{\label{sec:verify}{Verifying the code}}

We tested our R script on \textit{Moby Dick} and \textit{The Adventures of Huckleberry Finn}, comparing our findings against those of several online concordancers to further check that our algorithm was working correctly.  Specifically, we checked that the count of certain words within each text were similar to other published values.  The words were randomly chosen using the base R function \textit{sample}.  Prior to random selection the dfm was filtered such that only the top 100 words, sorted by their jargonness contribution (jargonness of the individual word times how often it occurs in the text) were considered for random sampling.  This approach was taken because those words would be most influential when determining the average jargonness per word for the text, and hence of greatest interest to this study.  Tables \ref{tab:testsI} and \ref{tab:testsII} display the values we obtained compared to those available through online resources for the aforementioned texts.  This check provides a base calibration of our script.

Within Tables \ref{tab:testsI} and \ref{tab:testsII} we provide two values for bench marking purposes.  The first value is the frequency of occurrence for that exact word.  The second value is the frequency of occurrence for both that exact word as well as any set of characters which contain the exact word, also known as a wild card.  For example, the second value would not only contain counts of the exact word ``light", but also ``lightning", ``skylight", ``flighty", etc.  The concordancer website only provides counts for the exact word and hence contains only a single value.

Inspection of the tables reveals that not all sources agree on the number of specific terms in the texts.  Certainly there is a correct answer, given parameters as noted above, and our script finds numbers that are similar to the numbers reported by other sources.  The largest discrepancy noted in the compared words comes from ``mast" within Table \ref{tab:testsI}, where our code extracts 28 exact occurrences of ``mast" compared to the first website's 224.  This difference results primarily from our script separating hyphenated words and the website's choice to retain them, given that when our script is run without separating hyphenated words it is found that there are 224 occurrences of ``mast", in agreement with the website. We do not know why the second website seems to miss 99 entries of the word mast, but it is likely due to words that contain mast. Additional word-by-word comparisons of the hyphenated word forms and their counts further supported this. The same behavior is seen with ``oil", as our script run without separating hyphenated words revealed 78 occurrences of ``oil", which is also in exact agreement with the website. This analysis provides an easily replicable test of the word count functionality and flexibility of our script. 
\begin{table}
	\caption{\label{tab:testsI}Counts for various words within \textit{Moby Dick}, as determined by the jargonness code (WJS) and sources found elsewhere.  In the WJS column, the leftmost value is a count for the word only (e.g. mast), the accompanying value is for both the word and various other forms (e.g. mastless).}
%	\begin{ruledtabular}
		\begin{tabular}{|c|c|c|c|}
\hline
			word   &   WJS script    & Website \citep{whalewords} & Concordancer \citep{classiccordance} \\ \hline
\hline
			vessel  &  46, 76 &  81  &   51   \\
\hline
			species  &  31, 31 &  31 &   30   \\
\hline
			oil    &78, 212 & 212  &   86   \\
\hline
			mast   &28, 224 & 224  &  125   \\
\hline
			interval &  25, 57 &   57  &   25   \\
\hline
			sperm   & 239, 270&  270 &  237 \\
\hline
		\end{tabular}
%	\end{ruledtabular}
\end{table}


\begin{table}
	\caption{\label{tab:testsII}Tests of code versus word counts found elsewhere for \textit{The Adventures of Huckleberry Finn}.  The leftmost value is a count for the word only (i.e. light), the accompanying value is for the word and a wild card (*light*).}
%	\begin{ruledtabular}
		\begin{tabular}{|c|c|c|c|}
\hline
			word   &   WJS script   & Website \citep{twaincordance} & Concordancer  \citep{classiccordance}  \\ \hline
\hline
			current & 22, 23  &  21, 22   &       21     \\
\hline
			light  & 50, 130&  50, 130  &       52       \\
\hline
			doan   & 30, 30 &  34, 35   &       35       \\
\hline
			tin   & 25, 327&  26, 249  &       25       \\
\hline
			den  & 26, 99 & 27, 96   &       28 \\
\hline
		\end{tabular}
%	\end{ruledtabular}
\end{table}




The jargonness calculations and primary script used in the body of this paper were created and performed by one of the authors.  Independently of this, another author generated their own methodology for calculating jargonness, as defined by this study. The only information shared between these two authors were the files from which both the scientific and English corpora were developed, the Quanteda framework, and the filtering criteria utilized within Quanteda; the purpose of independent coding was to ensure the methodology was robust and consistent between coders. The details for this second approach to calculating jargonness were notably different, with the general algorithm as follows:  

\begin{enumerate}
	\item Choose a set of texts to create a scientific corpus.
	\item Choose a set of texts to create a contemporary American English corpus.
	\item Create both corpora, with stop words, punctuation, numbers, and hyphens removed.
	\item Create a file containing the jargonness value for each word in the scientific corpus.  
	Words not in the scientific corpus have, by definition (see Section \ref{sec:calc}), a jargonness value of 0.   
	\item Prepare the text file whose jargon is to be calculated, removing punctuation, numbers, and hyphens.
	\item From the jargonness file, retrieve the jargonness value for each word within the text file.
	\item Calculate the mean jargonness per word for the text as a whole.
\end{enumerate}  

For all calculations done within this paper, it was found that both methodologies and associated scripts yielded the same values.


\nobibliography{jargonness}
\bibliographystyle{SageH}


\end{document}

