Laughing Out Loud

This may make you laugh.

And laughs are something people like to share. When people communicate via social media, they type "laughs." In a sample of a million words of Twitter messages in ten different languages, I found that about 0.5% of all "words" are laughs – "haha", "LOL", or other ways of typing out a chuckle.

Do people everywhere laugh equally?

Not on your life.

In a study of ten Western languages (English, German, Dutch, Norwegian, Swedish, Danish, French, Spanish, Italian, and Portuguese), I found enormous differences in the frequency of Twitter-laughs.

The Germans laugh least, with Twitter laughs making up under 0.1% of all words.

Other languages of Northern Europe were somewhat more prone to laughs than German. In increasing order of laugh frequency, Norwegian, French, Swedish, English, and Danish all came in below 0.4%.

And then there are the happy Latins. Laughing just more than the Danes, Portuguese has 0.5% laughs, and that's nothing compared to the Italians who Twitter-laugh in 0.9% of words. But the runaway laugh champions are Spanish speakers who type Twitter laughs for 1.4% of words.

The North-South pattern is noteworthy, but is broken by the Dutch, who out-laugh their neighbors like they're misplaced Latins, finishing way up at 0.8%.

The Dutch withstanding, the North-South trend is sharp and undeniable, as this color-coded map makes clear.

What's even funnier, the languages where people laugh more often, they also type longer laughs.

While three "ha"s are the preferred laugh in Spanish and Italian, they are not afraid to laugh longer. The five-ha laugh ("jajajajaja") is more common in Spanish than the two-ha laugh is in German. This graph shows how laugh length occurs in the five most-spoken languages. While Spanish runs away with the championship here at every length, notice that English is actually the runner-up for the two-ha laugh ("haha") with Italian strongly preferring a three-ha approach ("ahahah").

When you take into account the length as well as frequency of laughs, Spanish Twitter has 24 times more laughing than German, as measured in character count. This is not a subtle difference!

So, why is all of this happening? It's clear that more Twitter laughs come from the warmer and sunnier countries.This is true not only in Europe but also in the Americas, where the most speakers of English, Spanish, and Portuguese live. Statistically speaking, the laugh statistic is highly correlated with the latitude of the corresponding European capital (farther south: r=0.66), how sunny that city is (more sun: r=0.74), and inversely with the suicide rate (r=-0.74; this is the same if you choose the U.S., Mexico, and Brazil instead of the U.K., Spain, and Portugal).

So is it as simple as this: Warm, sunny weather makes people laugh a lot and immune to depression?

That may be part of it. But another idea to consider is that in Germany and Scandinavia Twitter is used comparatively more often for business and relatively less often for chatting. When one subtracts the social chat, then naturally less laughter remains.

Overall, it's not clear how much Twitter reflects life as a whole. Until we plant microphones everywhere and monitor all human communication, studies like this will just be suggestive of larger truths. But insofar as it goes, this study of Twitter laughs serves to support a lot of existing cultural stereotypes.

Social Media: Linguistic Anarchy?

Your schoolteacher would be horrified. As technology opens up new channels for people to communicate via the written word, the use of language in those channels becomes increasingly ill-formed and deviant. Social critics may look at this as a relaxation in standards, a harbinger of the decay of reason and civilization.

However, it's really not that bad. People feel differently about standards in language, and one might observe that if language use did not vary over time, we would still be speaking Latin, Anglo-Saxon, Proto-Indo-European, or some more primeval language. Whether you identify more with prescriptive linguistics (the use of instruction to make students keep in line with existing standards) or descriptive linguistics (the laissez-faire study of language to understand it, without concern for changing how people use it), the truth is that people still adhere to standards, and in many ways, those standards aren't so different in the era of electronic media than they were in the long-lost era of pens and inkwells.

How does English, say, on Twitter, differ from English in the news? Certainly one sees slang, profanity, misspelling, and neologisms. But the single greatest set of differences come from the different kind of interaction. The news is meant to sound like the voice of God, detached, objective, hovering over the topic and the reader alike in the Third Person. In contrast, many interactions in social media are person-to-person, openly subjective, inherently spoken by the First Person to the Second Person... often on the topic of the First Person and/or the Second Person.

For ten major European languages, I have computed word frequencies for a corpus of news and of Twitter posts, and here I focus on the words which are the most prevalent on Twitter as compared to the news (frequency in Twitter minus frequency in the news). And for English, the word at the top of that list is nothing that would give schoolteachers and the clergy a stroke. It is "I." In fact, of the 39 words topping that list, only the acronym "LOL" and the abbreviation "u" are the stuff of which schoolteacher nightmares are made. The other 37 are almost exclusively words that are very common in The Queen's English and American Standard English, but happen to be more prevalent in first person narration than in the third person. They are common words that are natural descriptors of situated language, where the speaker's and listener's identity, time, place, attitude, and -- more generally -- their context, are part of the discussion.

And so, with an eye towards the top 25 words on the (Twitter-minus-news) frequency list, we see the following categories:

First-and-Second Person forms: I, am, me, my, we...

Deixis (words referring to the speaker's or listener's situation): today, just...

Simple, plainspoken vocabulary, words that are common in the news, but still more common for writers who are not consulting the thesaurus to make their language more flowery: not, was, do, had, did, have, got...

Far lower on the list come the shock words: "shit", "fuck", "alot", "alright", "ima", etc. And even the abbreviations are understandable, when writers are constrained to squeeze their idea into the 140 character limit, and deal with keypads that make extra effort a true burden. A drowning person is likely to yell for help in something other than complete sentences, and a person straining to fit an idea into Twitter's constraints has a legitimate motive for abbreviating more than they otherwise might. All told, we see that people have a strong tendency to hold to convention -- not the universal adherence to convention that William Safire would have liked, but it is still the most common case.

This is true in other languages as well, and to much the same degree. Here are, according to Twitter-minus-news frequency, the top 25 words for the five leading Western European languages. They reflect more or less the same tendencies.

English: I, 's, not, am, me, my, was, he, do, we, had, lol, news, did, u, have, new, today, just, think, haha, got, 'd, game, she.

French: je, j, c, pas, est, ai, tu, mais, moi, me, que, ça, a, t, mon, suis, on, ma, si, y, fait, il, te, quand, m.

German: ich, du, ja, d, nicht, aber, mir, mal, hab, was, jetzt, so, ist, noch, mich, da, dann, bin, es, schon, das, war, wenn, auch, dir.

Spanish: no, me, q, te, es, ya, si, yo, lo, jajaja, mi, tu, a, d, mas, pero, XD, jaja, jajajaja, eso, México, son, hay, solo, x.

Italian: non, mi, ho, io, ma, che, d, XD, è, ti, se, u, a, me, sono, lo, o, no, ci, l, ora, çç, sei, mia, poi.

Interestingly, the negative adverb in each language appears quite high. This seems to indicate that journalists exercise a discipline to express things in terms of positives while people generally use the negative a larger proportion of the time.

The big cross-language difference that is evident from the above is in the tendency for Twitter users to type out a "laugh", and this particularly stands out on the Spanish list. This merits a fun and funny, follow-up post on how much people type out laughs in different languages. The results will probably not surprise you.

Parsing Twitter

The Internet has not re-invented language as such, but it has created many new registers that have to be parsed as such. Out-of-the-box NLP tools that were developed to parse the Wall Street Journal and other well-behaved text will fall down flat if they are used to process other niches around the Internet.

I have seen some of these phenomena going back a quarter of a century, in online chat. In a nutshell, people use Internet means of writing in ways more colloquial than formal writing tends (and tended) to be. But even without that broad sweep, there are many sub-niches of usage -- some determined by medium, and some determined by user population. (Besides the obvious segmentation into different national languages like English, German, and Chinese.)

Interest in parsing Twitter is suddenly getting hot, and while a lot of the linguistic behavior there resembles linguistic behavior in other online locales like chat rooms, email, and instant messaging, every niche ends up with its own rules (and lack thereof).

Here are some phenomena I've seen as I build a parser that is robust enough to handle Twitter:

1) Pro drop. Twitter in particular makes the first-person singular pronoun implicit. Many tweets look like English sentences that have the leading word "I" implied. In other cases, "I am" is implied.

2) Nonsentential statements. Sometimes a noun phrase stands alone, with an implied existential quantification out front. "Party tonight" means "There will be a party tonight."

3) A register that resembles Black English Vernacular has arisen. I would suggest that this new written form deliberately deviates from formal written standards. At the same time, it is economical, using shorter forms as rebuses for bulkier forms whenever the shorter form would be pronounced the same way. For example, rewriting "You know" as "u no" (4 characters instead of 8). One can feel William Safire quaking, but for those of us writing parsers, we must accept and embrace.

The first I noticed this was in the titles of songs written by Prince. The titles of songs on his first three albums never did this, but in albums released in 1981 and 1982, three of his songs had these elements in his titles (eg, "I would die 4 u"). You can see the deliberately contrary nature of his language by 1988, when he titled a song "Eye No", thus using a longer form instead of a standard shorter form. I don't know if Prince was significantly responsible for this phenomenon or not, but it has certainly caught on by now.

Incidentally, detecting a user's register is potentially quite valuable, since many business purposes for parsing Twitter would be involved with market analysis and market segmentation.

4) Acronyms and emoticons. These are so common in computer-mediated communication that it is impossible to be unaware of them. LOL.

5) Novel contractions, like "hella", "tryna", "weneva".

6) Repeating characters to establish emphasis. Eg, "welcomeeeeeeeeee". This is in some cases a challenge to parse (in principle, "good" is "god" with the "o" repeated). In other cases, it's easy to convert to standard usage, but it does defeat literal search mechanisms.

Notice that the aforementioned devices can occur in combination. For example, "lmaaooo" = "laughing my ass off" with the "a" and "o" repeated for emphasis.

7) Unique medium-specific entities like URLs and the Twitter features for directing a tweet to a user (eg, @FakeSteveJobs) or a topic (eg, #lost).

8) The substitution of characters that resemble other characters for one another. "3" can be used for "E", "0" for "o", "q" for "g".

9) Deliberate swapping of character order. For example, "teh" as a playful misspelling of "the". This can also combine with aforementioned devices. Eg, "pr0n" is a way to rewrite "porn".

Not every user partakes of these new linguistic devices, but a parser that is intended to wring meaning (and market intelligence) out of Twitter (or blogs or email or other electronic communication) ignores them at its peril. The more of these you miss, the more information you miss. And the people who have embraced these nonstandard devices represent a nontrivial amount of spending power.