Matt Dickman at Techno//Marketer has recently been posting about how sentiment is determined in online conversations while pointing out the difficulty posed with language and mentions Collective Intellect.

“Part of their analysis is of language within conversations and the sentiment that is displayed there. The sentiment is then tracked over time and can be a key metric in the success of a campaign. Their formula for extracting the sentiment is not publicly accessible so I am not sure how they calculate it.”

His question prompted our own response to explain how we determine sentiment. What follows is our current working approach hot from those that know.

Sentiment = The Collective Intellect tonality algorithm performs linguistic and statistical analysis on each post to determine its overall sentiment (positive, negative, or neutral). The CI algorithm is designed to operate effectively across a broad range of domain areas. The algorithm has been tested on several standard sentiment datasets (such as movie reviews) and consistently performs at a level close to human inter-rater accuracy.

And if you are wondering what human inter-rater accuracy is, here is that definition.

Human inter-rater accuracy (really correlation) is a measure of how well a set of independent raters correlate on rating a test set. For example, for a particular test set, if the sentiment ratings of a set of 4 human raters only correlate at 70% (on average) then that is the best accuracy level any automated system could hope to attain on that test set.

Clearly, there is a lot more to discover here, but that is a good reason why you should stay in touch with the social media strategy and activation experts at Collective Intellect.