EDMO Ireland Stance Classifier

An application that classifies the stance of a reply to a rumour-introducing source tweet. The classifier is trained to adapt to the rumours and topics that are of interest to Irish citizens and stakeholders.

The classifier was trained by firstly fine-tuning the BERTweet-large model with RumourEval 2019 training data. We continued fine-tuning the model with synthetic Twitter data in the Irish domain (generated using LLaMA2-chat (13b)), and then optimised it under a self-training framework with unlabelled real-world twitter conversations related to rumours that were fact-checked by The Journal FactCheck (a signatory of the International Fact-Checking Network).

The four possible stance categories are:

  • Support: The author of the reply supports the source tweet.
  • Deny: The author of the response disagrees with the source tweet.
  • Query: The author of the response asks for additional evidence in relation to the source tweet.
  • Comment: The author of the response makes their own comment without a clear stance towards the source tweet.

Although originally intended for processing threads of tweets, the classifier can be used with text from other sources (forum posts, YouTube comments etc.) as long as they are provided to the service in the right format. While the JSON object for a single tweet contains a lot of information the current version of the stance classifier only uses the text of the tweet. This means it is easy to generate compatible JSON from other sources.

As with tweets each comment should be containted within a single JSON object. Each JOSN object must contain the fields text and id_str which hold the text of the comment and a unique ID respectively. A third field in_reply_to_status_id_str is used to signifiy which other comment this one is in reply to. A simple example of one comment and a reply that is classified as supportive would be as follows:

{"text":"this is the original post", "id_str":"1"}

{"text":"I agree with the original post", "id_str":"2", "in_reply_to_status_id_str":"1"}

Note that you can provide as many tweets as you like (multiple replies to a single tweet, or multiple tweet/reply pairs etc.). If using the REST API documents should be sent as text/plain to avoid the JSON being parsed by GATE Cloud prior to the application running.

Default annotations
:TweetStance Each reply (for which the original tweet is supplied) is annotated by the application. Each annotation has the following features:
  • tweet_id, the ID of this tweet
  • in_reply_to, the ID of the tweet this one is in reply to
  • stance_class, the assigned stance class (one of support, deny, query, or comment)
  • support-score, the support score (all four scores add to 1)
  • deny-score, the deny score (all four scores add to 1)
  • query-score, the query score (all four scores add to 1)
  • comment-score, the comment score (all four scores add to 1)
1,200 free requests / day
Larger batches GBP0.80 / CPU hour

Use this pipeline

Single documents

You can process up to 1,200 documents per day free of charge using the REST API, at an average rate of 2 documents/sec. Higher quotas are available for research users by arrangement, contact us for details.

The API endpoint for this pipeline is:

https://cloud-api.gate.ac.uk/process/stance-edmo-ireland

Create API Key

Batches of documents

You can process any amount of data with this pipeline on a pay-as-you-go basis, for GBP0.80 per hour. This can be data you upload yourself, data you collected from Twitter, or the results of a previous job.

Reserve a job