After our Amamzon Mechanical Turk experiment, we took a closer look at the data. As discussed earlier, turkers had some troubles understanding the meaning of Tweetflows, due to the lack of an explanation. Omitting a detailed description was a deliberate choice: we wanted to know, if turkers were able to create Tweetflows with a minimal set of examples. We did not explain that Tweetflows are designed to be machine-readable and we did not discuss the structure of Tweetflow commands which can be separated into four distinct parts:
prefix – defining the type of command (e.g., service request, service provision)
command – consisting of an auxiliary verb, a verb and a noun
link to a resource
optional (semantic) suffix – providing additional information encoded as hashtags that allow for classification of the request
However, to facilitate the translation, we chose Tweetflow commands that turkers could easily derive from the examples. One interesting aspect is the use of hashtags, which lead to some confusion of the turkers:
I was not sure how to incorporate all of the hashtags
The same symbol # is used as different word to connect the words to form sentences.The commanding person may think to communicate different sentence and the receiving person may assume different sentence by using different words in place of # symbol to form sentence.It may lead to misunderstanding and communication gap.
I don’t know the exact prepositional phrases to put in front of #QRCode #Scanner #iPhone.
it has # in the tweet and I don’t know what it means by #
it was a little confusing to comprehend the meaning of the words after the “#” expression. I am uncertain whether I translated the meanng correctly but it did seem to make sense at the time.
As solution to this problem, some of the turkers just included the nouns into the Tweetflow commands or added the hashtags to the resource link:
ikangai, can you publish the description of http://www.ikangai.com//blog with QR Code of Scanner and iphone after johannes2112 completed the proofread of http://www.ikangai.com//blog with QR Code of Scanner and iphone ?
johannes2112, can you proofread the descriptions for QRCode, Scanner, and iPhone after ikangai has completed publishing? (see: http://www.ikangai.com/)
ikangai, will you publish the description (see: http://www.ikangai.com/QRCode/Scanner/iPhone)? johannes2112, can you proofread the description (see: http://www.ikangai.com/QRCode/Scanner/iPhone)?
One turker just included the hashtags in the natural sentence:
Ikangai, please publish the description (See: http://www.ikangai.com//apps/qcard-digital-business-card/ | SR) johannes2112, can you make a proof reading on the description on http://www.ikangai.com/ #QRCode #Scanner #iPhone]
It seems that we did not include enough meta data into the Tweeflow commands to give a deeper understanding. None of the turkers was able to combine the meta information #QRCode #Scanner #iPhone into a QRCode Scanner App for the iPhone. Obviously, we need to include more context information to point the user into the right direction.
We are thinking of structuring meta information so that people can make use of it. We are considering the use of curly braces to add key value pairs to Tweetflow commands:
However, we are not sure, if this would add additional confusion to the syntax of Tweetflows. This requires obviously an additional experiment with Tweetflows on Amazon Mechanical Turk.
your ikangai science team