IKANGAI Solutions. e.u.

After our Amamzon Mechanical Turk experiment, we took a closer look at the data. As discussed earlier, turkers had some troubles understanding the meaning of Tweetflows, due to the lack of an explanation. Omitting a detailed description was a deliberate choice: we wanted to know, if turkers were able to create Tweetflows with a minimal set of examples. We did not explain that Tweetflows are designed to be machine-readable and we did not discuss the structure of Tweetflow commands which can be separated into four distinct parts:

prefix – defining the type of command (e.g., service request, service provision)
command – consisting of an auxiliary verb, a verb and a noun
link to a resource
optional (semantic) suffix – providing additional information encoded as hashtags that allow for classification of the request

However, to facilitate the translation, we chose Tweetflow commands that turkers could easily derive from the examples. One interesting aspect is the use of hashtags, which lead to some confusion of the turkers:

I was not sure how to incorporate all of the hashtags

The same symbol # is used as different word to connect the words to form sentences.The commanding person may think to communicate different sentence and the receiving person may assume different sentence by using different words in place of # symbol to form sentence.It may lead to misunderstanding and communication gap.

I don’t know the exact prepositional phrases to put in front of #QRCode #Scanner #iPhone.

it has # in the tweet and I don’t know what it means by #

it was a little confusing to comprehend the meaning of the words after the “#” expression. I am uncertain whether I translated the meanng correctly but it did seem to make sense at the time.

As solution to this problem, some of the turkers just included the nouns into the Tweetflow commands or added the hashtags to the resource link:

ikangai, can you publish the description of http://www.ikangai.com//blog with QR Code of Scanner and iphone after johannes2112 completed the proofread of http://www.ikangai.com//blog with QR Code of Scanner and iphone ?

johannes2112, can you proofread the descriptions for QRCode, Scanner, and iPhone after ikangai has completed publishing? (see: http://www.ikangai.com/)

ikangai, will you publish the description (see: http://www.ikangai.com/QRCode/Scanner/iPhone)? johannes2112, can you proofread the description (see: http://www.ikangai.com/QRCode/Scanner/iPhone)?

One turker just included the hashtags in the natural sentence:

Ikangai, please publish the description (See: http://www.ikangai.com//apps/qcard-digital-business-card/ | SR) johannes2112, can you make a proof reading on the description on http://www.ikangai.com/ #QRCode #Scanner #iPhone]

It seems that we did not include enough meta data into the Tweeflow commands to give a deeper understanding. None of the turkers was able to combine the meta information #QRCode #Scanner #iPhone into a QRCode Scanner App for the iPhone. Obviously, we need to include more context information to point the user into the right direction.

We are thinking of structuring meta information so that people can make use of it. We are considering the use of curly braces to add key value pairs to Tweetflow commands:

{key=#value}

However, we are not sure, if this would add additional confusion to the syntax of Tweetflows. This requires obviously an additional experiment with Tweetflows on Amazon Mechanical Turk.

your ikangai science team

Our Tweetflow experiment is completed: we’ve received results from all 40 turkers. At a first look, it seems that Tweetflows are indeed human readable – all workers completed their tasks and the resulting Tweetflows look good. However, some of the comments are very interesting, especially if we consider the setting in which the Tweetflows were translated into natural language. We only provided a small set of examples, so the turkers had a starting point, but we offered no further explanations.

24 out of 40 turkers had problems with the task, and we received comments like these:

you get a general idea of the meaning but not a complete understanding. Not everyone would find these tweetflows easy to comprehend. They are not user friendly for all technology experience levels.

This is an absurd, mechanical language, that can often require more characters and cause greater confusion than natural English. Unless these tweets are being parsed by an app, I don’t see the point.

it had a lot of elements and these elements can be arranged in different ways and produce different meanings each time.

Once I figured out the syntax I was able to get the gist of the command but it took a little time to figure it out.

Much of the syntax is foreign to me. I had to guess based on the examples on this page.

A little difficult determining which parts of a sentence structure each part of the tweet referred to (i.e. subject, object)

The hashtag keywords aren’t together with the sentence part.

I’m not familiar with that syntax or programming syntax in general.

The first lesson that we learned from this experiment is that there is probably no such a thing as “natural programming language” besides natural language itself. The second lesson is that Tweetflows are indeed human readable, even if the syntax seems arbitrary for non technical people. However, this did not hinder the turkers from creating correct Tweetflows.

your ikangai science team

We posted a Amazon Mechanical Turk task (a so-called human intelligence task, HIT) a week ago and so far we received 7 the results of seven HITs (out of 50). Actually – based on past experience – we thought that we would have had attracted more workers (so-called Turkers) by now and expected to have a larger number of our HITs completed by now.
As a consequence, we posted a new version of the task (we also pay more ) and added additional things to do for the Turkers to do:

Tweetflows are a simple syntax that are used to ask users on Twitter to perform certain activities.

• Task 1: Translate SR @ikangai doProofread.Blogentry into an English sentence:

Example: SR @ikangai doWrite.Paper #Tweetflows #language-syntax tranlates into:

ikangai, can you write a Paper about the language-syntax of Tweetflows?

• Task 2: Translate SP @cerridan proofread.Blogentry into an English sentence:

Example: SP @ikangai write.Paper #Tweetflows #language-syntax tranlates into:

ikangai, I can write a Paper about the language-syntax of Tweetflows.

• Task 3: Translate the following four Tweetflow commands into four English sentences:

SR update.qcard http://www.ikangai.com//apps/qcard
SP @ikangai canUpdate.qcard http://www.ikangai.com//apps/qcard
SR @ikangai doUpdate.qcard http://www.ikangai.com//apps/qcard
SP @cerridan didUpdate.qcard http://www.ikangai.com//apps/qcard

• Task 4: Translate the following sequence into two English sentences:

[SR @ikangai doPublish.Description http://www.ikangai.com/ #QRCode #Scanner #iPhone | SR @johannes2112 proofread.Description http://www.ikangai.com/ #QRCode #Scanner #iPhone]

Example: [SR @ikangai doUpdate.Description http://www.ikangai.com//apps/qcard-digital-business-card/ | SR @cerridan doAdd.Paragraph http://www.ikangai.com//apps/qcard-digital-business-card/]  translates into:

ikangai, can you update the description (see: http://www.ikangai.com//apps/qcard-digital-business-card/)? cerridan, can you add a paragraph after ikangai has completed (see: http://www.ikangai.com//apps/qcard-digital-business-card/)?

• Task 5: Answer the following four questions:

1. Have you heard of Tweetflows before?

2. Did you find it difficult to get the menaing of Tweetflow commands like SR @ikangai doPublish.Description http://www.ikangai.com/ #QRCode #Scanner #iPhone ?

 3. Did you search the Web for Tweetflows?

4. If you searched the Web for Tweetflows, did you find the Tweetflow specification at the ikangai Web page http://www.ikangai.com//tweetflows-specification-version-1-0/ ?

your ikangai science team