Wikipedia:Article Feedback Tool/Version 5/Feedback evaluation

This page is currently inactive and is retained for historical reference. AFT5 was removed from all Wikimedia wikis on March 3, 2014.

Shortcut

WP:AFT

Article Feedback

The Feedback Evaluation System with components labelled.

This page in a nutshell: A process and tool for evaluating the usefulness of feedback generated by the Article Feedback Tool v5 is summarized.

As part of the first phase of testing Article Feedback Tool v5 (AFTv5), the feedback produced by each of the proposed interface widgets will need to be judged for usefulness. This page summarizes the technical details and usage instructions of the Feedback Evaluation System (FES), a web interface that volunteer Wikipedians will use to evaluate the quality of feedback received via different versions of the AFTv5 feedback form. This result of this evaluation will help make the decision about which feedback form will be included in the final AFTv5 deployment.

What do you mean by "hand-coding"?

Although it may sound like it, volunteers will not be asked to do any computer programming. "Hand-coding", in qualitative research, refers to the process of evaluating and categorizing items by hand (and with human eyes), as opposed to using a computer or other machine to do the work. Hand-coding is often used when human judgement is necessary in order to perform a proper evaluation. In the case of feedback received via AFTv5, a computer algorithm is insufficient for determining the usefulness, so volunteer Wikipedians will need to perform the evaluation manually. Therefore, the Feedback Evaluation System (described below) has been constructed to make the process of manually categorizing AFTv5 feedback quick and easy.

The Feedback Evaluation System

FES is designed to quickly organize the information necessary for evaluating feedback into a coherent graphical user interface. The system breaks down to three major parts:

Completed list: A list of feedback items that have been evaluated (coded)
Feedback form: A web form that will capture the volunteers evaluation.
Available list: A list of feedback items that need evaluation.

Feedback items lists

These two lists surround the Feedback form and represent the feedback items that a volunteer has been asked to evaluate. When the interface first loads, the available list on the bottom should be full and the completed list on the top should be empty. As a volunteer evaluates feedback using the interface, the feedback items will move from the available list to the completed list. Once the available list is empty, the assigned set has been completed.

Information provided in the lists for all feedback items (id, title, username).

The horizontal bars in the two feedback items lists represent an individual feedback items that have been randomly selected for evaluation. The information included for each feedback items includes:

Id: the internal id of the feedback item
Title: the title of the page for which the feedback was submitted
- This is the page that will be loaded into the feedback form's article pane when the feedback item is selected.

Information provided for completed feedback items

For items that have already been evaluated, icons which summarize the evaluation are included:

which categories did the feedback item belong to?
- suggestion (s)
- question (q)
- issue (i)
- praise (p)
- abuse (!)
- irrelevant (?)
was the feedback useful?
- "yes" (✓)
- "no" (✗)

Any feedback item can be selected (whether completed or available) to be loaded into the feedback form by clicking on it. The feedback item can then be updated by changing the values in the feedback form and clicking "save" again.

Feedback form

The feedback form is the main mechanism of FES. The form consists of two components:

Article pane: a scroll-able pane containing the content of the article at the time that feedback was posted
Evaluation form: a header containing information about the feedback item (id & page title), the feedback itself, and two questions about the feedback.

When the FES interface first loads, the next available feedback item will automatically be loaded into the Feedback form. The revision of the article at which the feedback was submitted will be loaded into the article pane. When the user completes the form and saves the evaluation, the feedback item will automatically be moved from the available list to the completed list with an updated set of icons and the next available feedback item will be loaded into the form.

Help tooltip for "What is the intent" is extended in the feedback form.

The two questions in the form include helpful tooltips which can be access by moving the mouse over the icon.

Is this useful?

This question is a judgement call that Wikipedians will be the most qualified individuals to answer. Here, volunteers should use their best judgement, but think broadly of the possible usefulness of feedback to any editors. It is only the most entirely useless feedback that should be categorized as "no" (not useful). Additionally, feedback items where the evaluation is unclear can be marked as "unsure".

What's the intent?

Although it is a necessity in Wikipedia to assume good faith, Wikipedians tend to have a keen eye for the likely intentions of others. Volunteers should take the feedback in the context of the article and try to express the likely intentions of the user who left the feedback. Many of the possible intentions can be selected, or none if none apply.

Volunteering

The following users have volunteered to use FES to help evaluate the usefulness of feedback generated by AFT5:

Procedure

Each volunteer who adds their username above will be contacted by either EpochFail or Okeyes (WMF) to get an introduction to the FES system and make sure that his/her questions are answered. A volunteer will then be provided with a link to FES that will load around 30 minutes worth of work into FES. Once a volunteer has evaluated the provided feedback items, he/she may request another set of items to work on.