Evaluation: Second User-Designer Session

A Cooperative evaluation with Paul using new device (scanning only):

Paul is given the device and a sheet of paper on which is printed an extract from Alice in Wonderland.

Designer: OK Paul, could you please scan the fifth word on the page.

Paul holds the scan button down and scans the word. He uses both hands to operate it as before. Paul clearly benefits from his previous experience with the device and scans the word correctly first time.

Paul: "It says rabbit"

Designer: "OK Paul, now because you are left handed we need to change the screen so it's the right way up for you. See if you can find the setting on your own."

Paul scrolls through the menu and find the setting, then changes it.

Paul: "That's better, I can hold it the right way now."

Paul reads out the word and its definition.

Designer: "Do you like this new device, is it more comfortable?"
Paul: "Yes, it's much easier to hold, it doesn't slide about as much as before."
Designer:"OK just like last time, I want you to just scan some words."

Paul scans various other words without problem, the novelty begins to wear off and Paul doesn't seem to be enjoying the device like he did before. He turns the sound on and laughs so more at the voice. Paul scans a word and looks confused.

Designer: "What's the matter Paul?"
Paul: "I don't understand this definition."

Paul reads out the definition and the designer explains the word.

Results

For Paul the novelty of the device is beginning to wear off, this isn't to say that the device isn't useful.

Screen inversion in the left-handed position is now solved and Paul was happy with the new settings.

The new comfort grips make the device easier to hold, an improvement on the previous model. The problem of complicated definitions is one which cannot easily be tackled. This shows that while the device is an excellent learning aid it is not a substitute for a parent, guardian or teacher.

Evaluation: Second Cognitive Walkthrough

Here I will perform another Cognitive Walkthrough of the word scanning device based on our new prototype that has been developed. The cognitive walkthrough will be of the same structure as the previous evaluation, which was based on our use cases for the device.

When evaluating each use case I will only talk about the changes that have been made to the device and how they effect each use case. The rest of the evaluation will be the same as the previous cognitive walkthough that was done on the first prototype.


The task:
To scan a word.

Actions required:

1. Find a word to scan with the device.
2. Place the head of the device on the first space on the left.
3. Press and hold the scan button and a beep will be made(if sound is on)
4. Move the pen slowly to the right across the word.
5. When the head of the pen has reached the first space after the word, release the scan button, a beep sound will be made again to indicate the scan has finished.

Evaluation:

The conclusions made from this walkthough are the same as before except that once a scan has been performed there is an indication on the LCD screen whether the scan was successful by showing the user the word scanned, or was unsuccessful producing an error message. It is now more clearer to the user when a scan has started and finished due to the built in 'beep' sound that is produced when the scan is about to commence and when it has ended after being passed over a word. However this sound is only produced if the volume is turned up and is not set to 0.

Scanning a word is now also easier to perform by the user as the controls are now accessible by both left handed and right handed people. This was an issue on the first prototype as the controls were awkward to get at if the user was of a left handed nature. This functionality should come naturally to the user as the option to flip the screen round for the correct orientation is one of the main options on the main menu system and is accessed in the same way as the sound, dictionary or thesaurus. The option to flip the screen is clearly indicated with the words "Flip Screen" on the menu, so there shouldn't be any confusion as to what this is, or what it does.

The task:
To access the dictionary definition after a successful scan.

Actions required:

1. Hold the device horizontally.
2. Use the scroll button beside the screen and move the scroll button down. This in turn will move the highlighted menu entry down.
3. Scroll down to ‘Dictionary definition’ on the menu.
4. Select the ‘Dictionary definition’ by pressing the scroll button in.
   

Evaluation:

This walkthough is mostly the same as the one made on the previous prototype. The main differences are that the wording in the main menu has been improved to make it completely clear that the resulting action will give the user a dictionary definition rather than a 'Dictionary'. This should make it more obvious to the user that when the 'Dictionary definition' is selected from the main menu, they should expect a definition of the scanned in word as this is still always printed on the top row of the screen, which gives an indication that the definition will be of this word.

The task:
To access the thesaurus entries for the word after a successful scan.

Actions required:

1. Hold the device horizontally.
2. Use the scroll button beside the screen and move the scroll button down. This in turn will move the highlighted menu entry down.
3. Scroll down to ‘Thesaurus entries’ on the menu.
4. Select the ‘Thesaurus entries’ by pressing the scroll button in.
   

Evaluation:

This use case is also has a similar evaluation to the previous prototype. The only difference is that of a similar nature to the dictionary use case, the wording used in the main menu of the word scanner has now been improved to clear up some confusion to the user that was presented when the word 'Thesaurus' was an option on the main menu. Now the menu has 'Thesaurus entries' in its place which should make it more obvious to the user that selecting this option will present further alternative words based on the scanned in word. As this is always shown at the top of the screen as before.

Evaluation: Second Heuristic Evaluation

Adapted Generic principles for heuristic evaluation:

1. Feedback: The child should be aware of what is happening.
2. Everyday child language: Use language that a child would be comfortable with.
3. Consistency: Responses should be the same when similar options are selected by the child and it should share conventions currently in use.
4. Recognition not recall: The child should be able to recognise what they should/can do at a particular step.
5. Simple design: Keep things crisp and simple and aesthetically pleasing.
6. Error recovery: Prevent errors from occurring, but if they do, provide a suitable response and an appropriate solution.
7. Documentation: Any documentation should be simple and concise, as a child will not read it.

General scope of the system:

The system displays a 'scanning' message when the Scan button is pressed. This feedback makes it obvious what is happening. The scanner now beeps when a word has been scanned correctly, thus telling the user that they can now use the menu system on the word.

The language the system uses is simple and easy to understand for children between 7 and 11, which is especially important for the Dictionary definitions and the Thesaurus entries.

Everytime a word is scanned, the options in the menu remain the same. This ensures that the system is consistant. The system also shares conventions found in devices such as pocket translators/personal organisers and mobile phones, where you scroll through the menus.

The design of the menus ensure that the next step is easy to recognise. It has been made easier in the redesign as the menus have more precise headings e.g. instead of Dictionary it now says Dictionary definition, so the child doesn't expect a full dictionary when they click on it. Furthermore the addition of the beep sounds when the user scans a word, helps to enforce the recognition of what to do next.

The word scanner provides only two input controls, namely the scan button and the clickable scroll wheel. This makes interaction simple. The design of the menus is also crisp and simple, avoiding any unecessary clutter. Futhermore, the new design is more comfortable to use with the addition of grips to the sides of the device, this will make using it easier.

The system displays an error message when a word failed to scan and also tells the user what to do next, so that they can recover from what they did.

The documentation is short and concise and should only really be required when the user first attempts to scan a word.


Specific interface elements:

The new size of the device may effect its 'pocket sized' status. It can still fit inside an adult's pocket, but a child's pocket is smaller. However, the increase in size is minimal, so it is unlikely to effect it.

There is still a concern that the definitions and entries offered by the Dictionary and Thesaurus may be too difficult for the child to understand. This would be an ongoing problem with this device, no matter how many prototypes were developed. It would simply need to improve with every version of the product ever released AND it can be developed by the updates the device will receive when plugged into a PC with Internet Access.

Evaluation - Re-Questionnaire Analysis

From the second round of questionnaires, it became quite obvious that the changes we made to the design of the product were a roaring success with the children testing them.

The changes to the ergonomic design were one of the main things picked up on by the test subjects, with added comfort in terms of grip, and ease of holding being cited as major points.

By allowing the children to reverse the text on the screen, it has enabled us to create a single device for both dexterities, which would allow production costs to be greatly cut for the single item. Other comments on the screen included praising the increased size as it allowed the text to be larger and clearer.

There were no real further suggestions from the test subjects - a request to read Italian words is less of a design issue, and simply would involve changing to a different language dictionary/theasurus file in the device.

The only further idea raised from the questionnaires was the possibility of releasing the products in a range of colours or other designs, but this is down to visual preference rather than design.

On the whole, I believe the re-design was a great success as it fulfilled all the points raised about the original prototype.

Evaluation: User-Designer

A Cooperative evaluation with Paul (scanning only):

Paul is given the device and a sheet of paper on which is printed an extract from Peter Pan.

Designer: OK Paul, could you please scan the third word on the page.

Paul presses the 'Scan' button, releases it and drags the device across the third word. Paul sees that nothing has happened on screen.

Paul: Which button should I press?
Designer: You need to hold the button while you drag it along.

Paul hold the device in the left hand and presses the 'Scan' button down with his right index finger, he then draws the device over the word.

Designer: Why are you using both hands?
Paul: I don't know, I think it's easier.

An error occurs.

Paul: It's not working.
Designer: Umm, make sure you go from the space before the word right up to the space after the word. You can use the light as a guide.

Paul re-scans the word correctly and turns the device around.

Paul: It's working! It says Dictionary and then Mother."

Paul reads out the definition. The designer realises that Paul has turned the device around because from the left handed position the screen is inverted.

Designer: Would it be better is the screen was the other way up?
Paul: Yeah.
Designer: Play around with it, scan some more words of your choice.

Paul scans various other words without problem, reading each definition aloud and looking more excited with each word correctly scanned.

Designer: You can get it to read the words out for you if you like, try turning the sound on.

Paul looks for the sound option and, after locating it, enables it. It takes around 30 seconds. Paul scans another words and listens to the device, he laughs.

Paul: It sounds like my friends mum!

Results

The device is clearly quite easy and exciting to use once you know how. For some it may not be obvious that the scan button must be held down and not just pressed. To solve this problem perhaps an error screen with an animation showing how to scan might be considered.

Screen inversion in the left-handed position is clearly a problem, this could be tackled by having a left handed mode which when enabled causes the screen to be inverted.

Users will each hold the device in their own way. Paul's two handed approach was not something considered during the design stage, this does not seem to be a problem though as Paul was quite happy with his own method.

Evaluation: Cognitive Walkthrough

Another way to evaluate the word scanner prototype is to do a cognitive walkthrough. In this process a series of steps are followed which a potential user would have to do to carry out an action on the word scanner. These steps can then be evaluated to see where possible problems may lie.

A cognitive walkthrough can be used to indicate issues that may arise when a user tries to learn how to use the device without the need of documentation.

Here is my adapted version of the cognitive walkthrough for the main use cases of the word-scanning device with the main users being children from 7-11 year olds. The guide I am using can be found here

The task:
To scan a word.

Actions required:

1. Find a word to scan with the device.
2. Place the head of the device on the first space on the left.
3. Press and hold the scan button.
4. Move the pen slowly to the right across the word.
5. When the head of the pen has reached the first space after the word, release the scan button.

Evaluation:

If the user wanted to scan a word with the device, then the sequence of actions will be required based on our prototype. The scan button has the word ‘scan’ written across it so the targeted user can easily identify this button. Pressing this button has the direct action of starting the scanning device. A user may not initially think that the button has to be held down during the scanning process and may assume you just press it once to start a scan.

The main control for this case is the ‘scan’ button, which is clearly visible and brightly coloured on the device. It is the only big button on the device and is clearly labelled. As the device is a word scanner the word ‘scan’ on the main control button should indicate to a user that this is the control to activate the scanner.

Once scan has been performed there is an indication on the LCD screen whether the scan was successful by showing the user the word scanned, or was unsuccessful producing an error message. It is unclear to the user when the scanning of a word is starting or has finished. The only feedback given back to the user is en error message or a scanned word on the LCD screen when finished.


The task:
To access the dictionary definition after a successful scan.

Actions required:

1. Hold the device horizontally.
2. Use the scroll button beside the screen and move the scroll button down. This in turn will move the highlighted menu entry down.
3. Scroll down to ‘Dictionary’ on the menu.
4. Select the ‘Dictionary’ by pressing the scroll button in.

Evaluation:

The actions for this case will only be required if the user wants to access the definition of the word they have just scanned into the device. The dictionary option in the main menu is only available after a successfully scanned in word.

The user should be able to see the menu item clearly in the menu. The main reason for this is because only 2 items of the menu are shown at any one time as the user scrolls through them due to the design of the LCD screens interface. There are also only a few items on the menu system. The scroll button on the device is very similar to the scroll button on a standard computer mouse. At first the user may not understand the scroll buttons controls to get through the menus or run though definitions, but after playing with the device it should become clear to the user especially if they use the scroll button on a mouse.

If the user is trying to get the definition of the word they have scanned, there could be slight confusion as to what the resulting action will be, revealing the definition of the word. As the menu system only says ‘Dictionary’ rather than clarifying that there will be only the definition of the scanned in word.

When the user selects the dictionary after a scan, it should be clear that the feedback given on the screen is the definition of the word. It is also useful to note that the scanned word is always at the top of the screen so the user always knows what the definition relates to.

The task:
To access the thesaurus entries for the word after a successful scan.

Actions required:

1. Hold the device horizontally.
2. Use the scroll button beside the screen and move the scroll button down. This in turn will move the highlighted menu entry down.
3. Scroll down to ‘Thesaurus’ on the menu.
4. Select the ‘Thesaurus’ by pressing the scroll button in.

Evaluation:

This task has similar actions to obtaining the dictionary definition of a scanned in word. The actions for this case will only be required if the user wants to access the thesaurus entries of the word they have just scanned into the device. The thesaurus option in the main menu is only available after a successfully scanned in word.

The user should be able to see the menu item clearly in the menu for the same reasons as the dictionary definition explained above.

If the user is trying to get the thesaurus entries of a word they have scanned, there could be slight confusion as to what the resulting action will be, displaying a list of alternative words. As the menu system only says ‘Thesaurus’ rather than clarifying that there will only be alternative words displayed for the scanned in word.

When the user selects the thesaurus after a scan, it should be clear that the feedback given on the screen is alternative words that can be used instead of the word that was scanned into the device.

Evaluation: Heuristic Evaluation

Adapted Generic principles for heuristic evaluation:

1. Feedback: The child should be aware of what is happening.
2. Everyday child language: Use language that a child would be comfortable with.
3. Consistency: Responses should be the same when similar options are selected by the child and it should share conventions currently in use.
4. Recognition not recall: The child should be able to recognise what they should/can do at a particular step.
5. Simple design: Keep things crisp and simple and aesthetically pleasing.
6. Error recovery: Prevent errors from occurring, but if they do, provide a suitable response and an appropriate solution.
7. Documentation: Any documentation should be simple and concise, as a child will not read it.

General scope of the system:

The system displays a 'scanning' message when the Scan button is pressed. This feedback makes it obvious what is happening. Perhaps a prompt is needed after the word is scanned so the child realises that they can now select the functions that interests them.

The language the system uses is simple and easy to understand for children between 7 and 11, which is especially important for the Dictionary definitions and the Thesaurus entries.

Everytime a word is scanned, the options in the menu remain the same. This ensures that the system is consistant. The system also shares conventions found in devices such as pocket translators/personal organisers and mobile phones, where you scroll through the menus.

As I mentioned under feedback, after a word is scanned the system should prompt the user to select a menu or make it clear that is what is required next. However, due to the system's consistancy and simple design, the next step is easy to recognise.

The word scanner provides only two input controls, namely the scan button and the clickable scroll wheel. This makes interaction simple. The design of the menus is also crisp and simple, avoiding any unecessary clutter.

The system displays an error message when a word failed to scan and also tells the user what to do next, so that they can recover from what they did.

The documentation is short and concise and should only really be required when the user first attempts to scan a word.


Specific interface elements:

The system should prompt the user to select one of the menu options after a scan.

On the menu, the words Dictionary and Thesaurus are ambiguous, they suggest that by clicking them you will be presented with a full system of words/definitions/entries, but this is not the case. They need to be made more specific like Dictionary definition and Thesaurus entries.

Principles from Russell's list that we eliminated:

Undo: The team decided to eliminate this principle. For the word scanner a possible undo function could have be provided for when the user clicks scan after scanning a word, thus losing the word from the device. However, the time it takes to rescan a word is trivial and so the function is not required and for children, adding it may make things more complicated.

Expert use: The word scanner does not have any place for shortcuts or accelerators of any kind, due to the simple scroll interface that the team designed.

Evaluation: Critical Appraisal

Our first prototype built on ours designs is around 8cm long and this feels a bit short in for my hand, obviously this product is aimed at users with smaller hands. The button layout is quite good and the size and placing of the buttons reflect their purpose.

The device is not particularly ergonomic and would be quite uncomfortable to hold for long periods of time. The device is a suitable size to be carried in the pocket or around the neck but could afford to be larger without compromising this. Operation is fairly intuitive and the menu's are clear, however the screen size is limited by the size of the device and could benefit from being enlarged.

Scanning words is simple although takes some getting used to. The combination of audio and visual output make the device fun and accessible although screen size is a hindrance of on-screen information.

Connecting the device to a PC is straight forward and is the same as all other USB style devices such as cameras and MP3 players. This activity may require assistance from a computer literate adult but could easily be demonstrated and understood.