This report describes the computer human interface results from the first of two usability tests of SBIR grant H133S080032 sponsored by the National Institute on Disability and Rehabilitation Research (NIDRR). The goal of the grant is to create a prototype conversational information manager (JotChat) using Tridbit natural language technology to assess the feasibility and usefulness of such a product for the general public and persons with visual impairment in particular, to further the development of Tridbit technology, and to contribute to the understanding of natural language processing and computer-human interfaces. Additional results, including progress reports, lists of sentences being targeted for comprehension (extension sets) and notes on advances in Tridbit technology are documented in other project reports.
The testing yielded two important results about how people interact conversationally with computers
1. Our users treated JotChat as a conversational partner
We hypothesized that because input was via keyboard and the prototype was still early stage, users would be able to comply with the test, but not necessarily with great enthusiasm. We were pleased to discover that, even given these limitations, users responded enthusiastically to interacting with a computer using English. Their experiences indicated the test was enjoyable and not stressful, the learning curve was rapid without requiring detailed instructions or references to a manual. Users learned how to best interact without being explicitly aware of their adjustment, much as we do when we meet new people. We are unaware of any other natural language application that has achieved this ease of interaction.
2. Sighted users were as enthusiastic as visually impaired users
We expected visually impaired users to be most enthusiastic since they represent an underserved population and would find conversational interface uniquely suited to them. By contrast, we expected sighted users to be more attached to the graphical user interfaces of conventional personal information managers. Instead we found that sighted and visually impaired users responded with similar enthusiasm to interacting with a computer using English. This, along with how readily our users imagined other uses of JotChat after experiencing this limited feature set, validates our long-term perspective that Tridbit technology will drive a wide range of applications for improving human interaction with computers and devices.
5 January 2009
Karen Blaedow, Principal Investigator
Custom Technology Ltd
karen@customtechnologyltd.com
and the JotChat team:
Neal Ewers
Kathy Ley
Matt Peterson
This usability test represents the execution of Task 5A.1 of our grant project plan. We asked 19 users to communicate in English with the JotChat prototype to solve 22 scenarios. The testing took place in over a two-week period in December 2008. The specifics of the test protocol are provided in the next subsection.
The specifics of how the Tridbit natural language technology works is not covered in this document but is explained in depth in:
[Blaedow, K. 2007] Babble: Simple Conversations With a Computer. Proceedings of the 2007 Semantic Technology Conference, San Jose, CA. URL = http://www.customtechnologyltd.com/docs/simpleconvers.pdf.
We anticipate a forthcoming paper to be published at the end of this project to describe further advances in Tridbit technology generated by the grant.
Sighted and visually impaired users were selected from the local community to represent a mix of genders, ages, and computer abilities. We selected more users over the age of 60 as they represent part of our target market, those who were less likely to have used computers in employment and those more likely to become visually impaired. In all there were 19 users: 13 sighted and 6 visually impaired. All 6 of our visually impaired users were blind. 2 of the blind users were tested over the phone; the remainder of the visually impaired and all the sighted users were tested in person. Other demographic breakdowns include:
|
|
Sex |
Age |
Computer skill |
|
|
11 female |
2 21-40 |
2 low |
|
|
8 male |
9 41-60 |
4 low/med |
|
|
|
8 over 60 |
4 medium |
|
|
|
|
7 med/high |
|
|
|
|
2 high |
After signing the consent form, users were read the introduction to the testing process and then presented with a practice question followed by the 22 scenarios. Appendix A contains the test script.
Most test sessions were conducted with one moderator who read the scenarios to the users, a note taker, and an observer. JotChat produced a log of each test session for later analysis.
An audio option allowed visually impaired users to hear their typed input echoed as well as hear synthesized audio of JotChat’s response. (Note that our visually impaired users were largely familiar with this form of computer interaction through the use of screen reader accessibility technology.) Phone sessions for 2 visually impaired users were conducted by a moderator reading the scenarios over the phone and typing into JotChat the user’s verbal responses. The user then heard JotChat’s audio response directly input into the phone line.
Each scenario asked the user to either get information from JotChat or provide information to JotChat. Users were told to use simple English sentences to communicate with JotChat. For example the first scenario asks the user:
You need to call your friend Paul but you don’t know his phone number. What would you type to get this information from JotChat?
The user responds by typing English sentences on the keyboard. Below is an example interaction:
Tester: Do you have a phone number for Paul?
JotChat: I don't understand. Can you think of another way to say it?
Tester: What is Paul's phone number?
JotChat: 234-3344
In this example, JotChat did not understand the user’s first response and asks them to think of another way to say it. The user then types a question JotChat understands and JotChat provides the requested phone number.
Appendix B contains a table for each scenario listing all the unique responses for that scenario along with the number of times it was given and whether JotChat understood it. These tables record a total of 695 sentences or phrases that users input in response to the scenarios. Of these 457, or about two thirds, were understood by JotChat.
It is worth perusing these tables to see the many ways people come up with to communicate the same request. For example, 19 unique responses are recorded for scenario 1 where users were to ask for Paul’s phone number. Scenario 6 had the most variations listing 31 ways users tried to ask for all of Joe’s phone numbers.
Seeing all these speech variations helps one appreciate what a complex problem it is to understand natural language. While this report will not get into the Tridbit model that enables JotChat to process English sentences, the nature of the sentences JotChat did and did not understand comes into play as a factor in allowing users to accept JotChat as a conversation partner. In general, JotChat could understand variations of an input that are hard for humans to discriminate. In other words it understands “The phone number of Paul is 123-4567” and “The phone number for Paul is 123-4567.” It would be very difficult for people to remember that only one or the other was valid.
Other variations for expressing the relationship between Paul and his phone number without changing the underlying meaning are also hard to discriminate, for example, “Paul’s phone number is 123-4567” or “123-4567 is the phone number for Paul.” JotChat recognizes many of the forms used to express these basic relationships so the user is not limited to using one specific way. It may even be the case that the underlying model is human-like in the way it accomplishes this, but the important thing for this discussion is that users sense that JotChat has human-like flexibility in what it understands and does not require cognitively difficult discriminations of them.
It is also interesting to note that some scenarios seemed to naturally elicit more variation than others. For example there were many subtle variations and ellipsis for the topics dealing with phone numbers and relationships, but fewer when the topic of the scenario was how many children Jane has (scenario 9) or Paul’s nickname (scenario 16). These observations correspond to Tridbit technology’s model that allows for specialized language patterns within specific topic areas, something that has the potential to resonate as human-like comprehension.
In addition to the raw responses and counts presented in Appendix B, Table A on the next page provides a one line summary of the responses for each scenario.
After completing the scenarios we asked the users what they liked, what they didn’t like and what other information would they like to have in JotChat. Appendix C on page 38 contains a summary of the users’ responses.
Users overwhelmingly responded that they liked interacting with JotChat. People understand conversations and have been using them since they were young children to exchange information. See appendix C for aspects of the system users specifically mentioned liking such as its simplicity, interactivity or artificial intelligence.
The only consistent response about dislikes was the desire to have JotChat understand more ways of expressing information. We will be adding to JotChat in the second half of this study in order to provide more successful ways of conversing with JotChat. We saw some small frustration when our users gave what they thought was a sensible input, only to not be understood. But any frustration was short-lived as they tried again and nearly always came up with a way to communicate.
Several users mentioned a preference for speech input over typing. But at least one user with experience with speech input devices declared he might prefer the certainty of typing.
Users came up with an intriguing list of information they would like to have in JotChat and things it might do in the future. We were impressed with their creativity and ideas. Schedules and lists were two areas we already had in mind, but users expanded on those and came up with others such as lists of when bills are due, order numbers for special batteries, coffee filters, product serial numbers or credit card numbers, access to public information such as phone books, bus schedules, library catalogs, radio and TV channel listings and program guides, online dictionaries and encyclopedias, as well as access to iTunes and other music downloads. Users were also interested in using JotChat as the front end to Skype and other web applications in addition to thermostats, TVs, etc. See appendix C for the complete list. The fact that users so readily came up with all these applications indicates they are hungry for applications that would serve their information needs in a more personal, intelligent and conversational way.
The Completed column indicates the number of users who were able to come up with at least one sentence or phrase that fulfilled the scenario. The 1st try column is the number who did it on their first try. The last two columns indicate how many unique inputs were entered and how many of those were understood.
|
Scenario |
Completed |
1st try |
Variations |
Understood |
|
1) You need to call your friend Paul but you don’t know his phone number. What would you type to get this information from JotChat? |
19 |
14 |
19 |
12 |
|
2) JotChat knows that Paul has a wife. How do you find who it is? |
19 |
16 |
16 |
10 |
|
3a) What if JotChat does not have the phone number of your friend, Alice. It is 221-4545. How would you enter this information? |
19 |
12 |
19 |
7 |
|
3b) Verify that JotChat now has Alice’s phone number. |
19 |
17 |
11 |
8 |
|
4) You want to tell JotChat the home phone number of Paul’s neighbor Joe. That number is 233-5555. How would you enter this information into JotChat? |
18 |
16 |
24 |
13 |
|
5) Joe works at a computer store. How would you ask JotChat for his number in order to contact him at work? |
17 |
12 |
21 |
8 |
|
6) What if you wanted to know all of Joe’s phone numbers? How would you get this information? |
16 |
7 |
31 |
7 |
|
7) Your friend Paul’s cell phone number is 222-3333. How would you give JotChat this information? |
18 |
16 |
15 |
6 |
|
8) Paul’s sister is Jane. You want to call her at home. How would you get the information you need to do this? |
19 |
15 |
22 |
16 |
|
9) How would you find out the number of children Jane has? |
19 |
18 |
7 |
6 |
|
10) How would you find out their names? |
19 |
10 |
20 |
9 |
|
11) Paul has a son. His name is Frank. How would you give JotChat that information? |
17 |
13 |
14 |
9 |
|
12) You have never met Joe’s mother, but you need to call her. How would you get help from JotChat on this? |
19 |
15 |
17 |
12 |
|
13) What would you ask to get Paul’s sister Jane’s email address from JotChat? |
18 |
10 |
26 |
11 |
|
14) Joe’s email address is joe@nomail.com. How would you enter this in JotChat? |
18 |
14 |
9 |
5 |
|
15) You know that Betty is the aunt of someone you know? How would you find out whose aunt she is? |
13 |
7 |
29 |
7 |
|
16) You know Paul has a nickname but you can’t remember it. Can you find this out from JotChat? |
19 |
16 |
9 |
6 |
|
17) Before Linda was married to Paul, her maiden name was smith. How would you let JotChat know this? |
18 |
13 |
17 |
12 |
|
18) If JotChat could interface with a phone, how would you ask it to connect you with Joe? |
19 |
12 |
24 |
12 |
|
19) How would you get Linda’s address from JotChat? |
19 |
17 |
12 |
9 |
|
20) Let’s say you didn’t remember who Linda was, how would you have JotChat give you this information? |
18 |
15 |
12 |
2 |
|
21) Paul is having a birthday soon. Get the date from JotChat. |
18 |
9 |
19 |
4 |
|
22a) There is someone named Jim, you want to find some information about him. How would you begin? |
19 |
9 |
15 |
3 |
|
22b) Which Jim? |
18 |
18 |
7 |
5 |
There is a continuum in the flexibility and sophistication of language-based computer human interaction. For example, “English-like” command languages include those offered by cable companies to control TVs, commands spoken to cell phones or even UNIX or SQL commands. One way to determine when the natural-language barrier has been crossed is to observe how users master and interact with a given language system. People have the ability and desire to communicate using natural language. Each individual finds it enjoyable rather than stressful when they can express a thought in his/her preferred way. On the other hand, people generally struggle to master “English-like” command languages, which lack the flexibility to allow people to express the same meaning in multiple ways.
Our results show that people considered JotChat to be more than an “English-like” command language, which is a very positive result since JotChat was not perfect in its understanding of what people told it. JotChat understood an average of 9 different ways of responding to a scenario, but still failed to understand an average of only one-third of the sentences users put to it. The tables show that JotChat understood the most direct and common responses, including most of the variations a user would find equivalent. Users seemed to pick up on this consistent but flexible language style and treat JotChat as a conversation partner. Evidence that users considered JotChat a conversation partner includes:
Humans naturally pay attention and adjust to the capabilities of their conversation partner. If their conversation partner has limits, such as an adult talking to small children or a native speaker talking to someone learning the language, the more capable speaker will adjust their language to what their partner can understand. We do this instinctively.
It is this instinct which allowed people to quickly train themselves with JotChat. We expected this would be the case so were careful not to allow users to view other users prior to their testing sessions. Despite the scenarios getting ever more complex, the subject’s experience from previous scenarios kept their performance improving. Ways of querying for phone numbers that failed to be understood in the first scenarios, such “Could you give me . . .”, “Do you have. . .” or “How would I find. . .” are quickly extinguished and are hardly used by the end of the test. Often, but not always, less assertive users seemed to adjust to JotChat better. It would make sense that a more assertive conversation participant would be less inclined to adjust for their partner.
If the user’s first couple sentences were not understood by JotChat, we let the user dictate whether they wanted to continue to try other ways of phrasing their request. Interestingly, the users seldom wanted to go on without figuring out a sentence JotChat would understand, even when we warned them the scenario was especially narrow in what it accepted. Of the 456 scenarios the users attempted, they abandoned only 19 of them without figuring out a sentence JotChat would understand. Scenario 15, a bit of a trick question to find who Betty is the aunt of, accounted for 6 of the cases where users gave up.
One of the characteristics about the system that encouraged this behavior was the user’s belief that they could find a way to communicate. Not just any way, which is also true of English-like command languages, but a way that “makes sense” to them and would help them in subsequent scenarios. Thus the discovery that direct questions like “What is Paul’s number?” work better than indirect questions such as “Can you give me Paul’s number?” can be processed easily by the user to adjust their speech to JotChat. If it were the case that JotChat understood “the number of Paul” but not “the number for Paul” (it understands both) this would be more difficult for humans to keep straight since the two expressions seem the same.
“Please” was used 22 times. That certainly indicates the users were not treating JotChat like an ordinary computer program. In addition, it was common for people to use the more respectful and polite ways of indirectly asking for information such as “Could you tell me . . .”, “Do you have. . .”, “Let me give you…” While JotChat can easily handle “please”, it unfortunately has a harder time with indirect language. In most cases, the users were quick to adjust their speaking styles to this. We plan to add the ability to understand more indirect language in the second half of the project.
Scenario 22 was constructed to create a situation where the users would have to resolve an ambiguous reference. The scenario asks them to “find some information” about a person named Jim. It turns out JotChat is aware of two people named Jim and asks the user to resolve this ambiguity by giving them the following choice:
Which Jim?
1. Jim Eastman ;
2. Jim Rockford ;
Only 4 of the 19 users thought to respond by typing a 1 or 2. When users were asked why they didn’t type a number, they generally said that it didn’t occur to them because they were in conversation mode.
The JotChat interface has an input box where the user types and a transcript box that has a record of the ongoing conversation. For visually impaired persons there is an accessibility interface that provides a great deal of control in navigating and reading this transcript box. But given the short time frame for the tests one would not expect the visually impaired users to take advantage of this information. The interesting thing to note however was that the sighted users did not appear to make much use of this information either.
One would expect that if the language was difficult to come up with, as might be the case with a command language like UNIX, one would look back in the transcript box to see what worked previously and type it again. Instead all users seemed to operate conversationally, where they simply used their natural ability to remember what was said previously, without needing to refer back to a transcript. When given the choice to edit what they had said or to say it again, most users chose the conversational approach and asked the question again in a different way.
None of the scenarios required specific information from previous scenarios, but there was enough context from one scenario to the next to create opportunities to use pronouns. While JotChat is capable of handling singular pronouns, few users made use of this. Those that figured it out (we didn’t tell users) generally liked the idea and continued to use pronouns. Unfortunately, the pronoun most likely to be used was “their”, asking “What are their names” to get Jane’s children’s names in scenario 10. “Their” is the only pronoun not implemented at the time of the test.
In the follow up questions asking users what they liked and disliked, not a single tester labeled the exercise as tedious, as we expected at least some would. Instead every single tester responded positively in terms of enjoying the interaction. They said it was fun, that they would like to interact with computers in this way and many wanted to be repeat testers.
In addition to these documented comments is the subjective observation that many of our users were somewhat apprehensive at the start of the test but after a few scenarios relaxed and became very comfortable interacting with JotChat. This seemed especially true of those who were computer novices as well as the visually impaired users who understandably assume that learning any new computer program would be a major undertaking.
Overall, there really was very little difference in how sighted and visually impaired users performed. One visually impaired user warned us that if the test took sighted people 45 minutes, it would probably take at least double that for a visually impaired tester. In fact that tester finished faster than the previous sighted tester. Conversation is a mode of interaction that does not significantly favor sighted users.
Sighted users do have the advantage of seeing the screen, but as stated above, they did not appear to make use of it. Visually impaired users must rely on the auditory output from JotChat. The accessibility interface allows some choice in the amount of auditory feedback ranging from echoing each keystroke to just echoing the sentence the user types followed by JotChat’s response. The voice we used for testing was Microsoft Mike. While none of the users complained about it, we plan to allow for more advanced voices in the next round of development.
While both sighted and visually impaired users expressed enthusiasm for using a conversational interface, it is an enabling technology for visually impaired. When the visually impaired users were asked for other things they would like to do with JotChat, they often listed things that would enable them to do things that are too difficult to do now. One blind user described things she stopped doing after losing her sight, such as keeping lists of books to read or product information that might again be possible to do with a future version of JotChat.
Given this is an enabling technology for visually impaired and an alternative way of doing things for sighted users, it remains valid to assume a JotChat application would have a much greater penetration amongst visually impaired users. While JotChat is unlikely to be a mainstream product among sighted users, at least until JotChat offers more capabilities, given the enthusiasm for conversational interaction among our sighted users it may have more of a following among sighted users than we previously expected. We should try to devise questions in the next round to get a more realistic assessment of how likely a user would be of using JotChat if it were available.
We saw a definite excitement about using JotChat technology. Perhaps we shouldn’t be surprised. For 30 years people have been adapting their brains to how a personal computer wants information. With JotChat, our users could see that finally, even in its primitive form, they could give a computer information in a form that was comfortable to them and get it back in a very natural way. Our users picked up that this was revolutionary, and they liked it. In fact, the only dislike consistently mentioned by users was the desire to have JotChat understand more ways of saying things. We are working on that.
The next round of testing will evaluate the feasibility of using spoken input and the issues we will encounter. In addition to gauging the current state of speech recognition, we expect to see differences in the users’ behavior. Users inputting speech are likely to phrase things differently than they would were they using a computer keyboard. We even saw hints of this in the two tests we conducted over the phone. Since these users dictated what they wanted to say to the person conducting the test, the net experience was akin to speech input. Phone testers tended to give somewhat longer, more complex inputs. They also seemed to be a little slower to adjust to the abilities of their conversational partner. Obviously this is a tiny sample, but it will be interesting to see whether this was the result of going through a third party who typed the information or if it foreshadows issues indicative of speech input.
It is hard to downplay the significance that having reliable speech recognition would make to the application of natural language technology such as Tridbits. However if the current state of speech recognition is not where we want it to be, we should take a page from one of our visually impaired users. The user stated that he had used lots of speech input products and mostly, they didn’t work. He felt he had much more control over what he entered and retrieved when using the keyboard. If we can make JotChat conversational and very easy to use, we may actually gain a part of the audience of visually impaired persons who currently struggle with speech input devices.
We are thrilled by our early results and eager to continue expanding JotChat to handle more conversational based inputs and outputs.
We feel JotChat holds great promise for the visually impaired community, helping to empower them to meet the complexity of the modern world with a sophisticated personal data management tool for contacts, calendars, and lists that adapts to their specific needs.
· Make sure JotChat has been closed and restarted with the log function on.
· Write down subject’s name and the time the test starts to match up with the log that JotChat keeps
· Make sure that the cursor is on the input box
Welcome to this research study sponsored by the National Institute on Disability and Rehabilitation Research, part of the Department of Education.
Today you will be helping us test the abilities of a conversational personal information manager we are developing. We call this application JotChat. It allows the user to store and retrieve a variety of different kinds of information, thus making it easier for people to use one application which contains many different types of knowledge.
You interact with JotChat by using a computer keyboard. We will give you a number of scenarios like looking up people's phone numbers, finding how many children a person has, etc. The scenarios involve imaginary people and information. We will not ask you to reveal information about yourself or people you know.
What we would like you to do is to use simple, direct, everyday language to ask for or enter information about the people in the scenarios that will be presented to you. You should try to use complete sentences and not cryptic input such as you might use in a search engine like Google.
A very important thing to remember is that we are not testing your ability to use the computer or to come up with correct answers. We are testing the computer’s ability to understand the natural language you will be using. In fact, we will learn a lot from you as you work because you will ask questions of the computer in ways we have not yet thought of.
Let’s begin by making sure you are comfortable. Place the keyboard on the table in a place that feels comfortable to you.
We would like to begin by having you practice a couple scenarios. You will type your sentence on the keyboard and you will then hear the response spoken by the computer. The information will also be visible on the screen.
Please wait until I have completed a scenario before you start typing. If you need me to repeat the scenario in part or whole, please ask, this is not a memory test.
If you are ready, let’s try the first practice scenario.
Please type, “What is the capital of Iowa.” Note that this type of “capital” is spelled with an “al” at the end. Press the enter key when you finish typing the sentence.
Right: the answer is Des Moines.
Now ask JotChat for the capital of any state you wish and press the Enter key.
Once again, JotChat will give you the answer.
OK, I think we are ready to begin. Just remember that nothing you do will be judged. We don’t care how slowly you type, how many mistakes you make, or whether or not JotChat can’t find the answer to your question. We are testing JotChat’s ability to correctly respond to your questions. There is a lot it doesn’t yet know, so your work today will help us make it smarter.
(Note to tester. If 1 or both of the likely dialog boxes pop up, we will help the subject work through them on the first 2 occasions they occur. After this, we will simply have them ignore the dialogs and help them get back to the input field).
(Note about explanation of dialog boxes. This text will not be read at this point. It will be used only when it is called for in the test session).
Text to be read when this happens.
Because JotChat needs to understand what you say, much as a person would do, it has to understand each word you use. At this time, it does not recognize the word <insert the word they typed>. Thus, you have 3 choices of things you can do. If the word is misspelled, you can ask JotChat for suggestions (the shortcut key for this is Alt S), If the word is the name of a person or place, you can add it to JotChat with Alt A, or you can ignore the word by pressing the Escape key and continue with this scenario. If you do ignore the word, you can either edit the text you have written to change the word, or you can press F3 to delete the text and start over. (or F6 to delete the word)
Do you have any questions at this time?
OK, I will read some scenarios about things I want you to ask JotChat. I won’t tell you exactly what to type because we want to discover the many different ways people use to ask questions and retrieve information. Remember to use simple, everyday language and try to type complete sentences.
Ready? Let’s begin.
1. You need to call your friend Paul but you don’t know his phone number. What would you type to get this information from JotChat?
2. JotChat knows that Paul has a wife. How do you find who it is?
3. What if JotChat does not have the phone number of your friend, Alice. It is 221-4545. How would you enter this information?
4. You want to tell JotChat the home phone number of Paul’s neighbor Joe. That number is 233-5555. How would you enter this information into JotChat?
5. Joe works at a computer store. How would you ask JotChat for his number in order to contact him at work?
6. What if you wanted to know all of Joe’s phone numbers? How would you get this information?
7. Your friend Paul’s cell phone number is 222-3333. How would you give JotChat this information?
8. Paul’s sister is Jane. You want to call her at home. How would you get the information you need to do this?
9. How would you find out the number of children Jane has?
10. How would you find out their names?
11. Paul has a son. His name is Frank. How would you give JotChat that information?
12. You have never met Joe’s mother, but you need to call her. How would you get help from JotChat on this?
13. What would you ask to get Paul’s sister Jane’s email address from JotChat?
14. Joe’s email address is joe@nomail.com. How would you enter this in JotChat?
15. You know that Betty is the aunt of someone you know? How would you find out whose aunt she is?
16. You know Paul has a nickname but you can’t remember it. Can you find this out from JotChat?
17. Before Linda was married to Paul, her maiden name was smith. How would you let JotChat know this?
18. If JotChat could interface with a phone, how would you ask it to connect you with Joe?
19. How would you get Linda’s address from JotChat?
20. Let’s say you didn’t remember who Linda was, how would you have JotChat give you this information?
21. Paul is having a birthday soon. Get the date from JotChat.
22. There is someone named Jim, you want to find some information about him. How would you begin?
At the end of the session, and if it is appropriate, ask:
· Did you like this way of interacting with the computer? Why or why not?
· What other kinds of information would you like to have this software provide for you?
· What types of tasks could this software handle for you?
· Do you have any additional questions or comments?
· Would you be interested in receiving additional info on this project or (depending on the tester) retesting in March?
This ends today’s test. Thank you very much for your help.
· Don’t forget to pay the subject
· Close and restart JotChat.
· Rename the log file using the code assigned to each tester.
· Email the file to Karen with the code name and start time of the subject.
Responses in italics include words that JotChat did not understand. ( indicates the response was given by one phone tester, a superscript indicates more than one.
You need to call your friend Paul but you don’t know his phone number. What would you type to get this information from JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
8 |
What is Paul’s phone number? |
|
2 |
What is Paul’s number? |
|
1 |
What’s Pauls number? ( |
|
1 |
Call Paul. |
|
1 |
What is Paul’s telephone number? |
|
1 |
Give me Pauls phone number. |
|
1 |
What's Paul's phone number? |
|
1 |
What is Pauls number? ( |
|
1 |
I need Paul’s phone number. |
|
1 |
Search for Pauls phone number. |
|
1 |
Tell me Paul’s phone number |
|
1 |
Find Pauls phone number |
|
|
|
|
20 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
How would I find unknown telephone number? |
|
1 |
How do find unknown telephone numbers? |
|
1 |
How do I find unknown telephone number? |
|
1 |
How do I find an unknown telephone numbers? |
|
1 |
Do you have a phone number for Paul? |
|
1 |
Do you have Paul’s phone number? |
|
1 |
Could you give me Paul's phone number? |
|
|
|
|
7 |
|
JotChat knows that Paul has a wife. How do you find who it is?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
5 |
What is Paul's wife's name? |
|
4 |
Who is Paul's wife? |
|
3 |
What is the name of Paul's wife? |
|
1 |
What is his wife's name? ( |
|
1 |
How do I find Paul's wife? |
|
1 |
Who is Pauls spouse? |
|
1 |
Tell me Paul's wife's name. |
|
1 |
What is Pauls wife's name? ( |
|
1 |
Say Paul's wife's name. |
|
1 |
Find Pauls wife. |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
What's his wife's name? ( |
|
1 |
What is Pauls wife name? |
|
1 |
Who is Paul married to? |
|
1 |
Need the name of Pauls wife. |
|
1 |
Need Pauls wife name. |
|
1 |
I need Pauls wife name. |
|
|
|
|
6 |
|
What if JotChat does not have the phone number of your friend, Alice. It is 221-4545. How would you enter this information?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
14 |
Alice's phone number is 221-4545. (2 |
|
2 |
Alices phone number is 2214545. |
|
1 |
The number for Alice is 221-4545. |
|
1 |
Phone 221-4545. I am unable to place a call but may have that ability in the future. |
|
1 |
Her number is 231-4545. |
|
1 |
Alice's number is 231-4545. |
|
1 |
Alice phone number 221-4545. (accepted but not properly understood) |
|
|
|
|
21 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
I don’t have Alice’s last name. Do you have her phone number? |
|
1 |
I know that Alice's phone number is 221-4545. |
|
1 |
Store Alice's phone number, which is 221-4545. |
|
1 |
Stupid computer, her phone number is 221-4545. |
|
1 |
Alice's is 221-4545. |
|
1 |
Record Alice's phone number. ( |
|
1 |
Include Alice's phone number 232-3333. ( |
|
1 |
Add Alice's phone number 221-4545. ( |
|
1 |
JotChat, I believe that you don’t have this phone number, can I give it to you? |
|
1 |
Alice's phone number 2214545. |
|
1 |
2214545 find. |
|
1 |
Find 2214545. |
|
|
|
|
12 |
|
Notes on phone numbers: Phone numbers are separated by dashes, spaces, and sometimes, not separated at all.
Verify that JotChat now has Alice’s phone number.
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
10 |
What is Alice's phone number? |
|
3 |
Give me Alice's phone number. ( |
|
1 |
Tell me Alice's phone number. |
|
1 |
Give me Alices phone number. |
|
1 |
Find Alices phone number. ( |
|
1 |
I need Alice's phone number. |
|
1 |
What is the phone number of Alice? |
|
1 |
What is Alices phone number? |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
What is Alice phone number? |
|
1 |
How do I get a phone number in JotChat? |
|
1 |
Would you give me Alice's phone number? |
|
|
|
|
3 |
|
You want to tell JotChat the home phone number of Paul’s neighbor Joe. That number is 233-5555. How would you enter this information into JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
4 |
Paul's neighbor Joe's phone number is 233-5555. |
|
2 |
Paul's neighbor Joe's home phone number is 233-5555. ( |
|
2 |
Joe's number is 233-5555. |
|
1 |
Paul's neighbor's phone number is 222-3333. ( |
|
1 |
Joe's phone number is 233-5555. |
|
1 |
Paul has a neighbor Joe. His phone number is 233-5555. |
|
1 |
The phone number of Joe, Pauls neighbor is 233-5555. |
|
1 |
Paul's neighbor Joe's number is 233-5555. |
|
1 |
Joe's home phone number is 233-5555. |
|
1 |
The phone number for Paul's neighbor Joe is 233-5555. |
|
1 |
The number of Paul's neighbor Joe is 233-5555. |
|
1 |
Pauls neighbor Joes phone number is 2335555. |
|
1 |
Joe's home phone 233-5555. Joe is Paul's neighbor. |
|
|
|
|
18 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Here’s Paul’s neighbor Joe’s phone number. ( |
|
1 |
Here is Paul’s neighbor Joe’s phone number. ( |
|
1 |
This is the phone number for Paul's neighbor Joe. ( |
|
1 |
This is the phone number for Paul's neighbor Joe 234-3344. ( |
|
1 |
I want to give you Paul's neighbor's phone number 222-3333. ( |
|
1 |
Joe, Pauls neighbor is at 233-5555. |
|
1 |
Pauls neighbor Joe is at 233-5555. |
|
1 |
Here’s Paul’s neighbor’s phone number 233-5555. |
|
1 |
This is Paul's neighbor's phone number 233-5555. |
|
1 |
I’ll give you Paul’s neighbor's phone number 233-5555. |
|
1 |
Let me give you Paul’s neighbor’s phone number 233-5555. |
|
|
|
|
11 |
|
Joe works at a computer store. How would you ask JotChat for his number in order to contact him at work?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
7 |
What is Joe's work number? ( |
|
5 |
What is Joe's work phone number? |
|
3 |
What is Joe's phone number? |
|
1 |
Tell me Joe's work phone number. |
|
1 |
Give me Joe's work number. ( |
|
1 |
Give me Joe's work phone number. |
|
1 |
Please give me Joe's work phone number. |
|
1 |
What is the work phone number for Paul's neighbor Joe? |
|
|
|
|
20 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
2 |
What is the phone number of the computer store where Joe works? |
|
1 |
What is the work number of Joe? |
|
1 |
What is the work phone number of Joe? |
|
1 |
Give me the phone number of Joe, Pauls neighbor who works at a computer store. |
|
1 |
Number of the computer store where Joe works. |
|
1 |
Phone number of Joe at the computer store. |
|
1 |
Find Joe's work number. ( |
|
1 |
Can you give me Joe's work phone number? |
|
1 |
What is Joe's phone number at work? |
|
1 |
What is Joe’s other phone number at the place that he works. |
|
1 |
Joe works at a computer store. |
|
1 |
What is the phone number of where Joe works? |
|
1 |
What is the phone number of the computer store? |
|
|
|
|
14 |
|
What if you wanted to know all of Joe’s phone numbers? How would you get this information?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
4 |
What are Joe's phone numbers? |
|
3 |
List Joe's phone numbers. ( |
|
3 |
What is Joe's phone number? (from prior question) |
|
2 |
What are all Joe's phone numbers? ( |
|
2 |
What are Joes phone numbers. |
|
1 |
Give me Joe's phone numbers. |
|
1 |
What are Joe's phone number? |
|
|
|
|
16 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
2 |
What are all of Joe's phone numbers? |
|
1 |
Give me a list of all of Joe's phone numbers please. ( |
|
1 |
Do you have a list of Joe's phone numbers? |
|
1 |
Joe has several phone numbers. Do you know what these numbers are? |
|
1 |
Joe has many phone numbers. Do you know the numbers are? |
|
1 |
Do you know all of Joe's phone numbers? |
|
1 |
Give me Joe's many phone numbers. |
|
1 |
Give me all of Joes numbers. |
|
1 |
Joes phone numbers are. |
|
1 |
Give me all the phone numbers you have for Joe. |
|
1 |
What telephone numbers do you have for Joe? |
|
1 |
What are all of Joe's telephone numbers? |
|
1 |
Give me all of Joe's numbers. ( |
|
1 |
Give me all of Joe's phone numbers. ( |
|
1 |
Give me Joe's home and work numbers. ( |
|
1 |
What are each of Joe’s phone numbers? |
|
1 |
Give me all of Joe's phone numbers on record. |
|
1 |
Need a list of Joe's phone numbers, please? |
|
1 |
A list of Joe's phone numbers. |
|
1 |
Please give me all of Joes phone numbers. |
|
1 |
What are all of Paul's friend's phone numbers. |
|
1 |
I need the list of Joe's phone numbers. |
|
1 |
Tell me all of the phone numbers that Joe has. |
|
1 |
Joe has more than one phone number. What are they? |
|
|
|
|
25 |
|
Your friend Paul’s cell phone number is 222-3333. How would you give JotChat this information?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
11 |
Paul’s cell phone number is 222-3333. ( |
|
3 |
Paul's cell number is 222-3333. |
|
2 |
Pauls cell phone number is 222-3333. |
|
1 |
Pauls cell number is 222-3333 |
|
1 |
Paul's phone number is 222-3333 |
|
1 |
Pauls phone number is 222-3333. |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Put Paul's cell phone number in, it is 222-3333. ( |
|
1 |
This is Paul's cell phone number 222-3333. ( |
|
1 |
I am going to give you Pauls cell phone number. ( |
|
1 |
I am going to give you Pauls cell phone number 222-3333. ( |
|
1 |
Add Paul's cell phone number. ( |
|
1 |
Add Paul's phone number. ( |
|
1 |
Include Pauls cell phone number. ( |
|
1 |
222-3333 222-3333. ( |
|
1 |
Paul's cell number 222-3333. |
|
|
|
|
9 |
|
Paul’s sister is Jane. You want to call her at home. How would you get the information you need to do this?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
What is Jane's home phone number? |
|
2 |
What is Paul's sister's home phone number? |
|
2 |
What is Jane's phone number. ( |
|
1 |
What is Paul's sister's phone number? |
|
1 |
What is Jane's home phone? |
|
1 |
What is the number of Pauls sibling, Jane. |
|
1 |
What is Jane's home number? |
|
1 |
What's Jane's home phone number? |
|
1 |
I want Paul's sister's home phone number. |
|
1 |
What is Janes home phone number? |
|
1 |
I need Jane's phone number. |
|
1 |
What is Paul's sister Jane's home phone number? |
|
1 |
What is the number of Paul's sister, Jane? |
|
1 |
What is Pauls sister Janes number? |
|
1 |
What is Paul's sister Jane's home number? |
|
1 |
What is Jane' number |
|
|
|
|
20 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
I want to call Jane. What is her number? ( 222-3333 (This is Alice’s number – didn’t understand 1st sentence) |
|
1 |
Home number of Jane, Pauls sister is. |
|
1 |
What is Jane, Pauls sisters, phone number? |
|
1 |
What is Jane, Pauls sister's, phone number? |
|
1 |
What is Paul's sister Jane's phone number at home? |
|
1 |
What is the phone number of Jane who is Paul's sister? |
|
|
|
|
6 |
|
How would you find out the number of children Jane has?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
14 |
How many children does Jane have? ( |
|
1 |
How many children does she have? ( |
|
1 |
How many children does Jane has? |
|
1 |
Jane has how many children? |
|
1 |
How many children does Paul's sister Jane have? |
|
1 |
How many children does Pauls sister Jane have? |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
How many kids does Jane have? |
|
|
|
|
1 |
|
How would you find out their names?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
8 |
What are Jane's children's names? (2 |
|
4 |
What are the names of Jane's children? |
|
2 |
What are the names of Janes children? |
|
1 |
What are the names of her children? |
|
1 |
The name of Jane's children. |
|
1 |
What are her childrens names? |
|
1 |
Give me the names of Jane's children. |
|
1 |
What are Jane's childrens names? |
|
1 |
Who are Jane's children? |
|
|
|
|
20 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
2 |
What are the children's names? ( |
|
1 |
What are the names of Jane's three children? |
|
1 |
What are the 3 children's names? |
|
1 |
Give me the names of Jane's childrens names. |
|
1 |
Give me the names of Jane's childrens. |
|
1 |
What are the names of Paul's sister Jane's children? |
|
1 |
What are their names? |
|
1 |
What are Jane’s kids? |
|
1 |
What are the childrens names? |
|
1 |
What are the names of Pauls sister Janes children? |
|
1 |
What are the names of Jane children? |
|
|
|
|
12 |
|
Paul has a son. His name is Frank. How would you give JotChat that information?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
7 |
Paul’s son’s name is Frank. |
|
3 |
Frank is Paul’s son. ( |
|
1 |
Frank is the name of Paul's son. |
|
1 |
Paul's son is Frank. |
|
1 |
Pauls sons name is Frank. |
|
1 |
The name of Paul's son is Frank. |
|
1 |
Frank is the son of Paul. |
|
1 |
Frank is Pauls son. |
|
1 |
Frank is son of Paul. |
|
|
|
|
17 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
Paul's son is named Frank. ( |
|
2 |
Paul has a son named frank. ( |
|
1 |
Do you know that Paul has a son named Frank? ( |
|
1 |
Paul fathered Frank. |
|
1 |
The name of Paul's son. |
|
|
|
|
8 |
|
You have never met Joe’s mother, but you need to call her. How would you get help from JotChat on this?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
6 |
What is Joe’s mother's phone number? ( |
|
2 |
What is the phone number of Joe's mother? |
|
2 |
Give me Joe's mother's phone number. |
|
1 |
Tell me Joe's mother's number. ( |
|
1 |
What is the number of Joes mother. |
|
1 |
What is Joe's mother's name? Emily What is her number? |
|
1 |
What's Joe's mother's phone number? |
|
1 |
I need Joe's mother's phone number. |
|
1 |
What is Joe's mother's name? Emily What is Emily's phone number? |
|
1 |
What is Joe's mother's number. |
|
1 |
What is the phone number of Joes mother. |
|
1 |
What is the phone number of Joe's mom? |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
How do I call Joe's mom? ( |
|
1 |
I want to know Joe's mother's number. ( |
|
1 |
Who's Joe's mother? |
|
1 |
What is Joe's mothers number. |
|
1 |
What is the phone number of Joe's number? |
|
|
|
|
5 |
|
What would you ask to get Paul’s sister Jane’s email address from JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
5 |
What is Jane’s email address? |
|
2 |
What is Paul's sister Jane's email address? |
|
2 |
What is Paul's sister's email address? ( |
|
1 |
Tell me Jane's email address. ( |
|
1 |
What is the email address of Pauls sister Jane? |
|
1 |
What's Jane's email address? |
|
1 |
Give me Jane's email address, please? |
|
1 |
Please give me Janes email address |
|
1 |
I need Jane's email address. |
|
1 |
What is Jane's e-mail address? (acceptance of “e-mail” inconsistent) |
|
1 |
What is the email address of Jane? |
|
|
|
|
17 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Can you tell me Jane’s email address? ( |
|
1 |
What is Jane's email? ( |
|
1 |
What is the email address of Jane, Pauls sibling? |
|
1 |
What is the email address of Jane, Pauls sister? |
|
1 |
What is Jane's e-mail? |
|
1 |
What is Jane's e-mail address? (acceptance of “e-mail” inconsistent) |
|
1 |
Need Jane's email address. |
|
1 |
I need Jane's e-mail address. |
|
1 |
What is the e-mail address of Paul's sister Jane? |
|
1 |
What is the email of Jane? |
|
1 |
What is the email address of Jane who is Pauls sister? |
|
1 |
Jane is Pauls sister what is her email. |
|
1 |
What is the email of Pauls sister Jane? (dump of all Jane’s info) |
|
1 |
What is the email address for Paul's sister Jane? |
|
1 |
Give me the email for Jane sister of Paul. |
|
|
|
|
15 |
|
Joe’s email address is joe@nomail.com. How would you enter this in JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
12 |
Joe's email address is joe@nomail.com. (2 |
|
4 |
Joes email address is joe@nomail.com |
|
1 |
Joe@nomail.com is Joe's email address. |
|
1 |
The email address of Joe is joe. |
|
1 |
The email address of Joe is joe@nomail.com. |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
Joe's email is joe@nomail.com |
|
1 |
Let me give you Joe’s email address Joe@nomail.com. ( |
|
1 |
This is Joe's email address joe@nomail.com. ( |
|
1 |
Joe's e-mail address is joe@nomail.com. |
|
|
|
|
6 |
|
You know that Betty is the aunt of someone you know? How would you find out whose aunt she is?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
Betty is whose aunt? |
|
3 |
Who’s aunt is Betty? |
|
3 |
Whose aunt is Betty? ( |
|
1 |
Betty is the aunt of whom? ( |
|
1 |
Who is Betty? |
|
1 |
Betty is the aunt of whom? |
|
1 |
Betty is aunt of who? |
|
|
|
|
13 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
Who is Betty the aunt of? ( |
|
2 |
Who is Betty related to? |
|
2 |
Who is Betty’s niece or nephew? |
|
2 |
Who is Betty’s niece? |
|
1 |
Betty is the aunt of someone. Who is it? ( |
|
1 |
Of whom is Betty the aunt? ( |
|
1 |
Of what person is Betty the aunt of? ( |
|
1 |
How is Betty related to? |
|
1 |
Is Betty the aunt of Joe? |
|
1 |
What are the names of Betty’s nieces and nephews? |
|
1 |
List the relatives of Betty. |
|
1 |
List the people that Betty and I know in common. |
|
1 |
Whose aunt is Aunt Betty? |
|
1 |
Who are Betty’s relatives? |
|
1 |
Who's aunt's name is Betty? |
|
1 |
Who’s nephew is Aunt Betty? |
|
1 |
Who is the aunt of Betty? |
|
1 |
Betty is an aunt for which friend? |
|
1 |
Betty is an aunt for who? |
|
1 |
Who is Aunt Betty? |
|
1 |
Betty is aunt for who? |
|
|
|
|
26 |
|
You know Paul has a nickname but you can’t remember it. Can you find this out from JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
12 |
What is Paul's nickname? ( |
|
3 |
What is Pauls nickname? |
|
1 |
Tell me Paul's nickname. ( |
|
1 |
Paul's nickname? |
|
1 |
What is the nickname of Paul? |
|
1 |
What are Pauls names? |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
What other names does Paul use? |
|
1 |
What's Paul's nickname? |
|
1 |
I need Paul's nick name. |
|
|
|
|
3 |
|
Before Linda was married to Paul, her maiden name was smith. How would you let JotChat know this?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
6 |
Linda's maiden name was Smith. |
|
5 |
Linda's maiden name is Smith. |
|
1 |
Paul's wife's maiden name was Smith. ( |
|
1 |
Pauls wife's maiden name was Smith. |
|
1 |
Smith was Paul's wife's maiden name. ( |
|
1 |
Linda Smith married Paul. |
|
1 |
Joe's wife Linda's maiden name was Smith. ( |
|
1 |
Joe's wife Linda's maiden name is Smith. ( |
|
1 |
Linda's name is Smith. |
|
1 |
Paul's wife Linda's maiden name is Smith. |
|
1 |
The maiden name of Paul's wife Linda is Smith. |
|
1 |
Lindas maiden name was Smith. |
|
|
|
|
21 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Smith was Paul’s wife’s original last name. ( |
|
1 |
Pauls wife is Linda Smith Miller. |
|
1 |
Linda's maiden name was Linda Smith. |
|
1 |
The maiden name for Linda who is Paul's wife is Smith. |
|
1 |
Paul’s wife, Linda’s birth name was Smith. |
|
|
|
|
5 |
|
If JotChat could interface with a phone, how would you ask it to connect you with Joe?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
7 |
Call Joe. (2 |
|
5 |
Please call Joe. |
|
1 |
Dial Joe. |
|
1 |
Please telephone Joe. |
|
1 |
Dial Joe's phone number? |
|
1 |
Please dial Joe's number. |
|
1 |
Please dial Joes number. |
|
1 |
Call up Joe. |
|
1 |
Please connect me to Joe. |
|
1 |
Call Joe work. |
|
1 |
Phone Joe work. |
|
1 |
Call Joe on the phone. |
|
|
|
|
22 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
I want to call Joe. ( |
|
1 |
Can you dial Joe’s phone number? |
|
1 |
Call Joe for me. |
|
1 |
Get Joe on the phone. |
|
1 |
Would you connect me with Joe? |
|
1 |
Please connect me with Joe. |
|
1 |
Joe at home. |
|
1 |
Call Joe at home. |
|
1 |
Please phone Joe. |
|
1 |
Please call Joe on the telephone for me. |
|
1 |
Please connect me with Joe. |
|
1 |
Please dial Joe's number for me. |
|
|
|
|
12 |
|
How would you get Linda’s address from JotChat?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
10 |
What is Linda's address? (2 |
|
2 |
What is Lindas address? |
|
2 |
Linda's address? |
|
1 |
Lindas address? |
|
1 |
What is Linda Miller's address? |
|
1 |
What is the address of Linda Miller? |
|
1 |
What's Linda's address? |
|
1 |
Please give me Linda's address. |
|
1 |
I need Linda's address. |
|
|
|
|
20 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
What's Linda's mailing address? ( |
|
1 |
Where do I write to Linda? ( |
|
1 |
Linda's address is? |
|
|
|
|
3 |
|
Let’s say you didn’t remember who Linda was, how would you have JotChat give you this information?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
17 |
Who is Linda? (2 |
|
1 |
Who is Linda Miller? |
|
|
|
|
18 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Who’s kin to Linda? ( |
|
1 |
Who’s Linda related to? ( |
|
1 |
Who’s Linda’s family? ( |
|
1 |
What do you know about Linda? ( |
|
1 |
How do I know Linda? ( |
|
1 |
Who is Linda's people? ( |
|
1 |
What information do you have on Linda? ( |
|
1 |
Tell me who Linda is. |
|
1 |
Give me Lindas information. |
|
1 |
Give me Lindas address and phone number. |
|
|
|
|
10 |
|
Paul is having a birthday soon. Get the date from JotChat.
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
13 |
When is Paul's birthday? (2 |
|
4 |
When is Pauls birthday? |
|
1 |
When is Paul Miller's birthday? |
|
1 |
What day is Paul's birthday? |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
4 |
What is Paul's birthday? ( |
|
3 |
What is the date of Paul's birthday? ( |
|
2 |
What is Pauls birthday? |
|
2 |
When was Paul born? |
|
1 |
Give me Paul's birthday. ( |
|
1 |
Give me Paul's birthday date. ( |
|
1 |
Paul’s birthday occurs on what date? |
|
1 |
Paul's birthday? |
|
1 |
What month and day is Paul's birthday? |
|
1 |
What is Paul’s birth date? |
|
1 |
What is Paul Millers birthday? |
|
1 |
What day was Paul born? |
|
1 |
Paul was born when? |
|
1 |
What is Paul’s birthdate? |
|
1 |
What is Pauls birthdate? |
|
|
|
|
22 |
|
There is someone named Jim, you want to find some information about him. How would you begin?
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
17 |
Who is Jim? (2 |
|
1 |
Where does Jim live? (but didn’t have the info) |
|
1 |
Give me Jim’s information. (but didn’t have the info) |
|
|
|
|
19 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
3 |
Tell me about Jim. |
|
2 |
What information do you have on Jim? |
|
1 |
What can you tell me about Jim? |
|
1 |
Please me me Jim's information. |
|
1 |
Do you know a person named Jim? |
|
1 |
Give me all the information you have on Jim. |
|
1 |
Tell me everything you know about Jim. |
|
1 |
Which Jim are you referring to? |
|
1 |
How do I get to know Jim? |
|
1 |
Information about Jim. |
|
1 |
Information for Jim for my records. |
|
1 |
I need any information you have on Jim. |
|
|
|
|
15 |
|
Which Jim?
1. Jim Eastman ;
2. Jim Rockford ;
Understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
11 |
Jim Eastman or Jim Rockford (2 |
|
4 |
1 or 2 |
|
4 |
Who is <Jim Eastman or Jim Rockford>? |
|
3 |
Eastman or Rockford |
|
1 |
Where does Jim Eastman live? (but didn’t have the info) |
|
|
|
|
23 |
|
Non- understood user inputs (includes any variation of capitalization, commas or end of sentence punctuation)
|
Count |
Response |
|
1 |
Do you know <Jim Eastman or Jim Rockford>? |
|
1 |
Which Jim? |
|
|
|
|
2 |
|
You can get multiple information with a single question.
It’s rather like artificial intelligence.
It is conversational. There are options to try other ways to get the answer. It’s just frustrating not to have the info you need and it is nice to have a place where you can have the information you need.
I like its simplicity
I liked being able to easily edit what I typed.
It gives you choices and makes you think.
Once it is perfected, it’s a good system and tester would enjoy using it
I would prefer this than going through a phone book.
I would love to have it on my kitchen counter, especially if it was voice activated
It remembers stuff and that it is conversational, so it has more of a warm feeling than other methods and that it talks. It’s a cool way to gather related information. I don’t know anything that really does this.
It was quicker than I anticipated.
It is game-like. I like the fact that it is interactive and talks to me.
It is like a 911 system. It pulls up what you want.
It offers an informal way of keeping records.
Could use comfortable, not cryptic language.
Glad not to have to use complete sentences. If I was texting, would use alternative spelling.
Easier to access than Braille note. Speech was very understandable.
I like the ability to type in my information because I have used voice input products and none of them give me as much control as typing on the keyboard. I don't have to worry about voice inflection.
At least 9 users mentioned wanting JotChat to understand more words and/or sentences or mentioned a specific response they didn’t like. Often that was their only dislike, but one said they preferred their little black book.
One user mentioned it being a lot to type, not liking using a keyboard and having to write out a complete sentence. Three people mentioned that they would rather have speech input.
One user who proclaimed not to like computers at all said “I don’t trust it. I’m not sure it is going to remember it. What do you mean that you don’t know? I just told you.” Nonetheless this user had little difficulty interacting with JotChat and did appear to enjoy the interaction.
One user wanted to have features for low vision, such as enlarged text size and white on black text.
Other comments:
“At times the simplicity made me be too specific. I made things more complex than I needed to.”
“There’s nothing not to like but can everything be kept private?”
JotChat’s conversational interface seemed to offer users an almost endless possibility of ideas for how they might use it. Our test users mentioned such things as an appointment calendar as well as lists of things such as addresses, groceries, books to read, product serial numbers, and lots more. Also mentioned was the gathering and preservation of information about family relationships. Users were also interested in the possibility of using JotChat as the front end of Skype and other web applications in addition to thermostats, TVs, etc. Many users mentioned the need to access public information such as phone books, bus schedules, radio and TV channel listings and program guides, online dictionaries and encyclopedias, as well as access to iTunes and other music downloads.
Several users commented on the ease of having one application that could contain a variety of public information in addition to the different kinds of personal information that might be entered and retrieved by the user. (We executed a demonstration project prior to this grant to show how a form of public information - television listings - could be queried using Tridbit technology.)
The following is a list of functions that our test users thought they would like to perform with JotChat. One of our activities going forward will be to determine which of these we would put on the JotChat product roadmap and which might be more suitable for another application of Tridbit technology.
· Want to know what is scheduled in my calendar for the future.
· Doing grocery lists.
· Doing Google type queries and having JotChat launch the browser and go to Google and find the results.
· Having data from outside sources as well as your own data you enter.
· Likes the relationships that can be established between people.
· Wonders about using credit card information to purchase something if the method of entry is speech. Wouldn’t people hear your number?
· Lists
o List of books read and those one wants to read.
o Order numbers for special batteries, coffee filters and other things.
o Lists of product serial numbers.
o Birthday list.
o List of addresses of clients that can be easily updated with additional entries and deletions.
o Keeping recipes, list of ingredients, etc.
o Lists of restaurants and type of food they have.
o Lists of grocery stores.
· Retrieving and printing out address labels. Somehow, being able to organize them in JotChat.
· A calendar that I could search for upcoming appointments.
· A calendar with a way to remind me of an appointment.
· Simple family tree. Use the relationship ability of JotChat to add family members and their relationship to other family members.
· When bills are due.
· Front end of refrigerator, perhaps reading the bar codes of what was there. This may also work to help people who are blind find products in many stores.
· Front end to numerous appliances; TV, fridge, thermostat, etc.
· Possible synchronization with Outlook.
· Date retrieval. What day of the week does Monday June second, 2009 come on? This is needed to know if I want to make an appointment on that date given that I have certain things I need to do on certain days of the week.
· Like to be able to type in phone numbers without dashes or spaces and have JotChat put them in.
· Access to public information.
o Phone books,
o Library catalogs
o Dictionaries and encyclopedias
o Wikipedias on various topics.
o Access to XM radio, other radio stations, TV channels, etc. What are their channel numbers? What is their program schedule?
o Access to bus schedules.
o Interface with iTunes and other music products either the product itself, or a web based search for music such as on Rhapsody.
o Front end of internet phone book.
o Location of shopping malls and the stores that are in them.
o Front end to map quest or some other way of finding directions from point A to point B. This is also needed for visually impaired people who need to tell their driver how to get to a particular place.