Graduate Research
Generating Multi-sentential Comparative Descriptions
My dissertation research investigates the automatic generation and selection of natural sounding utterances that contrast several qualities of multiple restaurants using contrastive markers such as however and on the one hand...on the other hand....
This investigation focusses on two main questions:
- How can we develop a uniform approach to generating multi-clausal natural language paraphrases involving contrastive connectives? Previous approaches to generating multi-clausal paraphrases (that can be realized within a single sentence or across multiple sentences) have used a separate grammar for realizing discourse phenomena (Forbes et al., 2003; Webber, 2004; Walker et al., 2002). In this study, we plan to develop a single discourse combinatory categorial grammar (DCCG) that is able to handle natural language generation (NLG) at both the sentence and discourse levels. This DCCG will then be used by the OpenCGG realizer to generate a variety of multi-clausal contrasts.
- Which properties and other features can be learned by a ranker to predict human preferences of automatically generated paraphrases of a multi-clausal logical form (LF)? Recent work by Walker et. al. (2007) use n-gram, concept and tree features to predict human preferences. However, these additional features don't perform much better than n-gram features alone. In this study, we intend to identify other factors associated with CONTRAST from both the theoretical and computational linguistic literature, as well as from corpus study on human authored examples involving contrastive markers. I have currently published an initial study of the SPaRKy Restaurant Corpus.
Learning to say it well: Reranking realizations by predicted synthesis quality
In this work, Michael White and I focus on the re-ranking of generated sentence parapharases for optimal synthesis quality. Thus far, we have developed a method for adapting a language generator to the strengths and weaknesses of a synthetic voice, thereby improving the naturalness of synthetic speech in a spoken language dialogue system. The method trains a discriminative reranker to select paraphrases that are predicted to sound natural when synthesized. The ranker is trained on realizer and synthesizer features in supervised fashion, using human judgements of synthetic voice quality on a sample of the paraphrases representative of the generator's capability. Results from a cross-validation study indicate that discriminative paraphrase reranking can achieve substantial improvements in naturalness on average, ameliorating the problem of highly variable synthesis quality typically encountered with today's unit selection synthesizers.
The Effect of Multiple Modalities in Dialogue Act Annotation
In this work, supervised by Chris Brew, I carry out tests to measure the effect of listening to the audio signal during annotation. What I find is that while hearing the audio signal while reading the text does not improve the annotators rate of annotation, it does improve the interrater reliability of the annotation process between two annotators, for a kappa improvement of &kappa = .701 over the score of &kappa = .623 for the text-only condition (no audio heard).
Internships
(MCT Inc) NASA Ames Research Center: Speech Dialogue Systems Group
Intern, Summer 2006
Returning to NASA Ames, I worked with Beth Ann Hockey on HOWe, a Java open-source dialogue system that reads manuals and has spoken query search functionality. My main role was to port the current natural language module of the document reader application from prolog to python, as well as aid in the design of new spoken word search functionality.
The Articulab @ Northwestern University
Intern, Summer 2005
I worked in the Articulab, headed by Justine Cassell. Research at the Articulab is quite varied (follow the link for more detailed information), but in general, the research here is focused on body language, and the role that it plays in verbal communication. During my time there, I looked at how people incorporate body movement into the flow of dialogue. There, in collaboration with Kristina Striegnitz, I was involved in a data collection project which focussed on the role of gesture in turn-taking, and in particular, the minute gestures of eye movement. I aided in the collection and analysis of audio/video data intended to yield interesting information on eye gaze and its role in turn-taking, as well as additional information on hand gestures and posture shifts in direction-giving dialogues.
(MCT Inc) NASA Ames Research Center: Speech Dialogue Systems Group
Intern, Summer 2004
Working with John Dowding, I integrated a planning module with a simulation of a real robotic astronaut, Robonaut. The goal was to enable Robonaut to plan his own movement of his arms and hands to fulfill his goals such as picking up objects and using them to complete tasks, given certain constraints (such as avoiding objects in the world, or kinematic constraints of how Robonaut's joints were hinged, etc.).
BCL Technologies, Inc: NLP Research Group
Intern, June-December 2001, Summer 2002
At BCL Technologies, I was an undergraduate member of an NLP team which designed a spoken language toolkit. This toolkit would allow programmers who lacked computational linguistic skills to to add spoken language command-and-control front ends to their new or existing applications by seeding the toolkit with sentences or corpora from the relevant domain. My role in this project was to aid in the toolkit design, and evaluate the various components of the toolkit, such as the parser and the automatic speech recognizer. In addition, I also conducted research into the dialogue manager.
Webmind, Inc: Natural Language Engineering
Intern, January-August 2000
At Webmind, Inc, I was part of the NLE team which was responsible for creating the language component of the Webmind Artificial Intelligence system. This module was intended to acquire a deep understanding of texts as well as answer queries in a conversational manner. My role in this project was to write the grammar used by its parser, as well as prepare a tagged corpus which I then used evaluate the proprietary tagger and tokenizer. In addition, I aided in prototyping a simple semantics extractor.
Publications
Nakatsu, Crystal and Michael White. 2010. Generating with Discourse Combinatory Categorial Grammar. Linguistic Issues in Language Technology, 4(1).[bib]
Nakatsu, Crystal. 2008. Learning Contrastive Connectives in Sentence Realization Ranking In Proceedings of the 9th SIGdial Workshop on Discourse and Dialogue, pages 76-79, Columbus, Ohio, June. Association for Computational Linguistics. [bib]
Nakatsu, Crystal and Michael White. 2006. Learning to
say it well: Reranking realizations by predicted synthesis
quality. In Proceedings of the 21st International
Conference on Computational Linguistics and 44th Annual Meeting
of the Association for Computational Linguistics, pages
1113-1120, Sydney, Australia, July. Association for
Computational Linguistics. [bib]
Hear examples of the varying degree of quality in synthesis here.
Nakatsu, Crystal and Chris Brew. 2006.
The Effect of Multiple Modalities in Dialogue Act
Annotation. In Proceedings of the 10th Workshop on the
Semantics and Pragmatics of Dialogue (SemDial-10), pages
191-192, Potsdam, Germany, September. Brandial '06. [bib]
Hassan Alam, Hua Cheng, Rachmat Hartono, Aman Kumar, Paul Llido, Crystal Nakatsu, Huy Nguyen, Fuad Rahman, Yuliya Tarnikova, Timotius Tjahjadi, Che Wilcox: Automatic Semantic Grouping in a Spoken Language User Interface Toolkit. COLING 2002
Hassan Alam, Hua Cheng, Rachmat Hartono, Aman Kumar, Paul Llido, Crystal Nakatsu, Fuad Rahman, Yuliya Tarnikova, Timotius Tjahjadi, Che Wilcox: Extending a Broad-Coverage Parser for a General NLP Toolkit. COLING 2002