For Microsoft Kool-Aid Drinkers, Non-paid MS Evangelists written by a Senior Consultant, Passionate about Tech

Voice Commands on the Fly with Windows Phone 8

imageWell, I have used Felix the Cat and Captain Caveman in my last two posts, so it is time to crack out a superhero, The Fly.  In the first four issues of Adventures of the Fly,  others took on the character and made him an adult lawyer who fought crime in Capital City. He was later partnered with Fly Girl.

So, as a continuation of my previous posts, Text to Speech in Windows Phone 8 and Voice Commands in Windows Phone 8, I thought I would expand on my previous post and show you how to update your voice commands on the fly.  For example, what if we wanted to add to a command by introducing a new word to the phrase that was spoken?  Up until now, we could only issue commands that we defined in our voice definition.  Well, today, I am going to walk you through how to add this functionality programmatically.

So, to extend our example, let’s open up or previous project, we will add a new pivot item, text field and a button to our main page, mainpage.xaml.  Here is the code:

   1: <phone:PivotItem CacheMode="{x:Null}" Header="Custom">
   2:     <StackPanel Margin="10,10,14,-10">
   3:         <TextBox x:Name="tbSet" Text=""/>
   4:         <Button Content="Set" HorizontalAlignment="Right" VerticalAlignment="Top" Click="SetCommand" />
   5:     </StackPanel>
   6: </phone:PivotItem>

Also, notice that I have added a Click event handler called “SetCommand”.   If you don’t recall how to get to the handler, or you haven’t created the handler in your code-behind, simply right click on “SetCommand” and select “Navigate To Event Handler”.  You should see the following in your mainpage.cs file:

   1: private void SetCommand(object sender, RoutedEventArgs e)
   2: {
   3:
   4: }

If you were to run your application right now, you would see the following main page screen with the custom pivot as seen below:

imageimage

Stop the app and let’s load up our voicedef.xml file.  The first thing we will do is give our command set a name, “SpeakApp”.  After that, let’s create a new command with the name, “GotoCustom”.  Now, let’s add a phrase list called, “Info”.  At this point, you might have remembered how to do a command, but you probably unsure what I mean by a phrase list, correct?  Well, a phrase list is a set of phrases that you can use to call a command with variable options.  Let’s look at the markup and I will explain.  Make sure that you voicedef.xml looks like the following:

   1: <?xml version="1.0" encoding="utf-8"?>
   2:
   3: <VoiceCommands xmlns="http://schemas.microsoft.com/voicecommands/1.0">
   4:   <CommandSet xml:lang="en-US" Name="SpeakApp">
   5:     <CommandPrefix>Speak App</CommandPrefix>
   6:     <Example> go to text... </Example>
   7:
   8:     <Command Name="GotoText">
   9:       <Example> Go to text </Example>
  10:       <ListenFor> go to text </ListenFor>
  11:       <ListenFor> [and] go to text </ListenFor>
  12:       <Feedback> Going to text... </Feedback>
  13:       <Navigate />
  14:     </Command>
  15:
  16:     <Command Name="GotoVoice">
  17:       <Example> Go to voice </Example>
  18:       <ListenFor> go to voice </ListenFor>
  19:       <ListenFor> [and] go to voice </ListenFor>
  20:       <Feedback> Going to voice... </Feedback>
  21:       <Navigate Target="/MainPage.xaml?pivot=voice"/>
  22:     </Command>
  23:
  24:     <Command Name="GotoCustom">
  25:       <Example> Go to custom </Example>
  26:       <ListenFor> go to {Info} </ListenFor>
  27:       <Feedback> Going to custom... </Feedback>
  28:       <Navigate />
  29:     </Command>
  30:
  31:     <PhraseList Label="Info">
  32:       <Item> Custom </Item>
  33:     </PhraseList>
  34:
  35:   </CommandSet>
  36: </VoiceCommands>

After looking at the code above, it is probably easy to see how we added the new “GotoCustom” command, however, did you notice anything different about it?  How about the {Info} element embedded in the <ListenFor> tag?  Have you seen that before?  Not from any of my earlier posts.  So, let me describe the purpose of that element.  The voice command information is basically saying (not sure about the actual implementation), when you get a command, with a <ListenFor> tag that has an element of {xxx}, go to the PhraseList with the Label of xxx, and dynamically create a bunch of <ListenFor> tags but replace the {xxx} with the items in the list.  Uh, what is a PhraseList?  As you can see from our code, it is a list of values that will be inserted into the <ListenFor> tags.  So, for example, our above code would translate the command to the following (theoretically):

   1: <Command Name="GotoCustom">
   2:   <Example> Go to custom </Example>
   3:   <ListenFor> go to Custom </ListenFor>
   4:   <Feedback> Going to custom... </Feedback>
   5:   <Navigate />
   6: </Command>

If you were to run the application, hit the Windows button on your device, then hold down the Windows button and speak, “Speak App go to custom”  it should open your app.

Cool huh?  Okay, but that just gets us plumbed so we can update the list on the fly.  Are you ready?  Let’s get crackin’!

Open up mainpage.cs.  Let’s insert the code below and discuss it.

   1: private async void UpdateVoicePhraseList(string commandSetName, string phraseListName, List<string> phraseList)
   2: {
   3:     VoiceCommandSet vcs = null;
   4:     if (VoiceCommandService.InstalledCommandSets.TryGetValue(commandSetName, out vcs))
   5:     {
   6:         await vcs.UpdatePhraseListAsync(phraseListName, phraseList);
   7:     }
   8: }

Basically, what we have done is create a function to update a specific phrase list for a give command set.  The first step is to look for our commandSetName in the Voice Command Service’s list of installed command sets.  We do this with the VoiceCommandService.InstalledCommandSets method.  This returns a VoiceCommandSet object if it finds the command set specified by commandSetName that we passed into the method.  If it finds it, we call the UpdatePhraseListAsync() method on that object to replace the phrase list, named phraseListName, with the phraseList sent into our UpdateVoicePhraseList() method.

Note that making the call to UpdatePhraseListAsync() replaces the entire contents of the phrase list we have specified with phraseListName. If you want to preserve the existing items in the list, you will need to add the old phrases to the list sent into the UpdateVoicePhraseList() method.

So, when do we call UpdateVoicePhraseList()?  Well, let’s go to our SetCommand() event handler in mainpage.cs.  Here we will add the following code:

   1: private void SetCommand(object sender, RoutedEventArgs e)
   2: {
   3:     if (!string.IsNullOrEmpty(tbSet.Text.Trim()))
   4:     {
   5:         // Add to Original List
   6:         UpdateVoicePhraseList("SpeakApp", "Info", new List<string>() { "custom", tbSet.Text });
   7:     }
   8:     else
   9:     {
  10:         // Clear back to Original
  11:         UpdateVoicePhraseList("SpeakApp", "Info", new List<string>() { "custom" });
  12:     }
  13: }

Basically, all we do in this event handler is the text from the textbox control, check if it is an empty string, and then call our UpdateVoicePhraseList() method.  As you can see, I chose to keep the original phrase, “custom”, around.  So, if the string is empty I just put “custom” in the list, else I add the new text to the list.

Are we ready to test it?  Well, sure, but I think that I will throw in one more piece.   I want to update the OnNavigatedTo() event to accept our new command phrases. All I do is check to see if Info came back on the querystring.  If not, I ignore it.  Otherwise, I move to the Custom pivot item and set the textbox to show the phrase.  See the code added below:

   1: if (NavigationContext.QueryString.Keys.Contains("Info"))
   2: {
   3:     pivotMain.SelectedIndex = 2;
   4:     tbSet.Text = "sent - " + NavigationContext.QueryString["Info"];
   5: }

Okay, now run it.  Does it behave how you would expect?  I hope so.  Like all voice recognition, the clearer you speak, the more accurate it will work.  It always helps to include good clear decisive commands as well.

Do you remember our SetupVoiceCommands() method?  Well, remember that when we run our app, we call this method to register our voice definition file, voicedef.xml, using the InstallCommandSetsFromFileAsync() method call.  This call registers our voice definition file for our application to the phone, which is removed when the application is removed.   You should be asking why each time I run my app, it doesn’t replace my phrase list.  Great question actually.  I was thinking that it might be a timing thing, or maybe a suspension issue, etc.  However, it turns out that the explanation is pretty simple.  The implementation of VoiceCommandService has the smarts built in to ignore loading the voicedef.xml file if there has been no change to the file.  So, once it is loaded, you don’t have to worry.  However, when we update the phrase list, the voicedef.xml gets updated and hence the next time the app is run, it will re-register your commands.  Note also that you can’t programmatically update the commands themselves.  So, you don’t have to worry about that changing without you knowing.  Hope that makes sense and helps ease any anxiety.

Now .. take heed, I have created the happy path for you, no real error checking and it isn’t the most elegant code.  It is written in a way that you can understand how to get started in terms of updating your voice definition phrase lists on The Fly.

Next up, hmmm … maybe a little in-app voice control?  Come back soon to see what is next …

 
Comments
 
Comments

No comments yet.

Leave a Reply

You must be logged in to post a comment.