Notes on Natural Interfaces with @sarahkjork #devsum13

Sarah starts off her presentation, asking if anyone in the audience had seen Minority Report. Just as Tom Cruise controlled his system, not using mouse and keyboard, but by "owning it" by motioning and gesturing, it's now possible to design systems, that utilises available sensor technology to do that very same thing.

She then proceeds to show off the Kinect camera, pointing out its hardware capabilities and showing off the Kinect toolkit that  is supplied by the Kinect SDK.

The Kinect SDK supports up to 4 connected Kinects.

On the SkeletonFrameReady-event (called 30 times per seconds), we get skeleton data.

Using the Kinect Sensor's CoordinateMapper.MapSkeletonPointToDepthPoint, we can get a canvas position from the normalised values that is returned from the sensor.

Sarah continues on to show a Speech control system, featuring controlling a turtle with basic voice commands (walk forward, backward, left, right).

For speech recognition, you set up a rule set in XML where you define words that you'd like to react on. You have to define this grammar for the application to work, just as you have to define a target language. The engine itself invokes its event SpeechRecognized when it has successfully identified a phrase and SpeechRecognitionRejected when the engine is unable to interpret what you're trying to say.

Last up, is facial recognition
You can track individual points in your face, and also facial rotation. Now, the latter can be used to identify gestures such as nodding approvingly or shaking your head. Further, you can use the GetAnimationUnitCoefficients() to get values on whether or not the user is smiling or frowning, eye brows raised.

Sarah points out over and over again that natural interfaces are different for each user, why coding for it is really hard. You have to consider a lot of different scenarios and test on a lot of different people to make a viable product.


Whereas the X-Box system wants you to calibrate your Kinect before you use it, for Windows, this is something you will have to manage yourself.

Comments

Post a Comment

Popular posts from this blog

Auto Mapper and Record Types - will they blend?

Image
TL;DR: Yes, they will. Thank you for your time. šŸ˜„ Background At Redgate, engineers enjoy a great benefit in that we get are expected to spend 10% of our time to improve on our craft.  Having suffered this week with manually having to copy a bunch of properties from a domain object to its HTTP representation, I decided to spend this week's 10% time to see if I could answer the following questions: Can use AutoMapper to simplify mapping between two types?  Does it also allow me to map between a regular type and a record type? What about the other way around? Can I use this to reduce the maintenance burden in my product? Mapping between two types [automapper fundamentals] To started and get a basic mapping done, created a .NET 5 ASP.NET Web API project, added the below package references and configured automapper : < PackageReference Include ="AutoMapper" Version ="10.1.1" /> < PackageReference Include ="AutoMapper.Extensions.Microsoft.DependencyInj
Read more

Unit testing your Azure functions - part 2: Queues and Blobs

In our last installment , we discovered how to start unit testing our Azure Functions, looking at the HTTP Trigger (and evaulating the function's HTTP response). This time, we'll take a look at using the QueueTrigger, getting supporting data from table storage and outputting Blobs. Yup, I'm tossing you into the deep end! What are we trying to accomplish? This time, we will test a function that performs some arbitrary business logic on bank accounts. The function responds to a storage queue request, logs each command and ensures that each received command is executed exactly once. Almost like a real-world function, huh? ;-) In the function above, we are introduced to three types that we need to deal with in our unit tests - the IQueryable interface, IAsyncCollector and the CloudBlockBlob concrete type. IQueryable is easy enough to work with: Introducing the Testable Async Collector When I first started exploring this topic, I figured I could just use my favourit
Read more

Testing WCF services with user credentials and binary endpoints

Image
This article discusses the following error messages that you might receive from a WCF service An error occurred when verifying security for the message HTTP/1.1 415 Cannot process the message because the content type 'application/soap+xml;charset=UTF-8;action="..."' was not the expected type 'application/soap+msbin1' When doing analysis work on existing services, invoking them in a safe test environment is a handy way of getting an understanding of their operations, especially if the documentation has gone missing (or was never written). SoapUI is a tool that can be used to test WCF Services, but did you know that Visual Studio also comes with its own set of tools? Let's start with SoapUI, which is available from  http://www.soapui.org/ . Its operations are quite straight-forward; however, you might run into issues when trying to access services that are protected by user credentials, or when accessing services with binary endpoints. I
Read more