Friday, January 31, 2014

Range Imaging - After session post

Yesterday, I delivered a session on Range Imaging at state level workshop in machine learning and computer vision at Modern College of Engineering, Pune to M.Tech students and faculty members. Funny part was they didn't recognize me as a speaker when I went there, but it happens with me due to my age. The workshop started and carried out very well. Hope it continued the same way on day 2 (that's today) as well.

As committed, uploading my presentation and other reference material here. You can have a look. Feel free to ask me queries if you have any.

You can find my presentation here - http://sdrv.ms/1bda1Ak

Other material :

Cooperative Stereo Vision (Carnegie Mellon University) Source code and executable - http://www.cs.cmu.edu/~clz/stereo.html

Carnegie Mellon Image database - http://www.cs.cmu.edu/afs/cs/project/vision/vasc/idb/www/html_permanent//index.html

Other resources :
 
Kinect for Windows - http://www.microsoft.com/en-us/kinectforwindows/
Creative Gesture Camera - http://click.intel.com/intelsdk/Creative_Interactive_Gesture_Camera_Developer_Kit-P2061.aspx
Intel Perceptual computing SDK - http://software.intel.com/en-us/vcsource/tools/perceptual-computing-sdk
PrimeSense - http://www.primesense.com/

Applications Videos :

 

Hope you have an amazing time ahead in Computer Vision Research and Development.

Special thanks to E&TC department, Modern College of Engineering, for calling me as guest speaker.

And as always, thanks for reading!

Cheers!

Saturday, August 31, 2013

Building Windows store games using Unity3D – After session post

I delivered my first session to professionals in this year’s PUG DevCon on 24th Aug 2013. It was fun. The response was amazing. To clear the doubts raised by audience and to provide them with the content I delivered during the session, posting this on my blog.

Unity3D is a popular game development engine. The recently released version 4.2 of Unity brought a revolution. Why? Well, it allows you to build 3D games on every mobile, tablet and desktop platform out there for FREE! Write once, deploy everywhere using a free version of Unity. J

What I will focus here on is, Windows Store games. Below are the prerequisites to build Windows Store Games using Unity3D –

1.     Windows 8 (as of now)

2.     Windows 8 developer license

3.     Visual Studio 2012 (11.0.50727.1)

4.     and obviously, Unity3D 4.2
 
I won’t go in to details of Unity game engine unlike my session. I will focus on windows store deployment part of our game.

So let’s start,

1.     Open Unity

2.     File->New Project


 
3.     Choose the directory, give a good name to your game and start developing your game
 
 

4.     Once you finish developing your game, go to File->Build Settings

5.     Choose Windows Store as a platform
 
 

6.     Select one out of the 4 options available

·         D3D C++

·         D3D C#

·         XAML C++

·         XAML C#

Note that, unity uses DirectX for rendering the game when you build it for Windows Store. Even if you choose one of the XAML options, it still renders using DirectX. XAML can be used as a layer on top of your game for GUI controls or other design part
 
 

7.     When you build, Unity creates a Visual Studio solution which you can open in Visual Studio 2012
 
 

8.     If you select one of the D3D options while building your game you will find an App.cs in your Visual Studio project

9.     If you select one of the XAML options while building your game you will find App.xaml and MainPage.xaml and their C Sharp files in your Visual Studio project

10.   Open Configuration Manager and choose platform (x86/x64/ARM) and choose mode (Debug/Release)
 
 

11.   Run and you are done :)

 

Now, let’s take a look at the 3 main things which Mayura demonstrated in the session.

1.     Debugging – One of the things you need to struggle for in Unity. You can’t debug your scripts using breakpoints in monodevelop or any other editor you use for scripting. You can add breakpoints in your unity script if you are developing for Windows Store. Let’s see, how.

Note that, you can only debug C Sharp scripts.

a.     Go to solution explorer, right click on solution, select Add->Existing project

b.     Select .csproj file created by unity while developing the game.

c.     You can see all the project hierarchy including your assets, resources, etc in solution explorer.

d.     Open your unity script, add a breakpoint and run the game

e.     Breakpoint will be hit and you are able to debug your code step by step

2.     XAML layer – You can add the XAML controls for GUI and managing snapped/portrait views. Open your MainPage.xaml and see the xaml code. You can see SwapChainBackgroundPanel control. Why do we use this control and not any other? Because this xaml control is designed to render DirectX content. Go ahead and add few Buttons, Labels, etc. Use some nice styling and run the game. You can see your xaml controls rendered on top layer and the game behind.

3.     Unity-XAML communication – Just adding the controls is not enough. If you want to use it as GUI, XAML should communicate with Unity and vice versa. When you build your game using one of the two XAML options, you can see the following code automatically created in your app.xaml.cs

 

private WinRTBridge.WinRTBridge _bridge;
private AppCallbacks appCallbacks;

public App()
{
                this.InitializeComponent();
                appCallbacks = new AppCallbacks(false);
}

protected override void OnLaunched(LaunchActivatedEventArgs args)
{
                Frame rootFrame = Window.Current.Content as Frame;

                if (rootFrame == null)
                {
                                var mainPage = new MainPage();
                                Window.Current.Content = mainPage;
                                Window.Current.Activate();

                                _bridge = new WinRTBridge.WinRTBridge();
                                appCallbacks.SetBridge(_bridge);
                                appCallbacks.SetSwapChainBackgroundPanel(mainPage.GetSwapChainBackgroundPanel());
                                appCallbacks.SetCoreWindowEvents(Window.Current.CoreWindow);
                                appCallbacks.InitializeD3DXAML();
                }

                Window.Current.Activate();
}


Let’s see what the code above means,

private WinRTBridge.WinRTBridge _bridge;

So first of all, what is WinRTBridge, it's used internally by Unity, to perform some of native-to-managed, managed-to-native operations, it's not intended to be used by developers. Due some WinRT platform restrictions, it cannot be created from Unity engine code, that's why WinRTBridge is being created here and is passed to Unity engine via appCallbacks.SetBridge(_bridge) instead.

 
appCallbacks = new AppCallbacks(false);

Now, let's take a closer look at AppCallbacks class. When you create it, you specify that your game will run on different thread, for backward compatibility reasons you can also specify that your application can run on UI thread, but that's not recommended, because there's a restriction from Microsoft - if your application won't become responsive after 5 seconds you'll fail to pass WACK (Windows Application Certification), read more here - http://msdn.microsoft.com/en-us/library/windowsphone/develop/hh184840.aspx, imagine if your first level is pretty big, it might take significant amount of time to load it, because your application is running on UI thread, UI will be unresponsive until your level is fully loaded. That's why it's recommend to always run your game on different thread.


Note: Code located in App.xaml.cs, MainPage.xaml.cs is always running on UI thread, unless called from InvokeOnAppThread function.


appCallbacks.SetSwapChainBackgroundPanel(mainPage.GetSwapChainBackgroundPanel());

This simply passes a XAML control to Unity which will be used as a render target for DirectX 11.

 
appCallbacks.SetCoreWindowEvents(Window.Current.CoreWindow);

Sets the core window for Unity, Unity subscribes to the following events (there may be more, depending on when this information was updated) :

·         VisibilityChanged
·         Closed
·         PointerCursor
·         SizeChanged
·         Activated
·         CharacterReceived
·         PointerPressed
·         PointerReleased
·         PointerMoved
·         PointerCaptureLost
·         PointerWheelChanged
·         AcceleratorKeyActivated

 
appCallbacks.InitializeD3DXAML();

This is main initialization function for Unity, it does following things:

•Parse command line arguments, set by AppCallbacks::AddCommandLineArg

•Initialize DirectX 11 device

•Load first level
 

Finally, you may face some problems while building the game for Windows store. E.g. you are able to play the game in unity but as soon as you try to build it for Windows store it gives compiler errors. That’s because you might have used some classes or types in Unity which are not supported by Windows store yet. Things that are not yet supported:

·         Network classes, WWW is supported though
·         Animation on script variables
·         AnimationEvent callback functions with arguments (you have to provide a function with no arguments or with AnimationEvent argument)
·         GameObject.SendMessage (partially works, but function which accepts the message must match the message sent, because the argument conversion doesn't work)
·         Cloth
·         Microphone
·         You can't access C# classes from JS or Boo scripts, you should be able to access JS, Boo classes from C#
·         JS and Boo scripts won't pass WACK at the moment
·         Fog doesn't work on devices with feature level < 9.3, you need to implement it manually, see this
 

You can download the sample game I created during the session here. Open it in Unity, Play around with it and build it for Windows store.

You can download some sample Unity- Windows store games here
 
Let me know in the comments or via any other media, if you have any doubts. I will be happy to solve them. Keep visiting for more stuff about Unity, Kinect and more

And as always, thanks for reading :)

Cheers…

Thursday, August 8, 2013

Few cool things which I failed to post on my blog in last six months


It’s been more than 6 months that I was away from my blog which I deeply regretted. It feels good to be back and many new gadgets and products released meanwhile which I failed to put on my blog time to time.

Mageca – A great common store to all gesture apps.



http://www.mageca.com/ is a great resource for users and great opportunity for developers. What I love in it is it’s for all the leading sensors available in market, right from Kinect to Leap motion. It’s a very nice initiative by Antonis Argyros, who is a friend of mine on LinkedIn.

Kinect for Xbox One – The next generation Kinect
Yes, it fulfills all the expectations we had from next version of Kinect. I won’t explain it’s features here, you better watch them in this video.


Eagerly waiting for next “Kinect for Windows” :)

Leap Motion and Airspace – The most accurate sensor (as they call it) and it’s own marketplace for apps
I was working on Leap Motion from 6 months as I had the development kit at office. Leap Motion sensor finally publicly released on 22nd July and new opportunities emerged for developers and a new fun device for users all around the world. The device is just for $80 and it works with Windows and Mac, and hence it brings gesture technology to masses. :)



It has it’s own marketplace – Airspace which has got few apps in it, but the number will definitely grow in near future. https://airspace.leapmotion.com/



I won’t go away from this blog now onwards. See you in my next post.
Cheers and as always, thanks for reading!