/backRow

comment(s) 

Windows Presentation Foundation (WPF) Media Center

http://www.brains-N-brawn.com/backRow 10/31/2005 casey chesnut

NOTE to Apple users. this article was not intended to piss you off. i wrote this as a learning exercise and thats it ... this is not a product. i can't actually use it myself because my video driver does not work with WPF. i know it doesn't match FrontRow ... and admit that in the Conclusion section. this article is intended for .NET developers that are starting to think about using WPF. i chose FrontRow because of its size and its UI. it wasn't too big functionality wise and its UI would be hard to do without WPF. i could have just as easily recreated Microsofts Media Center Edition UI ... but their functionality would have taken longer than i wanted to spend.

Introduction

the PDC got me all sorts of excited to start coding with WinFX. WinFX consists of 3 higher level pillars (WCF - Communications, WPF - Presentation, and WWF - Workflow) and a BCL (Base Class Library). the initial thought was to come up with an application idea that would use all 3 pillars ... referred to collectively as W*F. there are at least 1 book out on each, so i ordered all 3. the first book i read was on WCF. since i'd done a bunch of WS-* dev in the past, i felt really comfortable with the new model, and didn't bother writing any test code with it. immediately went on to read a WPF book. this new dev model looked more foreign to me and brought my confidence down. instead of pushing on, i changed plans and decided i should write an article concentrating exclusively on WPF first (which is this article). then i started looking around for ideas. since my current concentration is on developing for Media Center Edition (MCE), i caught the news of Apple's entry into the Media Center realm called FrontRow. as soon as i saw it, i knew that i could code it. it offers relatively simple media functionality and its UI was just begging to be done in WPF. not to mention, even though i've got Microsoft Media Center installed as my OS, i cant use it to watch videos because it doesnt have 64 megs of video ram (or a TV tuner) ... so i was needing to write a custom media player for it.

this article will detail how i used WPF and the Media Player SDK to prototype the core functionality of Apple's FrontRow. mine is called backRow for obvious reasons (from SimpsonsMath) :

the irony being that i have to sit up front because my eyes suck :(

FrontRow

Apple FrontRow provides a basic media experience including Music, Photos, DVD, and Video. they don't even match the functionality provided by the initial MCE release. lack of TV recording is the biggest missing feature. its only 'innovation' is the 6 button remote control. to get a good idea of its UI and its feature set i looked at the short videos on Apples website (linked above) and the press release video at news.com. this work was done by reverse engineering from the videos, i have never touched an iMac (or an iPod) in person ... nor would i ever want to (prefer the power and flexibility of MS products). if i cant program to it (meaning FrontRow, iPod, and MS PMC being closed to external developers) ... then whats the point? [NOTE check out Thomas Hawk's post about FrontRow vs MCE]

NOTE alot of people didn't understand the previous sentence, so let me explain. i know you can program iMac with Cocoa ... i did not say you couldn't. i know that there are hacks for programming the iPod. but those are hacks and not supported APIs. as a developer you want supported APIs so that if you write a 3rd party add-on, then you will expect it to keep working as new versions come out. MS has this same problem with their SPOT watches and Portable Media Center. there are hacks to program those devices, but no supported API ... so i don't own or write code for either of those. finally, i don't know of anyway to extend FrontRow? ... but Media Center Edition has a couple different ways ... get over it. 

WPF and Media Player SDK

Windows Presentation Foundation is the new UI stack for building .NET applications. i.e. GDI+ is dead. it is built on top of Direct3D and its main advantage is being able to composite a bunch of different graphic techniques : 3D, Images, Videos, Vector Drawing, Text, Animation. one of the better samples i've seen to demonstrate this power is The North Face Demo video. the idea is that it will allow us to build much richer applications to take advantage of the extra GPU power that many machines have today.

wrote this application on an XP SP2 machine. the development IDE was VS .NET 2005 Beta 2 with the September CTP of WinFX installed. the main hassle is that i had to uninstall the previous version of VS 2005 Beta 2 that i had already installed, to install the WinFX runtime beforehand (order matters). the part that currently sucks is that VS.NET does not have a drag and drop (e.g. WinForms) designer for XAML. there is a stand alone product (Sparkle) that is being developed to support designers, and ultimately VS.NET will get a designer (codenamed Cider). there are also some freeware designers such as Mobiforms Aurora. but since i dont mind pain, i went ahead and created the XAML for this by hand. at least VS.NET does provide XAML intellisense which works relatively well. there is also a tool called XamlPad which will give you a real time view of XAML, but its not exactly stable and does not provide intellisense.

for media playback it'll just use what is provided by WPF and the Media Player SDK when necessary (e.g. for DVD playback). although i could use the Media Center Edition SDK, i did not want to make that dependency so that this sample would be more accessible to other developers.

Menu

the first thing to handle was the splash screen. the real thing works by fading the desktop into the background and bringing up a 3D rotatable menu of options. for the fade, i used some pInvoke calls to get a screen shot of the desktop. that image was converted from a Bitmap to a BitmapImage object and associated with an <Image/> tag on the application that took up the entire screen. then 2 animations are kicked off. one animation changes the opacity of the image from opaque to transparent. the other animation changes the size of the image from full screen to 0. the combination of animations makes it look like it is fading into the background. now what i really wanted to do is have a 'live preview' of the desktop, instead of a screenshot. looked at the HwndHost to see if i could use that with a VisualBrush to create the effect ... but i was not able to figure this out. from a newsgroup response, it does not look like it is possible to create a 'live preview' of an application from WPF like Vista can do.

when you exit, FrontRow transitions out the 3D menu and slides back in the desktop. what i wanted to do was to just change the opacity of the whole window to 0 and then close the window. this would give the effect of the desktop coming back into focus. the problem is that WPF will not let you effect the opacity of the Window. to do this you need to revert to pInvoking some Win32 calls. this is lame, since you were able to adjust the opacity of Windows in WinForms. i did not want to pInvoke, so i just closed the window.

as the desktop fades away, the Menu is animated into view. the Menu consists of 4 items (Music, Photos, DVD, and Video). its a 3D menu that can be rotated. i used a Viewport3D to recreate the menu. all it consists of is 4 plane meshes that use an ImageBrush to texture the plane with the image. then i used animations to reposition the plane meshes to recreate the rotating of the menus. the only problem i have with this is that there is a little bit of tearing when the planes overlap. FrontRow animates the menu in from the lower left. WPF can certainly do this, but i chose not to. instead it is just brought into view through the reflowing of the document as the desktop image shrinks. FrontRow also has reflections from the menu items. er, um, i have seen some simple reflection samples, but i dont know how to apply that to Viewport3D elements. pretty sure it can be done though because i think that is what Microsoft Max is doing with their picture frame view? for the layout of this form i used a DockPanel. the menu items text is docked to the bottom and then the Viewport3D is set to fill the entire space through LastChildFill="True" on the DockPanel. i don't like LastChildFill. would prefer to specify explicitly which element i wanted to Fill using a syntax on that element like DockPanel.Dock="Fill". the reason being is because order of elements effects Z-Order. the element that is rendered last has the highest Z-Order. in this example, the Viewport3D must be the last element to Fill, but i would prefer for the Label to have the highest Z-Order and render on top of the Menu. hopefully MS will change this in a later WPF release; else let us manually override Z-Order. next, the Label is docked to the bottom. had to set its FontSize to make it larger. the problem is that it does not look like FontSize changes when the form is resized. so i need to figure out how to make Fonts scale properly. assumed that there it would automatically resize or that i could specify it with some HTML like syntax (FontSize="XLarge"). finally, FrontRow has a slick animation when a menu item is selected. basically the Label and the Viewport3D are translated to the top of the screen for the next page. the view of the Viewport3D is also changed so that the camera focuses in on the selected item. this could also be done in WPF, but it would have been more of a challenge that i'm currently up to. instead, i just collapse the MainMenu panel and then make the selected items menu visible.

Photos

the first sub menu i went after was Photos. for the Layout i used a Grid panel. the top most row is just the photo icon and title. the bottom row is more interesting. the left column is a 3D preview of a picture in the currently selected folder. the right column is a list of what pictures you want to view. attempted to add a 3D preview to the left hand side. the problem is that as soon as i added the 2nd Viewport3D to the project, then the application stopped receiving all mouse click events. this seems to be a bug with the current beta. tried some possible workarounds ... but they did not work. instead i just used a flat <Image/> tag to render the preview. since a flat image is simpler than a 3D model, went ahead and added reflection. one thing their image preview does is periodically cycle the images. mine just displays the first image ... making it cycle is really simple, but that wasn't going to teach me anything about WPF, so i didn't bother. for the ListBox i just have it read the directories from the MyPictures directory. then i stylized the selected event to have a gradient highlight box. FrontRow works a little differently, in that the ListBox moves and the selection maker does not. not sure how to do that, but i bet it can be done. also, their selection maker has rounded corners. for WPF, i think that can be done by making a custom ControlTemplate for the ListBoxItem. finally, they have a sweet little animation that runs on the highlight when a folder is selected. i bet that can be done in WPF ... but i have no desire to trick that up

when you select a folder from the ListBox, then an image montage begins to cycle. for this, i just customized the ImageMontage control included with The North Face demo. the one problem i had with this was getting the Canvas panel to receive keyboard events. ended up creating a PhotosMontage wrapper panel, but that had problems too. the hack i ended up doing was putting an empty ListBox on the panel and set focus to that ... then the parent panel was able to receive keyboard events. found out later that i what i really should have done was set the property Focusable='True' on the panel, and then i would not need the ListBox hack :) NOTE, this implementation only goes one folder level down. if there are any subfolder beyond that, then the ImageMontage canvas will just read those subdirectories for images too. but you won't be able to drill into those folders to only display those specific images ... without some more code.

 

DVD

the next submenu was DVD playback. this is where MediaPlayer comes into the picture because it has a limited API for DVD. the problem is it is an ActiveX control, and WPF does not directly support ActiveX. instead it supports hosting WinForms which can host ActiveX (NOTE MHender's post). so i created a WinForms UserControl and embedded MediaPlayer and wrapped it in an object model for what backRow needed to support. to test it i created a super simple WinForms application just to make sure everything was working ... and it was! then i embedded the WinForms control in WPF and wired everything up. the first screen is for 'Reading the DVD'. i put this screen in place but only for error messages. nor did i create an animation. if the DVD is read correctly then it just goes to the next screen.

if the DVD is in your drive (and you have a DVD Decoder) installed, then it will just auto start. this is where i started having problems. first, i could not initially see the video in my implementation. i could hear the audio to know that it was playing ... but no video. to debug i created a simpler WPF application to host the WinForms DvdControl ... and it worked, the video rendered. by slowly building up the main app functionality into the test app, i discovered that the problem was the usage of the <ImageBrush/> element in the Main Menus Viewport3D. if i switched that out and used a SolidColorBrush instead then the video would render. an MS employee tried to recreate this, but it worked for him. turns out that it is probably a video driver problem ... and that some drivers currently dont play friendly with WPF. when i tried installing the latest and greatest video driver, then my application started crashing as soon as an <ImageBrush/> was used. so i rolled back to my older driver. this pretty much sucks, because i have to turn off the images of the Main Menu to get DVD playback. about the only way for me to get around this limitation would be to buy a new notebook with a newer video card.

the second problem has to do with user input. the WinForms application allowed me to get keyboard focus in the MediaPlayer and then i could arrow key my way through the DVDs menu and select items with the spacebar. great ... except i could not get this same functionality in WPF. tried calling Focus() all over the place, but keyboard input would not work. at least mouse input still worked.

outside of the actual menu for the DVD, FrontRow also has their own menu for interacting with the DVD. built the DvdControl to expose these methods so creating a UI on top of it was super simple. but this also gave me problems. first, i could not overlay this panel on top of the hosted MediaPlayer. actually, it did overlay ... but it could not be seen. i knew it was there, because i could keyboard down three times and hit the spacebar, which ejected the disc ... but i could not see the actual menu. crap, so then i tried to collapse the panel containing the DvdControl ... nope ... didn't work either. once, the DvdControl rendered, i could not collapse it. its like it took over the window and wouldn't give it back. an MS person helping me out also discovered that any hosted WinForms is not collapsible (e.g. System.Windows.Forms.Button). pretty sure this is a bug. that pretty much kills the playback functionality because once you get into it, then you cant get out of it without killing the app ... ouch.

finally, i really wanted to do the blur effect when the applications menu is overlaid on top of the MediaPlayer. WPF actually does have a blur effect, but since the overlaid WPF Menu panel would not render ... then you dont get to see it live in the application. but because i wanted to see it so badly, i threw it together real quick in XamlPad and this is what it looks like.

 

the final screen shots are of a playing DVD. the backRow screen shot shows MediaPlayers control panel at the bottom ... just so you can see that it really is hosting MediaPlayer. of course you can turn this off ... as captured by the DVD Menu screen shot above.

 

Videos 

third sub menu was for Video playback. FrontRow has the ability to get Movie Trailers, Music Videos, TV Shows, and Video Podcasts from remote servers. didn't want to write a server side piece too, so i just worked on client side playback from the 'My Videos' folder. the menuing system for Videos lets you drill down before previewing and then selecting a video to play. i took a shortcut and basically combined these into the same screen. backRow has a live video preview on the left and a folder and file list on the right. folders are followed by the > angle bracket to represent they can be further drilled into. when you select a folder, then the current panel will animate off to the left and the next display will animate in from the right. FrontRow has this same behavior ... and its pretty slick. there video preview is also hosted in a 3D space. WPF can support this as well

ran into some issues with these screens as well. mainly with the <MediaElement/>. the MediaElement is what is doing the video playback. its built into WPF and wraps MediaPlayer. if MediaElement supported DVD playback ... then i would have used it above as well. first, it was not intuitive to figure out how to instantiate this thing in code. its super easy in XAML, but code wasnt as straight forward. created a MediaTimeline with a Uri to a file path ... nothing happened. searching around turned up a code fragment for calling MediaClock.Controller.Start(). created a MediaClock from the MediaTimeline and then called Start(). could hear audio for about a second and then it stopped. ok ... looks like an object lifetime issue. notice that the MediaElement has a MediaClock property, so i take the MediaClock created from the MediaTimeline and associate it with the MediaElement ... and video starts playing. yeah! but that was just a lucky guess ... definitely need some samples for how this should be done in code. also, the MediaElement would periodically puke out a ComException for 'Out of Memory'. once it got in this state ... you're mostly screwed. it doesn't seem to make any attempt to recover from this bad state? maybe this is just fixed by me upgrading notebooks as well :) NOTE how ugly MessageBoxes look compared to the sexiness of WPF. for you're full on WPF apps, you will want to create equally sexy MessageBoxes. actually, i wish WPF would just provide some API for this. e.g. let me call MesssageBox.Show("loose.xaml") to specify what XAML should be rendered.

 

the following screens just show 2 videos playing full screen in both FrontRow and /backRow.

the following screen is for FrontRows Movie Trailer functionality. it shows the poster art for recent movies and lets you watch the movie trailers when selected. my assumption is that they are using the Apple RSS feed for movie trailers? looked around for a similar RSS feed for WMV formatted movie trailers ... but could not find it. really surprised that MPAA / movie producers do not have this service to promote their movies. not surprising since i already consider MPAA to be a bunch of idiots. anybody from MSN listening? oh well. if there was such a feed, then it would be simple to create this screen in WPF. it would just be a ListBox with <Image/> elements. the Music Video and Video blog functionality would be just as easy. the ability to purchase TV is the only thing that would be hard

 

Music

the last menu item is for music playback. for this functionality i relied on the MediaPlayer SDK to handle the Library functionality and the WPF MediaElement to do the actual playback. wrote a little class to just handle shuffling songs, getting artists, albums, genres, and all songs. FrontRow also supports Playlists, Podcasts, Audiobooks, etc... which i'm not recreating here. anyway, this implementation will let you drill down from Artists to their Albums to Songs on that album to the Now Playing screen. so the first screen is real simple and just displays the menu of how you can start searching through your music.

the next screen shows the list of all Artists in your music library. NOTE i did not add the functionality of being able to select to play all songs from an artist at these higher levels. instead, i make you drill all the way down to a specific song to play. lame ... but it would be simple to add that functionality, so i didn't bother.

the next screen shows all the songs for an Album. this UI is similar to the Video UI in that the screen transition has the current screen flying off to the left and the next screen flying in from the right. if you escape to back out of the current screen then the transition happens in the opposite direction.

the final screen is for actual music playback. wrote a little method to try and grab the album art and also have an updating ProgressBar at the bottom. NOTE i did not add the functionality to let you click on the progress bar and jump to that spot in the song, but it looks like the MediaElement would be able to support that.

 

since i had some music logic now, i went back and added music support to the Photo montage. when you are viewing photos it just starts playing some random song.

Sound Effects

the next step was to add some sound effects. went to kazaa and downloaded a bunch of short sound clips and found the ones that might fit. to play them i used the SoundPlayer class. it will take a Stream or a path to a file. then it can Play() (asynch), PlaySync(), or PlayLooping(). started out with Play() (asynch) but the playback was awful. it sounded like most of the .wav files were corrupt. if i switched to PlaySync() then they sounded just fine but the main thread was frozen. as a quick hack i just wrote a little extra code to create a new Thread for each sound effect and then call PlaySync() on that thread. not elegant ... but works the way i expected Play() to work.

also, i'd like to see XAML support sound effects built-in, so designers can add it instead of having to put it in my code. something to the effect of

<SoundPlayer Source="{EmbeddedResource click.wav}" Playback="Async" />.

then that could be triggered by a Style.Trigger. an audible style ...

aside, i'd also like XAML to have some way of doing #if. something like <Element x:If="Debug" /> so that i could swap in and out Elements without having to comment in/out XML all the time. this would have come in handy when i was switching the ImageBrush element in and out to test the video card driver problem. since XAML is basically code ... then give it some more of the common source code extensions.

Portable Media Center and WPF/E

the exercise above was to prototype FrontRow on a desktop computer. just so happens that the iPod has a similar (scaled down) UI. i would love to be able to recreate the iPod UI on a Portable Media Center. the problem is that PMCs are not open for development even though they are running the Compact Framework. another reason is that MS is working on WPF/E (Windows Presentation Framework Everywhere) that would run on non IE browsers and mobile devices. would be perfect for PMCs ... so somebody open that platform up for devs. i think PMCs are getting their ass kicked in the market ... so what would it hurt :)

Remote

from the start, i had worked on making every screen controllable with both the mouse and the keyboard. although, i had some unexpected behavior with user input (especially with DVD interop), i was able to get this to work for the most part. for the next step, about the only 'innovation' i'll recognize from Apple has to do with their remote control. it only has 6 buttons compared to the 40+ buttons on MCE remote controls. for some end users, i think thats a good approach.

followed Apple's same lead for this project as well. for keyboard input the valid keys are :

the MCE remote controls sends these same key commands automatically. so this application basically gets remote control support for free ... and only uses 6 buttons from the MCE remote. NOTE it doesn't have the initial 'green button' capability to start the application. would have to create a service that was always listening to start the application. instead, its expected for you to manually start the application and then use the remote control. 

Speech

the 6 button remote is nice ... but i know how to do better. 5 buttons? 4 buttons? ... how about zero! this could be done with the System.Speech namespace that is part of WinFX, or using a RCW COM Wrapper over SAPI 5.1. first, i threw in some speech synthesis on the menu panel. this worked fine. the next part was to add in some basic speech recognition. the problem is that it kept getting stuck when trying to create the SpeechRecognizer class? don't know what was happening, so i just commented it out. the API for both is super simple, the hard part would be making a complete Grammar and refactoring the UI to support voice shortcuts. e.g. saying 'play songs by moby' from anywhere in the UI. anyway, i already implemented this similar concept for MCE in the /mceSapi and /mceSalt articles ... so i wasn't too upset that i couldn't get speech recognition to work here. i'll definitely try again when the next WinFX beta comes out

Videos of usage

these are some videos of backRow in use. to see corresponding Apple FrontRow videos, you can go here.

the first video is of the 3D main menu. i'm not running it full screen but you can see how it fades the desktop away. NOTE it can be run in full screen

Main Menu

the following video is of the Music functionality. it is pretty much identical.

Music

this clip shows Photo functionality. notice the reflection. if there wasn't a bug, then it could be rotated in 3D space as well

Photo

the following clip is DVD playback. the main menu starts out as colored panels because video playback is where i have video driver problems. because of other interop problems this is the least similar to FrontRow.

DVD

the last clip is for Videos. this is pretty much the same, except i changed the way the UI works to reduce screen transitions. the 3D menu at the end is colored panels because of the video driver problem.

Videos

Conclusion

this article detailed a rapid prototype of Apple's FrontRow application using a beta version of WPF (Sept CTP) and the MediaPlayer SDK. it is not even close to production quality, this was just a means for me to experiment with the next generation presentation technology with a real world app. nor does this implementation match the visual polish or functionality of FrontRow. for functionality, it does duplicate the core functionality, and most of the other features would be simple to implement. for visual chrome, WPF seems to support everything that would be needed. it took just over 2 weeks (spare time) to get to this stage, starting with no programming experience for WPF, and no visual designer. if the goal was to recreate FrontRow, i estimate that the job would take 1 month for 2 people (1 developer and 1 designer) full time. they would probably have to wait for the next beta version of WPF to come out too, for some of the current bugs to be fixed. not to mention they would get the added benefit of working with the released version of VS2005 (opposed to Beta 2). finally, i recommend that they have some pretty beefy development machines with lots of RAM and a higher end GPU.

overall, i'm impressed with WPF. it makes some development scenarios seem trivial that would have been near impossible with previous APIs. there is just tons of power and flexibility to be exploited. i'm still adjusting to thinking about how to program these apps differently and what these programs should look like. now i find myself watching video advertisements more closely, realizing that alot of those effects could be done with WPF. i'm hesitant to give a thumbs up to immediately start development with it though ... unless you are used to the bleeding edge. most people should probably hold out til the next beta. and there needs to be a community push for a visual designer when this is released. without that, i don't see how MS expects it to be adopted. and that begs the question of who should adopt it? for advertising and Media Center Edition development ... it makes perfect sense. all of the MCE 'Online Spotlight' web sites should start looking at WPF Web Browser Applications (WBA) for their 10' experience (AJAX cannot compete). for the enterprise ... i don't see it. and for mobility, they would really have to make it alot more battery friendly. it does make sense for tablets that can change display orientation so that the apps just scale automagically ... but i might give up that programming power to conserve battery power.

Source

this was built with VS.NET 2005 B2, Sept 2005 CTP of WinFX, and the Media Player 10 SDK. remember, this was just a rapid prototype for me to experiment with, so all of this code is throw away.

Updates

10/31/06 - went ahead and updated the code to run on Vista RC2 and the Sept 2006 CTP of .NET 3.0.

Future

looking at WWF next, then W*F. later