Blue Tunes Video Overview

Basic algorithm for using Wiimote to control cursor

Using the Wiimote as a way to operate a mouse cursor on a PC is an idea that has come across the minds of many in the home brew community. GlovePIE allows for this task to be done using algorithms related to the camera within the Wiimote and the IR sensor bar used with the Wii console.

This is a recent project I have started and am hoping to expand upon in order to provide a precise and easy-to-use experience for PC users who may opt to using a Wiimote to operate their PCs.

This entry will focus on the basic algorithm I will be using to perform the function of operating a mouse cursor.

These algorithms simply allow you to move the cursor around the screen. There are no button assignments made yet, thus there is no "left-clicking" or "right-clicking." The following are descriptions of the functions and numbers found in the first equation:

• mouse.CursorPosY : The mouse cursor position on the Y axis. Assigning a correct algorithm to this variable allows the user to move the cursor up and down.
• round : Allows us to round up the equation within the first set of parenthesis, thus giving the user smoother mouse movement on the screen.
• wiimote1.dot1y : The Wiimote relies on two IR points provided by the sensor bar in order to give more accurate readings. In GlovePIE, these points are referred to as "dots." Thus, dot1y is the y-position of one of the points being detected.
• wiimote1.dot2y : This is the y-position of the second IR point being detected.
• ((wiimote1.dot1y+wiimote1.dot2y)/2)/760*1200 : This equation allows us to average the two y-positions, thus giving a more accurate reading.
• ((wiimote1.dot1y+wiimote1.dot2y)/2)/760*1200 : 760 is the maximum height reading that can be achieved using the Wiimote, with 0 being the lowest. This number can be determined by doing a simple test with the debug function where debug = wiimote1.dot1y. The average is divided by this number in order to create a percentage and a number between 0 and 1.
• ((wiimote1.dot1y+wiimote1.dot2y)/2)/760*1200 : 1200 is the pixel resolution of the height of the monitor currently being used. The previous part of the equation is multiplied by the height of the resolution in order to pinpoint exactly It is important that this number is adjusted according to the resolution of the monitor being used.

The variables in the algorithm reflecting the x-axis position are similar to the algorithm of the y-axis. However, due to the way the Wiimote was designed, the x-axis is inverted. In order to reflect this, the length resolution, 1920, is subtracted by round(((wiimote1.dot1x+wiimote1.dot2x)/2)/1012*1920)

What is Human-computer interaction?

The definition for human-computer interaction is one that is not agreed upon, but in order to put this research project in perspective and understand HCI on a broad level, the definition provided by HCI Bibliography will be used:

Human-computer interaction is a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them.

Human-computer interaction deals with primarily two things: the user and the computer. The user will use some sort of input device to issue commands, and the interaction is displayed in an output device. Typical input devices include a mouse and a keyboard. An output device typically paired with a mouse and keyboard would be a computer monitor. When commands are entered into the keyboard, or if the scroll wheel is rotated on the mouse, these actions are reflected on the monitor through animations being displayed.

Why is home brew development so popular with the Wiimote?
According to VGchartz, Wii hardware sales worldwide are estimated to be 54.58 million as of October 24, 2009. This comes out to be 48.4% of the video game console market share, with the Xbox 360 holding 29% and the Playstation 3 holding 22.6% respectively. Also, this number far exceeds sales of other computer devices including tablet PCs.

The innovation behind the Wiimote is the main obvious factor which has resulted in the high volume of sales. People are excited by unique methods of controlling ones character interactively through the use of motion. Thus with the number of sales and the excitement the Wiimote brings to its customers, people started to evaluate other methods of using the Wiimote technology.

What types of programs are available?
Currently, when it comes to interaction with the Wiimote, most home brew applications rely on the motion of the user controlling the Wiimote as well as the commands entered by pushing buttons on the controller. There seem to be two different types of programs available for the use of the Wiimote. The first are scripts written to allow the user to use the Wiimote for existing PC applications. This is primarily done through GlovePIE. The second would be ideas created entirely from scratch, and typically run via Microsoft's Visual Studio. Johnny Lee is responsible for the popular applications that fit into this category, including the virtual head tracker application, a finger tracking application, and an artificial smart board application.

These different applications may be difficult to run for some, as there are some prerequisites to having the applications work properly, especially with Johnny Lee's work. However, anyone who is at least somewhat smart when it comes to working with computers should not have any issues.

Some applications are easier to modify and understand than others. GlovePIE scripts tend to be easy to interpret and programming a script for it can be done in at a quick pace. Johnny Lee's projects tend to be more complex and more difficult to understand without any extensive prior programming experience.

Google Earth

One of the samples scripts that comes along with the free download of glovePIE allows a user to take control of the application Google Earth using the Wiimote.

The following comments are found at the beginning of the script:

Google Earth Interface using Wii Remote
by J.Coulston

Instructions:
Start the GlovePIE script, then calibrate your Wiimote (see next section).
After calibrating, make sure Google Earth is the top window.
Tilting up, down, left and right will simulate arrow keys.
Holding B while tilting up and down will tilt the view up and down.
Holding B while tilting left and right will rotate the view.
Pressing + and - will zoom when the Wiimote is level.
Pressing Home will center the view.
Pressing 1 will toggle fullscreen.

After starting the script I calibrated the Wiimote by referring to these instructions left by the author:

//Calibrate your Wiimote!
//Place the Wiimote face up on a flat surface. Change these values until the
//debug line next to the run button shows zero for each axis.
var.xOffset = 10
var.yOffset = -38
var.zOffset = 8

I then ran Google Earth as my top screen to see how user friendly this script actually was.

Tilting up and down worked as expected. But being able to rotate the globe left and right can be a little tricky at first. Tilting was probably not an accurate description of how to accomplish rotating left in right. I had to roll the Wiimote left or right in order to get the desired effects. One other issue I noticed was that there was only one speed setting. Speed was not reflected by how much I tilted and rolled.

Holding the B trigger down while tilting and rolling worked, but the same complaints listed above reflect on this function as well.

The - and + buttons did not perform the described function even after calibrating the Wiimote. These buttons seemed to just tilt the view of the camera up and down. I was not able to easily zoom in and out like I would with a mouse.

The Home button did center the view as expected, and pushing 1 did put the application in and out of full-screen mode.

A look at the code:
Some functions can be easily modified by simply making easy changes in the script.

The most noticeable problem users will probably face is issues with zooming in and out. The code responsible for this function looks like this:

if Wiimote.minus then
Ctrl+Down = not(abs(var.zRot)>10)
else
Ctrl = false
endif

if Wiimote.plus then
Ctrl+Up = not(abs(var.zRot)>10)
else
Ctrl = false
endif

Since the Wiimote is obviously replacing only keyboard commands, I decided to evaluate what Ctrl+Down and Ctrl+Up do when applied to Google Earth. As expected, these commands pan the camera up in down rather than zoom.

A recent post describes what is necessary to apply the keyboard buttons as an output source, so by using this knowledge, I can alter the code to get the desired results when using the Wiimote.

if Wiimote.minus then
key.Minus = not(abs(var.zRot)>10)
else
key.Minus = false
endif

if Wiimote.plus then
key.Equals = not(abs(var.zRot)>10)
else
key.Equals = false
endif

Unfortunately, I have yet to figure out a way to vary the speed of rotation/zooming using the Wiimote (similar to how a mouse would display such an effect). It is possible to mimic the mouse buttons and movement, however, I have not been able to calculate an algorithm in which the Wiimote can create an analog form of speed.

Blue Tunes

The first application tested was an interesting and simple one entitled Blue Tunes.

What is it?

Blue Tunes is a software application that allows a user to control his or her music and video library via Wii remote.

What is required?

• Connection between a Windows based PC and Wiimote
• Blue Tunes software (Download)
• Media player

What media players can be used? (* indicates which media players were tested)

iTunes*
Winamp
Napster
Windows Media Player*
VLC*
Quintessential Player
Media Player Classic
Foobar2000
Yahoo! Music Jukebox
MusikCube
MediaMonkey
Apollo
BS.Player
Songbird (beta v0.3)
GOM Player
Creative MediaSource
JetAudio
Hulu Desktop

Impressions:

Blue Tunes is easy to download and install. Being able to run it successfully is simple and self-explanatory, too. There are a variety of control options available which allow the user to play, pause, skip forward/backward, increase/decrease volume, increase/decrease bass, go to the next/previous track, add to a playlist, and turn the shuffle on/off. Each individual button has two assignments. Its assignment is determined by whether or not the B-trigger is being held. Blue Tunes does provide a graphic layout explaining the functions of each button.

Blue Tunes would be a great and easy tool to use during a party scene or whenever music needs to be played in general. With iTunes, the response is nearly flawless, and the distance that one can use it from is exceptional. (Approx 50 yards)

General complaints: Using the Bluetooth connection in general can drain the Wiimotes battery rather quickly. If the user isn't careful, the Wiimote can be left on for a couple of hours without being used, thus requiring a fresh pair of batteries soon after. I see this as being an issue with multiple related applications, and wish there could be an easy way to turn the Wiimote on/off while being able to maintain a connection to the PC.

iTunes: Blue Tunes was designed around iTunes, thus no noticeable issues were found while using the Wiimote to perform based on the default button layout. The response time between the application and the Wiimote was great. All the buttons worked as described.

Windows Media Player: Although most of the basic functions worked like they were suppose to with WMP (play/pause, next/previous track, volume), some wouldn't work at all. Skipping forward/backward didn't work as expected, and once the shuffle was turned on, the only available working function was the play/pause button. Turning the shuffle back off didn't fix the issue.

VLC: Although VLC is a great media player for multiple media formats, using Blue Tunes with it does not seem practical. The play/pause and volume functions work fine. There are no playlists with VLC, so being able to go to the previous/next track is useless. The skip forward/backward functions did not seem to work either.

GlovePIE 1: "Hello"

Being able to write a script in GlovePIE is pretty simple, especially with previous programming experience. Watching Youtube tutorial videos provided by user theriddler24 can be quite useful for newbies. As of the writing of this entry, no detailed text tutorials have been found.

The buttons on the Wiimote obviously correspond to a boolean value. If the button is pushed down, its value becomes true. Likewise, if the button is left up, its value becomes false.

Being able to map the buttons of an input device to the buttons of an output device is done by typing and running:

[outputdevice].buttonname = [inputdevice].buttonname

When coding scripts which allow the Wiimote to work with PC applications, we typically will treat the keyboard as the output device and the Wiimote as the input device. Thus, if we want to output the letter 'a' on a text program using a Wiimote, we would write a line of script in GlovePIE that would look something like this:

keyboard.a = wiimote1.a

*Note that wiimote1 represents the first Wiimote connected via bluetooth. Wiimote2 would represent the second, and so on. GlovePIE allows one to program up to eight Wiimotes.

Running the following script allows us to output the text "hello " to an output window GlovePIE provides. To access the output window, on the toolbar select TroubleShooter -> Remote Test.

Notice that this script also allows us to use the backspace key by pressing the B-trigger on the Wiimote. The cursor can also be moved by pressing the corresponding button on the direction pad. On the output screen, the text "hello " is displayed every time the A-button on the Wiimote is pressed.

GlovePIE

GlovePIE is a popular input emulator which allows for users to play PC based games and other applications through other means, rather than just a keyboard and mouse.

GlovePIE (Glove Programmable Input Emulator) was created by Carl Kenner and was originally intended to be a way of emulating keyboard and mouse using a virtual glove. The project grew to be more than just allowing a virtual glove, and there is now tremendous support for using the emulator with a Wiimote. Once downloaded from the creator's site, dozens of scripts are available which allow one to play first person shooters, manipulate Google Earth, or operate Firefox all through the use of just a Wiimote. These scripts are openly available to look at and modify. An example of what the scripts look like can be seen by clicking the image below:

Connecting the Wiimote to PC via Bluetooth

In order to have the Wiimote interact with a PC, a connection via Bluetooth is required. Some computers, primarily laptops, do have a Bluetooth receiver already built in, but Bluetooth dongles that act as receivers can also be purchased at varying prices online and at retail stores. People have had issues with different Bluetooth dongles, so doing a little bit of research is a good idea when deciding which model to purchase. Wiibrew has a lengthy list of working dongles as well what is necessary to connect each specific dongle to the Wiimote. I am currently using a Cirago Bluetooth dongle, which has not required a third-party driver (like BlueSoleil) to work properly. I have not had any issues connecting to this dongle, and I personally recommend it.

After following the instructions when installing the driver packaged with the dongle, connecting the Wiimote is pretty simple. Have the receiver search for available Bluetooth connections or new connections. Shortly after doing this, hit buttons 1 and 2 on the Wiimote, which will sync it with the receiver. Blinking LED lights on the Wiimote will indicate this process has started. Note that the Wiimote does not need to be turned on in anyway prior to syncing it.

If connected properly, the Bluetooth receiver should have the Wiimote as an available device that is titled "Nintendo RVL-CNT-01." The LED lights will continue to blink on the Wiimote. Although different Bluetooth receivers will have different settings displays, a screen similar to this should display accessible devices:

That's it! The following entries will describe some of the available programs for the Wiimote-PC interaction, what is required to run them, and how well they work.

Sources:
Wii Projects (using BlueSoleil)
Youtube video by jdavidharper

Who is Johnny Lee?

Johnny Chung Lee is regarded by many as one who has sparked a significant amount of excitement towards home brew development for the Wii, especially when it comes to innovative ideas in regards to human-computer interaction with the Wii remote.

One of his more popular innovations deals with the ability to use the Wiimote as a part of a head tracking display. This discovery has excited many due to the new immersible possibilities the idea provides when it comes to game development.

Developments in Human-Computer Interaction: Motion Technology and the Wii Remote

*note: this entry may be modified with updated information in the future.

Description:

Human-computer interaction (HCI) is defined by the Association for Computing Machinery as “a discipline concerned with the design, evaluation and implementation of interactive computing systems for human use and with the study of major phenomena surrounding them.” HCI revolves around the user interface, whether the user interface is software or hardware based. Various developments in recent history have made HCI more reliable and easy to use, especially with hardware peripherals. Multi-touch surfaces provide a way for users to more naturally interact with mobile devices. Voice recognition technology is becoming more reliable and sophisticated.

One interesting piece of technology released to the general public in America about three years ago was the Nintendo Wii remote. The Wii console became a hot consumer item due to the remote’s motion sensing capability, which relied on an accelerometer, optical sensor, and blue tooth technology to interact with the console. Due to its interesting and entertaining features, the Wii remote has gained a considerable amount of support and attention by hackers and open-source programmers in hopes of allowing the remote to function with other devices, like a home computer. New software programs are released on a regular basis which takes advantage of the remote’s technology.

Objective:

• This research project will primarily focus on interactions between Nintendo’s Wii remote, which relies heavily on motion and blue-tooth technology, and a home desktop computer.
• Current open-source software designed around the use of the Wii remote will be studied and evaluated.
• If possible, modifications of different available software programs will be tested in order to evaluate changes.
• Another goal is to program an application that could perhaps use the Wii remote in a unique way. However, no specific idea is set in stone yet concerning this.
• Videos will be taken in order to display how the Wii remote is used with different software applications. Any pros and cons will be noted and displayed in the video, if possible.
• A blog will be created and regularly updated with notes on how the research is going. The videos will be available to view here as well.
• A final paper will be written on the research. The paper will give a brief introduction on what HCI is, how the Wii remote technology works, and how one can hook up a Wii remote to a PC. The objectives on this syllabus will be covered, and the questions below, as well as any future questions, will be addressed.

Questions:

• How have recent developments in HCI changed the way we interact with computers?
• Why is development involving the Wii remote popular in the open-source community?
• Are there any general pros and cons with using the Wii remote as an input device?
• What kinds of user applications seem to work best? Why?
• What kinds of user applications are difficult to use? Why?
• How easy is it for new users to get adjusted with using the Wii remote to interact with a PC?
• How difficult is programming an application for the Wii remote? What language(s) is used?

Hardware used (may vary):

Wii Remote

Blue-tooth dongle

PC

IR Array

Wii IR sensor bar

Software used:

TBD

Processes to be addressed:

How to connect a Wii remote to a PC

How to work a Wii remote with different available software applications

What software is needed to program the Wii remote

Sources and Support:

http://johnnylee.net/projects/wii

http://www.wiimoteproject.com

http://wiibrew.org