Remotely and virtually connecting people has been an ongoing challenge and pursuit of people in modern society. Engaging and interacting with remote users who are in different locations has been realized and widely used for quite some time through the dissemination of commercially available video chat/conference applications. While these applications suffice for recreational and even business purposes, their offerings are limited in breadth and depth. With the emergence and proliferation of virtual reality and augmented reality applications and hardware, the dynamic of remote communication goes beyond simple voice and video chat. People can now share experiences and work together. Remote collaboration through the implementation of virtual reality and augmented reality tools and techniques can widen the breadth and scope of collaborative experiences.
“JankIn Head ”
JackIn Head, a “human to human telepresence” research project undertaken by Shunichi Kasahara (Sony CSL) and Jun Rekimoto (University of Tokyo, Sony CSL) aims to provide new avenues for first-person point-of-view(pov) content creation and real-time immersive experiences. There are two versions of the JackIn Head system: offline and online.
Offline Version
The offline version consists of a helmet fitted with 6 wide-angle cameras that allow the wearer to capture their environment and experiences. Unlike traditional head/helmet-mounted cameras that record a wide, but limited viewing angle from one perspective, the offline version of JackIn Head captures 360 degrees around the wearer. The researchers have recorded video in full HD (1920×1080) at up to 120fps (frames per second). Not only is the viewing angle expanded, but the experience for remote users and viewers is much more pleasant as the JackIn Head system performs spherical video stabilization. Traditional head or body-mounted cameras and camera systems only record the angle where the lens is facing. Also, they pick up every bump and movement, however slight, that the person wearing the camera experiences. While the person with the camera mounted on their head or body may be able to compensate for these differences and variables in movement, direction and speed, the viewer watching these first-person pov videos will see and follow everything as the camera does, jitters and all. In cases, this leads to losing interest in the content and, in some cases, motion sickness.
As a result of this stabilization and the use of multiple cameras, a more robust and less shaky viewing experience is made possible. Some of the demonstration videos at the exhibition included first-person pov videos of wearers running on a track with hurdles, figure skating, riding a bike, kickboxing and a gymnast on a trampoline. The difference between the unstabilized and stabilized videos is very easy to understand. For example, in the kickboxing video, unstabilized version has many sudden and quick movements when the person wearing the camera rig is attacking or getting attacked.
From the pov of the viewer, watching this content gives a shaky and abrupt feeling. The stabilized version, on the other hand, feels like watching a choreographed dance between two fighters. In this demo video, as in the others, the stabilized version gives a unique first person perspective that is fluid and easy to watch.
Online Version
The online version of JackIn Head has real-time telepresence applications. In this version, one user, the body user, wears the unit that captures the video. Another remotely located user, the ghost user, is able to have an immersive video experience through the use of a head-mounted display. This version of JackIn Head has very unique and distinct look quite different from the offline version. At first glance, the hardware resembles headphones that are the same size as, if not slightly bigger than, the ears of the person wearing them. They look similar to the ear-covering hair buns that Princess Leia sports in the Star Wars movies. There is a reason behind this design. There is an HD fish-eye lens camera on each end/ear. The head mounted display allows the person wearing the JackIn head unit flexibility in movement by causing a minimum amount of inconvenience in terms of weight and obstruction with respect to having to place a camera or cameras at various places on and around the person’s head.
The two fish-eye lens cameras capture 360-degree video from around the body user. The video is stitched and stabilized in real-time by a computer that is hard-wire connected to the body user. The computer’s gpu handles these processes. As a result of this, the ghost user remotely has an omnidirectional view of the physical environment around the body user. This is a significant contribution for collaborative work because the ghost user is free to look around without having to ask the body user to move his or her head in a certain direction or angle each time the ghost user want to see something around the body user. The transmitted omnidirectional video allows the ghost user to freely observe in all directions while using a head-mounted display. Similar to the offline version of JackIn Head, omnidirectional image stabilization provides an enjoyable and comfortable viewing experience for the ghost user. Unlike the offline version, the video is stabilized in real-time to allow for simultaneous video communication between the body user and ghost user.
In current existing remote video collaboration systems, the remote user or viewer depends heavily on the user fitted with the camera for what they can and want to see. For example if the remote user wants to look to the left out of the field of view of the camera, they need to tell the user with the head mounted camera to turn their head and/or body to the left. Also, the position of the camera on the body affects what the remote user is able to see. If the camera is placed on top of the head or on a helmet, the view will be limited by the viewing angle of the camera and the user will not be able to see what is below.
The dual fish-eye lens cameras incorporated in the online version of JackIn Head solve that problem by providing an almost unrestricted spherical view for the ghost user. Besides helping to prevent dizziness and motion sickness for the ghost user, the online version provides a great degree of freedom of movement for the body user wearing the actual JackIn Head unit. The body user is free to look at what they want, when they want and how they want. Since the ghost user gets an omnidirectional pov with the two fish-eye lens cameras, the relative position of the body user’s head does not interfere with the remote visual experience of the ghost user.
Providing a seamless, smooth and almost unrestricted view for the ghost user while allowing the body user to undertake his/her work freely is an important contribution of the real-time JackIn Head system. Collaborative work is made possible as the ghost user is at once given the point of view of the body user as well as the space surrounding the body user. This is demonstrated when the ghost user wearing the head-mounted display looks down and can see the hands of the body user as they are moving. If the body user and ghost user are physically co-located relatively close to one another (for example a few meters apart), the ghost user can see themselves from the perspective of the body user through the JackIn Head camera by turning in the direction of the ghost user. this is booth eerie and fascinating all at once. The potential for physically separated team members to virtually work together is made very clear.