Creator: Elisabeth de Kleer
Communities Project
Last week, I wrapped up a big
project for another company and was finally able to put some real energy into my
LaunchPad project. One of my goals for the summer was to explore ways of
integrating VR with 360 video, so I started by playing around with a new tool: a 3D scanner created by a company called Matterport for
capturing architectural spaces. The scanner captures 360 panoramas and stitches them together to create a space you can navigate around in by jumping from node to node (see examples: https://matterport.com/gallery/category/historic/). The panoramas are designed to be explored in a browser, but I wanted them for a different purpose: to upload into Unity and use as backdrops for other things, like 360 videos.
I chose as my first location an Icelandic cargo ship that doubles as a co-working community space in
the San Francisco Bay. http://www.7x7.com/a-boat-owned-by-two-tech-geniuses-is-charting-a-new-course-in-coworkin-1786696246...
It took a full two days to scan
the entire ship, which included a crew deck, card deck, top deck, and
wheelhouse. After each scan, the camera had to be repositioned within five feet
of the preceding scan to ensure that the photos would align correctly. An iPad
served as a handy interface.
Once the scan was complete, then
the real work began. As I mentioned above, the Matterport scans are intended to be experienced as
panoramas, but I wanted to see if I could create VR environments that could be
freely explored. To do that, I had to download the
object and texture files from the Matterport interface and upload them into
Unity.
The first thing I noticed about the
meshes in Unity was the HUGE polygon count… thousands and thousands of them.
The second thing I noticed was that the textures looked… odd. A little bit like military camo (I took a screenshot of a messy room where this is especially pronounced).
It was hard to visualize
how it would look in VR, so I exported it to a gaming console that can’t be
named. Okay, it was the HTC Vive (which was all I had available to me).
Exploring the scanned environment
inside VR was educational and helped clarify what I was looking at in Unity. Some
parts of the scan translated really well into VR, like the floors, ceilings,
and basically any flat surface. However, the objects with lots of three
dimensionality were heavily fragmented and full of gaps. I quickly realized
that the these were areas that weren’t captured in the initial scans because
they were spots that were out of sight of the Matterport scanner. For the panorama
images Matterport creates, this doesn’t matter… but for VR, it creates a
problem. (Photo of some friends exploring the VR ship below, gaps and all.)
I didn't get a good photo of the gaps while it was running, but here's a screenshot of the environment in Unity. You get the idea...
I think I can eliminate the gaps by choosing locations that are relatively bare to begin with and taking twice as many scans to capture any objects in the room from all sides. I think that'll get me 90% of the way, but for the additional 10%, I'm going to need to find a way to smooth out the meshes so they look less boxy. I've been told this is challenging, even for programmers (which I am not). Does anyone here at LaunchPad have experience with this? I need to talk to someone more technically-minded to find out if I'm in over my head or what.
As a reminder, my goal is to bring this 3D scanner and 360 camera with me to China when I return later this year and
capture the spaces I shared in my earlier blog posts.
IN OTHER NEWS…
Now that I have free time, I’ve
resumed work on a couple other VR projects that I hope to get funding for at
some point as well. I didn’t pick them as my LaunchPad project, because I was
under the impression that we had to do something BRAND NEW. However, I’d love
to share them with all of you at some point. One is a mixed-reality documentary
series about using VR to help people overcome fears and phobias, which I have a very short promo for. The other is a series of 360 videos for a non-profit organization that helps inmates prepare for life on the outside, which is still in its nascent stages. Splitting my time between these two projects and LaunchPad is going to be challenging this month.
Week 6
So...this week I got a commitment from a DP! Laura James has been working with 360 cameras for a while, is well known in the community, does amazing underwater 360 stuff, and we’ve had great conversations about the story/characters in my project, Revenge F.
Also had a wonderful conversation with my producer, Susan LaSalle. She gave me great feedback on the script/characters. Then helped me go through the budget proposal. She is also well connected in the area, and really interested in 360 video as a new media.
I also have a location for the 3 short scenes we plan to shoot for the proof of concept.
On the Unity side, met with a company called SIXR, and they have a script code for gaze detection. Still talking to them about how I can use that - if they want to donate it, or be paid a contractor fee...or what. So that’s the next step.
Have a couple of auditions set up next week for talent, and a location scout session with the DP/producer. Hoping for a shoot date on August 16, and post up until the deadline after that.
Meantime, it’s scriptwriting and storyboarding!
Whew! Crazy awesome week!
Micah Week 6 | 7/23 - 7/29
Personal Observations/Progress
- I've been in a good working groove due to pulling out some old CD's and building a nice playlist. Monday was a 90's hip-hop day: Black Moon, Alkholics, Loot Pack, Slum Village and Souls of Mischief. Tuesday turned out to be oldies: Shuggie Otis, Sly and the Family Stone and Parliament/Funkadelic. I think for Wednesday I'll dig back in the crates for 90's R&B stuff 😉 I'm sure my neighbors hate me (or love me)
- Saw 'Valerian' on Wednesday and outside of being mesmerized by all of the CG and imaginative art direction, it was cool to see how VR was used in this fantastic future world. I won't spoil it for others, but there is a really cool section in the beginning where tourist use VR devices to enter another dimension. What's funny is after seeing the film, I read an article that discussed this scene and how it may be possible much sooner than you think. I just thought is was a cool and practical application of "future" VR. Also, the film was a nice creative break.
- Next week is Siggraph 2017! I'm really looking forward to attending this year and I'm glad it's in LA. I haven't attended Siggraph since I graduated from college. Lots of VR stuff in the mix this year as well. I'll take notes and will post any significant findings on the OLP FB page.
Production Progress
Task: Set Up Camera Paths
After getting a few characters in the scene, my next major task consisted of getting my camera rig set up. My project is an "on-rails" type of experience, meaning the camera isn't controlled by the viewer, but the movement path is predefined and the movement speed is set. I know that this combination can make some viewers sick, which is why I wanted to spend an adequate amount of time working on this system.
To date, I have purchased and used three different camera systems and each one has its pros and cons. The first system I used was the free "Camera Path Creator" which was great until I learned that it won't work with the VR camera. I'm now using the most expensive system called "Camera Path Animator" which seems to be worth the $30 pricetag thus far. It allows you to preview the camera position and view in the editor, as you lay out your waypoints. It's a bit tricky to get the camera to point the way you want it to at first, but once I figured out the orientation controls it seems to work well.
Given that my scene is fairly large and has some turns, the camera movement must be smooth as silk and some of the other systems were just too jerky. Getting the camera movement right is probably the most important aspect of my project and it's definitely taking a lot of time to tweak.
Given that my scene is fairly large and has some turns, the camera movement must be smooth as silk and some of the other systems were just too jerky. Getting the camera movement right is probably the most important aspect of my project and it's definitely taking a lot of time to tweak.
Task: Transitions and Loading
So, after getting my camera paths worked out (still not perfect) I moved on to tackle another key element - scene transitions. When making games in Unity, transitions and loading aren't really much of a consideration because you typically just cut to a 2D screen or billboard as the next scene loads. Obviously you can't cut to a 2D screen in VR so this required a couple hours of consideration. I got my scenes to load on trigger but making a smooth transition is still a bit of a challenge and there aren't many good solutions on the Asset Store. However, I think I have figured out a workaround. I'll spend much of Saturday working on my thesis.
This week was full of parenting duties and day job deadlines, but I did manage to get some OLP work accomplished. I am still working on effectively layering video and still imagery in unity and trying to find the best optimization for gear. To that end, I downloaded FFMpeg that Eric had mentioned during the most recent video conference and familiarized myself with the video encoding pipeline. After being stumped at first by Windows Powershell syntax (I'm primarily a Mac user), I was able to figure out how to use command.exe instead and encode some videos based on the FFMpeg cheat sheet via the purplepill.io website. I was also excited to try out the new 360 video integration with Unity that Sarah Stumbo recently provided. This was very helpful, and I was able to layer up some test imagery with an alpha channel on top of some placeholder video animation. I ended up with three nested spheres and was very happy with the results in Rift, but was running into errors with the new panoramic shader when I attempted to build and run on gear. I plan to follow up with Unity support to troubleshoot this issue but I'm very encouraged. Looking forward to a productive week ahead! 
Journey to Memphis
This past week I drove 11
hours up and 11hrs back from North Carolina to Memphis Tennessee. But man was
it worth it! I’m happy to say that the National Civil Rights Museum (NCRM) will
be an official collaborator to the “I Am A Man” VR project! The NCRM will look
to use my VR project as an addition to their MLK 50 Celebration, which
commemorates the death of MLK next spring. The VR piece will also be the center
of a new VR exhibit in the museum, hopefully with an opening around the same
time (April 2018).
While
I was at the museum I was able to talk with the director about my vision for
the VR piece to work as a new exhibit in the museum, and I also discussed my
vision for turning the piece into a travelling exhibit that would take this VR Civil
Rights experience into public schools and libraries.
Overall
they loved the unfinished prototype I showed them and gave me a wealth of
feedback on the historical accuracy of my narrative. The museum will serve as a
major consultant the project moving forward and they will also provide any archived photos
and films that might help with the project (some they still do not own).
The
NCRM itself takes you on an emotional journey. When you go through the museum you are hit
with an experience that sheds light on the journey of African Americans in this
country. At times I felt proud reading the captions and seeing the pictures. At
other moments I stood speechless and fought back tears. The museum is attached
to the Lorrain Motel where King was assassinated and standing just feet away
from where he was slain was hard for me. Words cannot express my full range of
emotions.
on the work that the museum is doing to tell these stories. I have believed in
my “I Am a Man” VR project well before this collaboration, but now have even
greater conviction to make it spectacular!!
