Virtual Reality and Progressive Web Apps
Technical issues behind vr.fleka.me
I loved this project from the very beginning. When Miloš told me about his idea, I simply said: “It’s impossible. Let’s do it!”
We went through several ideas before finalizing it, but at the end I’m very happy that we decided to put the focus on the dreams and imagination of the kids. You can read more about the full story behind the scenes on our official blog at www.fleka.me, let’s get technical here!
Problems, problems, problems…
Fullscreen
The idea is to show a 360° video that shows the kid walking into a space station — our office :) , going towards the space ship — our rocket ;) , traveling into the space, shooting some asteroids — the ThreeJS project, and finally shows a video of the mum calling the kid back to lunch, ending the experience (but only after showing again, after the video, some messages).
So, first things first, the video player: we needed a video player, able to play 360° videos, embeddable, available for mobile devices as well, no matter of the browser. It seemed easy, however it revealed to be a nightmare!
Different browsers behave in different ways: for example, some of them show the address bar always, some of them hide the address bar when in landscape mode. And we needed a fullscreen experience at all times.
On Android, we were able to achieve it using Chrome and setting the page to open in fullscreen. But on iOS you cannot do that in Chrome (nor any other browser due operative system limitations). But that is the default behavior on Safari (when the correct metas in the HTML are specified, of course).
We decided then to show a message to the users that will not use Chrome on Android and Safari on iOS explaining them to open the page with the specific browser.
We wanted at least to use the URL Scheme on iOS to let the users simply click and open Safari right away from any other browser/app, but it seems that Safari doesn’t have a dedicated URL Scheme… mysteries of Apple…
Finally we had the possibility to have a fullscreen web app on mobile (both Android and iOS) and a responsive web app on desktop. The perfect starting point we were searching for.
360° video
To be able to play a vr video we immediately thought to use Youtube to embed their player into our web app, and we did it; we tested it on desktop and everything was ok but finally we opened the project on mobile and, surprise! Youtube doesn’t allow vr videos to be embedded on a webpage on mobile devices! Ooooook…
We then had 2 options:
1- find another player
2- build our own video player
… yes, I’m joking regarding point 2! ;)
Bitmovin was our second candidate. The player worked correctly, both on desktop and on mobile, but we had a couple of issues on mobile:
- the video, after the first 20/30 seconds, started to lag, playing slowly; it seems that this was a known bug at that time by the Bitmovin team. Their support has been great and replied to us right away, very quickly. They should already have solved this problem by now;
- the player, for the very first time, forced the user to answer to several questions, like which type of cardboard they were using, which orientation, etc… this was a huge problem for us.
That’s why we had to give up with Bitmovin and start searching for something else.
Finally, we found Omnivirt (thanks Boris!): their platform is great, it allows you to distribute and analyze the content in different ways. And their player works frameless, it also has a javascript API and Omnivirt hosts your video, streaming it from their servers, buffering it at the best possible quality at any given time. It looked too good to be true!
Autoplay
We integrated the Omnivirt player, tested it on Desktop and Mobile phones, checked the performances, everything was good. Except one thing: on iOS you cannot autoplay videos… aaargh.
How we solved this? Well, there were a couple of possible hacks, but we preferred not to go there, instead we used the User Interface to help us with this. We already had to explain to the user how the cardboard works, meaning that they had to insert their phone into the cardboard and that they can push the only button to shoot (and destroy the asteroids).
This is the trick we used: we kept the video behind all the other elements — the blue area in the image; we added a simple div with our graphics on the top — the grey area — and, as you can see, the “button” on the cardboard is simply triggering a tap event inside the green area, meaning that tapping there is going to play the video.
Then, using the javascript API of Omnivirt, we intercept the “play” event of the video and we hide ourselves our div, leaving all the space to the video only, again, a plain fullscreen experience.
We did this again later, when we have to switch between our ThreeJS experience and the video. Luckily, once the video has been tapped by the user, you can play and pause it anytime via javascript; this allows us to create a fluid experience between videos and Threejs, switching several times between one and the other.
Technologies
We wanted to play hard with this project, and be innovative not only with the idea but also with the realization, and that’s why we picked the latest available cutting edge tools:
Webpack
We were working on this project till November 2016, so we had the possibility to use the version 1.x only (1.13.3 at the end).
Switching from an environment where we are used to have NodeJS with Grunt, Bower and Vagrant machines, the first impression was really positive. Simply run npm start and you already have a server setup, dedicated local URL for testing, live reload out of the box and possibility to have images already optimized and other static assets concatenated and minimized.
After configuring it correctly, with a simple npm build we have a “dist” folder that can be directly uploaded on the production server. Webpack let us also separate the development and the production environment specifying different configurations with 2 different .config.js files.
In thewebpack.production.config.js file, for example, we specified the rule to give the correct hashed name to our different javascript files:
src/[id].[hash].bundle.js or for the images something like:
file?hash=sha512&digest=hex&name=assets/[hash].[ext]
But, before publishing it, you don’t want to test the development environment only, you also need to test the generated code that, in our settings, is saved into the “dist/” folder. The quickest way we found, working with Mac, is to use the built-in php command with php -S localhost:8000 configured with a custom npm run dist.
Finally, this app is built mainly for mobile devices, so we needed also a very easy and fast way to test it on mobile, without having to upload all the times our files on some server. ngrok worked like a charm! It allows you to forward your local port, returning to you an http and https URL that you can open right away on your mobile devices. We had to install it locally, but it’s very easy and bullet proof.
Finally this is how our “scripts” in the Webpack package.json file looked like:
AngularJS
The version used is 1.5.8 and we used Components as well, following our plan to switch our projects to the new version of Angular.
Being a SPA, what we didn’t want to was to force the users to load too many data from their mobile phone only because they clicked on our link. That’s why we implemented the lazy-load functionality, using oclazyload. Together with webpack, we have been able to split our code in several small files, loading only the controllers and the views that the user is visiting, defined in each single state.
Regarding states, we always rely on AngularUI Router. Here one example of one route that defines the controller to be loaded and its .html view.
Together with some Webpack configuration, it’s possible to hash the file names, to avoid caching problems and to increase the possibility to use correctly CDN and reduce traffic on our server, besides being faster for the users to load the web app.
To be able to have always the video in background, so that we can play and pause it when we need it, we have a “video” partial state with our video.html and video.controller.js. Basically, it’s what we usually do with our headers, for example, but this time it was the video and it was behind every other element.
ThreeJS
https://threejs.org/ it’s amazing. We had several problems using it correctly with Webpack and other libraries, but it’s very easy to configure and setup a 3D world in javascript, setting the camera, the lights, your 3D elements (for example the asteroids in our project) and use the Renderer to animate everything. Here you can see an example of our project structure:
We basically tried to use a different Class for each single element. This helped us a lot to extend and implement new functionalities while building the project, when new ideas popped in.
The goal was to have small Classes that were easily instantiable (statically) from other classes when needed and easy to maintain and extend. We usually received request from graphic designers to change the universe background or the look of the asteroids, and let’s not talk about the cockpit :) This made easier to fulfill all the requests, reducing drastically the development timing between requests and staging releases for our internal team to preview.
Blender
We had to create the asteroids. How to do that? Well, I personally found a tutorial online that used Blender to create a hyper-realistic Asteroid Fields, to produce a very cool video. We didn’t have the need to be so realistic, nor to have so many detail, so I successfully installed the software, followed the tutorial and created our first asteroids. The problem (again, problems, yes!) was being able to export it in a format valid for ThreeJS. Luckily I found a plugin available for ThreeJS that I installed and let me export a .json file that I loaded in our ThreeJS project — not without problems of course :) I had to play a lot with the settings of that plugin before understanding how to export a valid and not huge .json.
Once imported, I noticed that ThreeJS was not happy at all with me creating 500 asteroids out of that JSON data… everything was super slow, laggy, horrible… the problem was that I wasn’t caching the object and use it 500 hundred times, I was creating 500 huge (in terms of graphical memory) objects. So, I learned to use the cache in ThreeJS and everything, finally, look great!
I applied the textures our graphic designer created for us, with different bump masks and different colors, and playing with the indipendent rotation and velocity, the asteroid fields finally look good.
Netlify and Ngrok
If you’re not already using it for you staging flow, you really have to give Netlify a try! I connected my “develop” branch on Bitbucket to Netlify and with a simple push I was able to open an URL and preview the project, without the need to upload anything! I also added on Netlify a Slack hook, to be notified when the custom command I specified (for example npm build) were done and when the project was ready on my custom URL (yes, you can set a custom domain FOR FREE!).
ngrok instead helped me to debug the project using my iPhone, tunneling my custom 8080 port to a ngrok URL. If you pay, you can also set your custom domain if you need it.
Amazon S3
Finally, our production server is a bucket on S3. This way we didn’t have to worry about the traffic generated by the user on our existing servers, we didn’t have to setup a server just for this nor use other servers that were already setup for other projects. The project, once distributed with Webpack, is completely static, so it’s a perfect fit for S3 static website hosting.
If you have a cardboard (anyone will make the job), you can enjoy the project here: http://vr.fleka.me or you can simply watch our video on our Facebook page:
