final

paper.tex

@@ -3,38 +3,58 @@
 \usepackage{graphicx}
 \usepackage{lipsum}
 \usepackage{url}
+\usepackage{caption}
+\usepackage{titling}
+\usepackage{csvsimple}
+
+\bibliographystyle{IEEEtran}
+
+\newcommand \myabstract[2][.8]{
+  \renewcommand\maketitlehookd{
+    \mbox{}\medskip\par
+    \centering
+    \begin{minipage}{.5\textwidth}
+      #2
+    \end{minipage}}}
 
 \title{Rendering Portals in Virtual Reality}
 \author{Milan van Zanten\\\normalsize{University of Basel}\\\normalsize{milan.vanzanten@unibas.ch}}
+\myabstract{
+	\begin{center}
+		\textbf{Abstract}
+	\end{center}
+
+	Portals have many applications in the field of computer graphics. Recently, they have found use as a way of artificially increasing the available space in a virtual reality (VR) environment. In this paper, we will cover a technique for making the transition through a portal unnoticeable to the user. Additionally, we will measure the performance impact of rendering portals in a test scene and provide some insight into possible optimisations.}
+
+\captionsetup{font=footnotesize}
 
 \begin{document}
 
 \maketitle
 
-% \begin{center}
-% 	\textbf{Abstract}
-
-% 	WIP
-% \end{center}
-
 \section{Introduction}
 There are several uses for portals in computer graphics including determining the visibility of parts of a 3D scene\cite{lowe:2005}, dividing a scene up into separate areas, rendering a mirror surface, or as traversable portals that can be seen and moved through. These uses can be broadly categorised as either an optimisation technique or a rendering trick. The former two of the mentioned applications are used to determine geometry that can be ignored when rendering a scene and speed up the process. Mirror surfaces and traversable portals though are effects purposefully implemented in an application to benefit the experience.
 
-There are already many implementations of traversable portals in media like video games or architectural visualisations\cite{aliaga:1997}. In this paper we will focus on an application of traversable portals concerning the space limitations in a virtual reality (VR) experience.
+There are already many implementations of traversable portals in media like video games or architectural visualisations\cite{aliaga:1997}. In this paper, we will focus on an application of traversable portals concerning the space limitations in a VR experience.
 
-One of the main challenges when implementing VR applications is immersion, since errors in tracking and latency are noticed particularly strong\cite{abrash:2013}. In an effort to maximise immersion, most of VR has moved from seated experiences with movement limited to three degrees of freedom (just rotation) to ``room-scale'' tracking. Here, in addition to the rotation of the VR headset, the user's position is tracked either via external devices with fixed positions or by cameras that analyse the surroundings and use computer vision algorithms to determine the position. With the added positional tracking, the six degrees of freedom allow the user to move around the room freely. Thus, the only limitation now is the available space. To move around virtual worlds larger than the available space, several different methods have been developed\cite{boletsis:2017}. Examples include head-directed locomotion, point \& teleport and more. Indisputably though, the technique that preserves immersion the most is actual walking inside the real space.
+One of the main challenges when implementing VR applications is immersion, because errors in tracking and latency are noticed particularly strong\cite{abrash:2013}. In an effort to maximise immersion, most of VR has moved from seated experiences with movement limited to three degrees of freedom (just rotation) to ``room-scale'' tracking. In addition to the rotation of the VR headset, the user's position is tracked either via external devices with fixed positions or by cameras that analyse the surroundings and use computer vision algorithms to determine the position. With the added positional tracking, the six degrees of freedom allow the user to move around the room freely. The only limitation now is the available space.
 
-A recent method to circumvent the space limitations of walking inside a real space is the concept of impossible spaces. Overlapping rooms are connected through portals into a single space many times larger than the initial rooms themselves. If such an arrangement is made while factoring in the real available space, the whole virtual space can be accessed by passing through the portals. An example layout can be seen in Figure~\ref{fig1}.
+To move around virtual worlds larger than the available space, several different methods have been developed\cite{boletsis:2017}. Examples include head-directed locomotion and point \& teleport. Indisputably though, the technique that preserves immersion the most is actual walking inside the real space.
 
-\begin{figure}
+\subsection{Impossible Spaces}
+A recent method to circumvent the space limitations of walking in a real space is the concept of impossible spaces. Overlapping rooms are connected through portals into a single space many times larger than the initial rooms themselves. If such an arrangement is made while factoring in the real available space, the whole virtual space can be accessed by passing through the portals. An example layout can be seen in Figure~\ref{fig:impossible-spaces}.
+
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/impossible-spaces.png}
 	\caption{Four rooms with portals (in red) can all be accessed without leaving the smaller real space\cite{lochner:2021}.}
-	\label{fig1}
+	\label{fig:impossible-spaces}
 \end{figure}
 
-To allow for such impossible spaces to exist, the aforementioned portals are necessary. When viewed, they show what the user would be seeing through the portal in the other room, and if a user crosses the plane of a portal, they are transported to the connecting one. In virtual reality, implementing such a portal system poses some extra challenges.
+To allow for such impossible spaces to exist, the aforementioned portals are necessary. When viewed, they show what the user would be seeing through the portal in the other room, and if a user crosses the plane of a portal, they are transported to the connecting one.
 
-The goal of this paper is to explore how classical portals from monoscopic applications can be implemented in stereoscopic VR. We will focus on two main objectives:
+In virtual reality, implementing such a portal system poses some extra challenges.
+
+The goal of this paper is to explore how classical portals from monoscopic\footnote{An application that only shows one viewpoint is called monoscopic, whereas a VR application that renders one viewpoint per eye is called stereoscopic. A detailed explanation of the terminology can be found in Taştı and Avcı\cite{monoscopic-stereoscopic:2020}.} applications can be implemented in stereoscopic VR. We will focus on two main objectives:
 
 \begin{enumerate}
 	\item How can portals be implemented in a way that the user can look and pass through them without noticing?
@@ -43,66 +63,146 @@ The goal of this paper is to explore how classical portals from monoscopic appli
 
 \section{Unobtrusive Transitions}
 \label{unobtrusive-transitions}
-Portals in computer graphics are generally just flat quads onto which the view from within another portal is drawn. In monoscopic applications, a single camera moving from space $A$ through a portal into space $B$ will see a flat image of what is on the other side of it as long as it is still in space $A$. As soon as it crosses the portal plane, it is being transported to space $B$ and will from that point on render that space directly.
+Portals in computer graphics are generally just flat quads\footnote{A flat, rectangular object that can be used to show an image in a 3D environment.} onto which the view from within another portal is drawn. In monoscopic applications, a single camera moving from space $A$ through a portal into space $B$ will see a flat image of what is on the other side of it as long as it is still in space $A$. As soon as it crosses the portal plane, it is being transported to space $B$ and will render that space directly.
 
-The way a VR scene is rendered though raises the question of how to handle the teleportation of the two separate cameras rendering each eye. For example, consider the centre point between the eyes that could be used to decide when the portal plane has been crossed. If the user views the portal from an angle they could clip through the portal with one eye when they enter it before being transported, as shown in Figure~\ref{fig2}. This results in a short flicker when the user passes through a portal at best and a completely wrong view on one eye if the user stops inside the portal.
-
-\begin{figure}
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/head-clipping.png}
 	\caption{What should the right-eye camera render if it is inside the portal wall (in grey), but the centre of the head (in red) has not crossed the portal plane? If nothing is done, the blue part of the user's field of view would not render the next room, but whatever is inside or behind the portal wall.}
-	\label{fig2}
+	\label{fig:obtrusive}
 \end{figure}
 
-One way to solve this problem could be transporting each eye separately whenever it passes through the portal, but that idea conflicts with one of the performance optimisations we will discuss in the next section.
+The way a VR scene is rendered, though, raises the question of how to handle the teleportation of the two separate cameras rendering each eye without the user noticing. For example, consider the centre point between the eyes that could be used to decide when the portal plane has been crossed. If the user views the portal from an angle they could clip through the portal with one eye when they enter it before being transported, as shown in Figure~\ref{fig:obtrusive}.
 
-If we want to keep the optimisations we introduce, a different method of making sure the transition through a portal is unnoticeable by the user, no matter how slowly they move or where they look. This method takes advantage of back-face culling\footnote{A triangle in computer graphics is generally considered to have a front side and a backside, determined by whether the points of the triangle appear in clockwise or counter-clockwise order from where they are viewed. Back-face culling reduces the amount of drawn triangles by only drawing the triangles that are facing the camera.}.
+At best, this results in a short flicker when the user passes through a portal. If the user stops inside the portal, however, it results in a completely wrong view on one eye.
 
-Instead of the portal being a plane, we will have it be a box. Every surface of the box is shaded with the view from the other room. If one eye now clips inside the portal box, it will still see the other sides of the box that show the next room, instead of what was behind the portal originally. The last problem now is that if this eye now wants to look back out from the portal box into the old room, it should not see the portal plane. This is where we use back-face culling to make sure the front side of the portal box is only visible from the outside. A visualisation of what each eye now sees in this scenario can be found in Figure~\ref{fig3}.
+One way to solve this problem could be to transport each eye separately whenever it passes through the portal, but that idea conflicts with one of the performance optimisations we will discuss in the next section.
 
-\begin{figure}
+If we want to keep the optimisations we introduce, a different method of making sure the transition through a portal is unnoticeable by the user, no matter how slowly they move or where they look. This method takes advantage of back-face culling\footnote{A triangle in computer graphics is generally considered to have a front side and a backside, determined by whether the points of the triangle appear in clockwise or counter-clockwise order from where they are viewed. Jeon et al.\cite{back-face-culling:2007} show how back-face culling reduces the amount of drawn triangles by only drawing the triangles that are facing the camera.}.
+
+Instead of the portal being a plane, we will have it be a box. Every surface of the box is shaded with the view from the other room. If one eye clips inside the portal box, it will still see the other sides of the box that show the next room, instead of what was behind the portal originally.
+
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/backface-culling.png}
-	\caption{The sides of the portal box are only visible from the side with a solid red line. Therefore, the right eye can see space $B$ on the inner wall and at the same look out from the portal box back into space $A$.}
-	\label{fig3}
+	\caption{The sides of the portal box are only visible from the side with a solid red line. Therefore, the right eye can see space $B$ on the inner wall and simultaneously look out of the portal box back into space $A$.}
+	\label{fig:unobtrusive}
 \end{figure}
 
+The final problem is that if this eye wants to look back out from the portal box into the old room, it should not see the portal plane. This is where we use back-face culling to make sure the front side of the portal box is only visible from the outside. A visualisation of what each eye sees in this scenario can be found in Figure~\ref{fig:unobtrusive}.
+
+\section{Performance Impact}
+To better understand the impact rendering multiple portals in VR has, we created a test application with three pairs of connected portals. With this test scene, we can measure the performance impact of rendering one, two or three pairs of portals and compare it to the baseline of no portals. An illustration of this test scene can be found in Figures \ref{fig:test-scene-0} through \ref{fig:test-scene-6}.
+
+\begin{figure}[ht]
+	\includegraphics[width=\linewidth]{assets/test-scenes/0-portals.png}
+	\caption{The test scene with no portals enabled.}
+	\label{fig:test-scene-0}
+\end{figure}
+
+\begin{figure}[ht]
+	\includegraphics[width=\linewidth]{assets/test-scenes/2-portals.png}
+	\caption{The connected blue and green portals are enabled.}
+	\label{fig:test-scene-2}
+\end{figure}
+
+\begin{figure}[ht]
+	\includegraphics[width=\linewidth]{assets/test-scenes/4-portals.png}
+	\caption{The test scene with four portals enabled.}
+	\label{fig:test-scene-4}
+\end{figure}
+
+\begin{figure}[ht]
+	\includegraphics[width=\linewidth]{assets/test-scenes/6-portals.png}
+	\caption{The test scene with all six portals enabled.}
+	\label{fig:test-scene-6}
+\end{figure}
+
+We measured the rendering performance of this test scene four times, each time with a different number of active portal pairs. The tests were all performed on an Nvidia GeForce GTX 1070 with the only changes between the tests being the number of active portals. The results of our tests can be found in the following table:
+
+\vspace{5mm}
+\begin{tabular}{l|l|l|l|l}
+	\textbf{Portals} & \textbf{0} & \textbf{2} & \textbf{4} & \textbf{6}
+	\csvreader{assets/measurements.csv}{}
+	{\\\hline\csvcoli&\csvcolii&\csvcoliii&\csvcoliv&\csvcolv}
+	\caption{}
+	\label{tab:measurements}
+\end{tabular}
+\vspace{5mm}
+
+The measured values were the average frame rate of the application in frames per second, the average time taken for the whole frame and the individual times in the main and render threads on the CPU and the GPU render thread.
+
+Since our focus is on the performance impact of rendering the portals, we focus on the frame rate and the GPU frame time. Those two measurements are visualised in Figure \ref{fig:measurements}.
+
+\begin{center}
+	\begin{figure}[ht]
+		\includegraphics[width=.75\linewidth]{assets/measurements.png}
+		\caption{}
+		\label{fig:measurements}
+	\end{figure}
+\end{center}
+
+We note that rendering only a single pair of portals already incurs a large cost, as seen in the decrease of the frame rate by approximately $37\%$ from $44.7$ fps down to $27.9$ fps. The almost $50$ fold increase in GPU frame time suggests that this performance loss happens mostly on the GPU. The baseline test without any portals does not use any textures. Ji et al.\cite{lod:2005} mention the performance cost of pushing texture data to the GPU, which explains the increased frame time when using textures to render the portals.
+
+Drawing the scene without portals can be done in two render passes, one for each eye. Every added portal requires drawing the scene again from each eye, thus a single portal pair increases the number of passes from two to six. With three pairs of portals, the scene requires a total of 14 render passes. The graph in Figure \ref{fig:measurements} shows this growth of the frame time.
+
 \section{Optimisation Considerations}
-Each portal requires rendering an additional viewpoint. When rendering the portals in VR, where each eye is rendered by its own camera, there are now two additional viewpoints to render from. In the naive case where each viewpoint is rendered the same, we produce quadruple the amount of work compared to a basic non-VR scene without portals. We will present two optimisations that can reduce the rendering time and analyse their impact.
 
-The first optimisation --- using the stencil buffer to only render what is seen through the portal --- is concerned with improving the rendering time of portals in general.
+\begin{figure}[ht]
+	\includegraphics[width=\linewidth]{assets/render-steps/combined.png}
+	\caption{This figure shows how the left eye of our test scene is being rendered. It begins by rendering the view from the blue portal (top left), followed by the geometry of the blue platform (top right). Similarly, the rest of the portals are rendered first (bottom left) and the platform they are on second (bottom right).}
+	\label{fig:render-steps}
+\end{figure}
 
-In contrast, the second optimisation improves the overall performance of rendering VR by not pushing the whole scene to the GPU twice and rendering a texture for each eye, but rather rendering both eyes onto a single texture while only keeping a single copy of each rendered object in memory.
+Rendering each portal portal requires an additional viewpoint. In VR, where each eye is rendered by its own camera, there are now two additional viewpoints to render from per portal. In the naïve case, where each viewpoint is rendered the same, we produce quadruple the amount of work compared to a monoscopic scene without portals.
+
+Rendering the whole view from a portal can lead to a lot of wasted performance. For example, the further away a portal is, the fewer pixels on the screen it occupies. Rendering the view from the portal in full resolution can be wasteful if the portal is far from the camera or completely redundant if the user is not looking at the portal.
+
+Furthermore, the portal itself might be blocked by other geometry in front of it. Figure \ref{fig:render-steps} shows how the two portals rendered in the bottom left image are subsequently being blocked by the green bowl and the yellow cubes in the final image.
+
+We will present two optimisations that can reduce the rendering time and analyse their impact.
 
 \subsection{Stencil Buffer}
-The stencil buffer on graphics cards is used to mask out an area on the screen that will not be drawn to. If the portals are first rendered with a shader that simply marks the stencil buffer and nothing else, in a second pass, the virtual cameras behind those portals can then fill the remaining area with the view from the other side of the portals. This way, no redundant pixels are drawn and the amount of pixels that need to be drawn stays practically constant no matter how many portals are visible on the screen. An example demonstrating how a scene with two portals is split up by a stencil buffer can be found in Figure~\ref{fig4}.
+The first optimisation we consider uses the so-called stencil buffer to only render what is seen through the portal. This can improve the rendering time of portals in general.
 
-\begin{figure}
+The stencil buffer on graphics cards is used to mask out an area on the screen that will not be drawn to. If the portals are first rendered with a shader that simply marks the stencil buffer and nothing else, in a second pass, the virtual cameras behind those portals can then fill the remaining area with the view from the other side of the portals. This way, no redundant pixels are drawn and the amount of pixels that need to be drawn stays practically constant no matter how many portals are visible on the screen. An example demonstrating how a scene with two portals is split up by a stencil buffer can be found in Figure~\ref{fig:stencil-portals}.
+
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/three-portals.png}
 	\caption{The user is standing in a green room looking at two portals to a yellow and a magenta room. The amount of rendering can in this case be divided by three if the pairs of cameras only render a masked out area of their room instead of the whole picture.}
-	\label{fig4}
+	\label{fig:stencil-portals}
 \end{figure}
 
+As mentioned by Vlachos\cite{vlachos:2015}\cite{vlachos:2016}, the stencil buffer is also used to cull the pixels not visible in the VR headset. This is apparent in Figure \ref{fig:render-steps}, where a rounded off part at the bottom of the blue platform would never be visible to the user and is left out.
+
 \subsection{Single-Pass Instanced Rendering}
-The naïve way of rendering a stereoscopic image is to simply render it two times, once from the perspective of each eye. This is called multi-pass rendering and is illustrated in Figure~\ref{fig5}. The advantage of this technique is its simplicity in implementing it. Shader code from monoscopic applications can be used in multi-pass rendering without any changes since we are essentially just rendering two images per frame.
+In contrast to the stencil buffer, the second optimisation improves the overall performance of rendering VR by not pushing the whole scene to the GPU twice and rendering a texture for each eye, but rather rendering both eyes onto a single texture while only keeping a single copy of each rendered object in memory.
 
-\begin{figure}
+The naïve way of rendering a stereoscopic image is to simply render it two times, once from the perspective of each eye. This is called multi-pass rendering and is illustrated in Figure~\ref{fig:multi-pass}. The advantage of this technique is its simplicity in implementing it. Shader code from monoscopic applications can be used in multi-pass rendering without any changes since we are essentially just rendering two images per frame.
+
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/multi-pass.png}
 	\caption{When rendering in multiple passes, all work is doubled. This illustration shows how basically the three objects are treated as though they were separate objects\cite{nvidia-sps}.}
-	\label{fig5}
+	\label{fig:multi-pass}
 \end{figure}
 
-This method comes with a large performance downside though. When rendering portals, the amount of work might in the worst case be multiplied by the number of portals. Thus, the rendering overhead caused by multi-pass rendering is in this case not acceptable.
+This method comes with a large performance downside. In the worst case, the amount of rendering work is multiplied by the number of portals. Thus, the rendering overhead caused by multi-pass rendering is in this case not acceptable.
 
-The second method described in Figure~\ref{fig6} --- single-pass instanced rendering --- improves upon this by rendering both the left and right eyes onto the same texture. Additionally, the geometry only exists on the GPU once, as well as any shadow maps and other view-independent data. The objects are rendered twice as different instances from the two perspectives of the eyes.
+The second method described in Figure~\ref{fig:single-pass} --- single-pass instanced rendering --- improves upon this by rendering both the left and right eyes onto the same texture. Additionally, the geometry only exists on the GPU once, as well as any shadow maps and other view-independent data. The objects are rendered twice as different instances from the two perspectives of the eyes.
 
-\begin{figure}
+\begin{figure}[ht]
 	\includegraphics[width=\linewidth]{assets/single-pass.png}
 	\caption{When rendering in an instanced single pass, the whole geometry of the scene is processed only once, which significantly reduces the amount of work\cite{nvidia-sps}.}
-	\label{fig6}
+	\label{fig:single-pass}
 \end{figure}
 
 Single-pass instanced rendering should greatly improve the performance, but as mentioned in Section~\ref{unobtrusive-transitions} rendering both eyes in one pass eliminates the possibility of transporting them through the portal separately from each other.
 
-\section{Performance Comparison}
+\section{Conclusion}
+
+We have shown how to implement portals in VR that allow for an unobtrusive transition through them. This can help immersion in an impossible space of artificially increased size.
+
+The tests with our implementation of portals shows that it is very expensive to render each pair of portals. Currently, a scene should be limited to having as few portals visible as possible. In an experience where the user progresses through a series of rooms in a linear fashion, the amount of portals could be limited by disabling portals of previously visited rooms as soon as the next one is entered.
+
+The mentioned performance optimisations might help making a VR experience that has many portals visible at the same time possible. Using the stencil buffer to render only the necessary parts of each portal could in theory allow for an unlimited amount of portals shown at the same time. We showed that the amount of pixels rendered in this case stays practically constant.
 
 \bibliographystyle{plain}
 \bibliography{refs}