Rendering Your Light Field

⇦ Creating iOS App    HOME    Interlacing ⇨


Now that you have a working game template, you are ready to render a light field.


Note: How to Render a Light Field

A light field is simply the set of all the rays of light within a space. We obviously can’t compute all the light rays; we will have to make do with a discrete subset of them. How do we pick which rays to compute?

The technique we will use in this tutorial is to compute the colors your eyes should see when looking at the scene from several different positions. Then we will attempt to arrange those colors on the screen under the parallax barrier, so that they can only be seen when your eye is near the positions from which those colors were generated.

The first step is to figure out which color should be emitted from each pixel on a screen to generate the same light rays to your eye as you would see if you were to view the real scene. For this tutorial, we will use a technique called generalized perspective projection. Using this technique, you will construct a projection matrix to map the 3D points in the scene onto the screen.


To render our light field, we will first render our scene from several different viewpoints, drawing each view to a small rectangle on a screen-sized buffer. Then we will interlace those views so that the proper pixels are always visible from behind the parallax barrier depending on the viewer’s perspective.

Software Architecture:

From a bird’s eyeview, your software architecture pipeline would look like this. The class diagram below demonstrates the organization of and interaction between various classes:

Software Architecture Class Diagram represents the Organization and Interaction of Classes and APIs

Software Architecture Class Diagram represents the Organization and Interaction of Classes and APIs

  1. LightFieldViewController Sets up the scene graph and the rendering technique.
  2. SCNView Is a user interface widget that performs the actual rendering.
  3. SCNTechnique Describes the rendering passes required to draw the scene from 25 viewpoints, then interlace.
    • metalFragmentShader describes the interlace logic used by technique for rendering
  4. SCNNode Represents a node in the scene graph. (each camera and light are nodes in the scene graph)
  5. OffAxisCameraNode Renders each viewpoint using a custom projection matrix which is defined inside it.
  6. LightFieldRenderer Updates all the viewpoint positions and projections at each frame.

Creating the SCNTechnique

In order to render our views, we will make use of SceneKit’s SCNTechnique class. This class defines the sequence of passes that the SceneKit renderer should perform each frame.

To make use of SCNTechnique, we must construct a dictionary, a set of key-value pairs, describing the passes we wish to perform, the inputs and outputs of each, and the sequence in which they should be executed. It’s worth taking a look at the documentation for SCNTechnique to get a better understanding of the dictionary format.

We will be rendering our scene from 25 different viewpoints. So we want to generate a final dictionary that looks like this:

["symbols": {
    superviewSizeX =     {
        type = int;
    };
    superviewSizeY =     {
        type = int;
    };
    viewsCountX =     {
        type = int;
    };
    viewsCountY =     {
        type = int;
    };
}, "targets": {
    multiviewTexture =     {
        type = color;
    };
}, "passes": {
    interlace =     {
        colorStates =         {
            clear = false;
        };
        draw = "DRAW_QUAD";
        inputs =         {
            multiviewTexture = multiviewTexture;
            superviewSizeX = superviewSizeX;
            superviewSizeY = superviewSizeY;
            viewsCountX = viewsCountX;
            viewsCountY = viewsCountY;
        };
        metalFragmentShader = LightFieldFragment;
        metalVertexShader = LightFieldVertex;
        outputs =         {
            color = COLOR;
        };
        program = Interlace;
    };
    view0 =     {
        colorStates =         {
            clear = 1;
        };
        draw = "DRAW_SCENE";
        outputs =         {
            color = multiviewTexture;
        };
        pointOfView = camera0;
        viewport = "0 0 248 441";
    };
    view1 =     {
        colorStates =         {
            clear = 0;
        };
        draw = "DRAW_SCENE";
        outputs =         {
            color = multiviewTexture;
        };
        pointOfView = camera1;
        viewport = "248 0 248 441";
    };

    ...

    view24 =     {
        colorStates =         {
            clear = 0;
        };
        draw = "DRAW_SCENE";
        outputs =         {
            color = multiviewTexture;
        };
        pointOfView = camera9;
        viewport = "992 441 248 441";
    };
    viewTemplate =     {
        draw = "DRAW_SCENE";
        outputs =         {
            color = multiviewTexture;
        };
    };
}, "sequence": (
    view0,
    view1,

    ...

    view24,
    interlace
)]
  • We define our symbols. These are the parameters that we will be passing in to our interlacing shader.
  • We define the buffer where the 25 views will be drawn to be a buffer for storing color.
  • We describe each rendering pass. The order of these definitions doesn’t matter; they will be executed in the order defined by the sequence array.
  • Each pass defines whether the buffer it is drawing to should be cleared first: for the first view, yes; for each subsequent view and the interlacing pass, no.
  • Each pass defines what should be drawn: for the views, the whole scene; for the interlacing pass, a single rectangle filling the screen, textured with the buffer containing all the views.
  • Each pass defines to what output it should be drawn: for the views, the intermediate buffer defined in “targets”; for the interlacing pass, “COLOR”, or the actual color that will be shown on the screen.

The view passes additionally define the point of view from which the scene should be rendered. Each one lists the name of a different camera which we must place in the scene. The view passes also define the pixel coordinates of the rectangle to which the scene should be drawn. Each view is drawn to 1/25th of the buffer.

The interlacing pass defines a set of inputs; these are the symbols we saw earlier, as well as the intermediate buffer filled by the view passes. The technique’s symbol names are mapped to the variable names to be used in the pass’s shaders. The interlacing pass also provides names for a Metal fragment shader, a Metal vertex shader, and an OpenGL program (which contains both a vertex and a fragment shader). We are required to provide a dummy OpenGL program even though we are only using Metal.

After defining all our passes, we define their order in “sequence”.

We need to pass this dictionary into the constructor of SCNTechnique. We could create a JSON or PLIST file containing the entire dictionary, but the 25 views contain a lot of redundancy. We opted to create a PLIST file which includes only a single view pass containing the information shared across views, and omits the sequence list. We read this PLIST file into our app and generate the 25 views and the sequence programatically. This way, we can easily adjust the number of views without recreating the technique dictionary each time.

To create a PLIST file, create a new file from the File menu, and select iOS > Resource > Property List.Create New File

Create Property List

Fill the property list with our data.

SCNTechnique Property List

Now go back to the viewDidLoad() function of your GameViewController class and add the following at the bottom:

    let techniquePath = NSBundle.mainBundle().pathForResource("LightFieldTechnique", ofType: "plist")!
    var techniqueDescription = NSDictionary(contentsOfFile: techniquePath) as! [String : AnyObject]

The techniqueDescription is a var so that we can dynamically add our views to it.

You should use proper error handling such as a guard statement to make sure your code doesn’t crash if the file load fails instead of implicitly unwrapping the optionals using the exclamation mark; however, for simplicity we will omit it here.

Then you can get the passes dictionary and the viewTemplate pass:

var passes = techniqueDescription[passesKey] as! [String : AnyObject]
let viewTemplate = passes[viewTemplateKey] as! [String : AnyObject]

Once again we use the var keyword for passes because we will be updating it.

Now we create a loop to iterate over all our view positions and define a pass for each, assuming we have already defined values for viewsCountY, viewsCountX, viewSizeY, viewSizeX, headPosition, and interViewpointDistance. We could have constants defined for these values, or loaded them from another PLIST file.

LightFieldProperties PLIST

We calculate the view sizes by dividing the superview size (the size of the entire screen) by the number of views along each axis.

    let superviewSizeX = Int(view.frame.width) * Int(view.contentScaleFactor);
    let superviewSizeY = Int(view.frame.height) * Int(view.contentScaleFactor);
    let viewSizeX = superviewSizeX / viewsCountX
    let viewSizeY = superviewSizeY / viewsCountY

iOS provides sizes in resolution-independent points instead of pixels, so we have to use contentScaleFactor to get the actual pixel resolution.

Ignore the construction of the projection matrix and the screen properties for now. We will discuss them in the next section. Let’s look at what determines the inter-camera distance before we get into setting up the cameras for all the views:


Note: Selecting a Head Position and Inter-Viewpoint Distance

The specific head position you pick doesn’t matter. Choose a starting viewpoint that is comfortable for viewers of your display. Then you can calculate your head box size using the formula  f/z = p/x, where f is the distance between your parallax barrier and the pixels on the screen, p is the width of the pixels under a single pinhole, z is the distance of your head from the screen, and x is the width of your head box.

SuperpixelSideView

Divide your head box size by the number of views horizontally and vertically to get your inter-viewpoint distance.


Now, we loop over all the views in x and y dimensions and create and position cameras while setting the proper projection matrix by passing in the function that we discuss in the next section:

    var sequence = [String]()
    // the multiview texture should only be cleared in the first pass
    var cleared : Bool = false
    for y in 0..<viewsCountY {
        for x in 0..<viewsCountX {
            let viewNumber = y * viewsCountX + x
            var viewPass = viewTemplate;
            viewPass["viewport"] = "\(x * viewSizeX) \(y * viewSizeY) \(viewSizeX) \(viewSizeY)"
            
            // create and add a camera to the scene
            let cameraNode = SCNNode()
            cameraNode.camera = SCNCamera()

            // place the camera
            let cameraX = Float(x - Int(viewsCountX / 2)) * interViewpointDistance
            let cameraY = Float(y - Int(viewsCountY / 2)) * interViewpointDistance
            cameraNode.position = SCNVector3(x: headPosition.x + cameraX, y: headPosition.y + cameraY, z: headPosition.z)
            
            // set the camera projection transform
            cameraNode.camera!.setProjectionTransform(
                computeOffAxisPerspectiveProjectionMatrix(screenSize,
                    screenPosition: screenPosition,
                    screenOrientation: screenOrientation,
                    cameraPosition: cameraNode.position,
                    near: Float(near), far: Float(far)))
            let cameraName = "camera\(viewNumber)"
            cameraNode.name = cameraName
            scene.rootNode.addChildNode(cameraNode)
            
            viewPass["pointOfView"] = cameraName
            viewPass["colorStates"] = ["clear" : !cleared]
            if (!cleared) {
                cleared = true;
            }
            let viewName = "view\(viewNumber)"
            passes[viewName] = viewPass
            sequence.append(viewName)
        }
    }
    sequence.append("interlace")
    techniqueDescription["passes"] = passes
    techniqueDescription["sequence"] = sequence

Once the dictionary is populated, we can pass it in to the constructor of SCNTechnique and then assign the newly constructed object to our SCNView instance. Finally, we assign the values of the symbols to be used in the shader.

    let technique = SCNTechnique(dictionary:techniqueDescription)
    scnView.technique = technique
    scnView.technique?.setObject(superviewSizeX, forKeyedSubscript: "superviewSizeX");
    scnView.technique?.setObject(superviewSizeY, forKeyedSubscript: "superviewSizeY");
    scnView.technique?.setObject(viewsCountX, forKeyedSubscript: "viewsCountX");
    scnView.technique?.setObject(viewsCountY, forKeyedSubscript: "viewsCountY");

Note: Understanding Off-Axis Perspective Projection:

Unlike traditional perspective projection, in which the viewpoint is assumed to be in front of the center of the screen, we need to be able to generate a proper projection from any viewpoint. The traditional perspective projection would incorrectly give rise to the “toe-in” stereo effect. Please see Robert Kooima’s excellent paper on generalized perspective projection to gain a better understanding of the off-axis perspective projection matrix we will be generating here.

Off-Axis camera projection allows for the images from all the viewpoints to be centered about the central reference viewpoint, without introducing any perspective distortion.

Off-Axis camera projection allows for the images from all the viewpoints to be centered about the central reference viewpoint, without introducing any perspective distortion.

In a real scene capture, off-axis projection is the process of registering the image captured from a differnet viewpoint without any camera rotation to the central reference viewpoint.

OffAxis_Perspective1


Computing An Off-Axis Perspective Projection

To compute the correct perspective projection matrix when your viewpoint is not necessarily at the center of the screen, you need to figure out the distance from your viewpoint projected onto the screen plane to each of the edges of the screen. These four values, combined with near and far clipping planes (use values that ensure the content you want to render is between near and far) can be used to compute an off-axis frustum.

SceneKit doesn’t have much capability to do vector and matrix calculations, but GLKit has considerably more. So we can compute our projection by converting our SCNVectors and SCNMatrixs to their corresponding GLKit types.

    func computeOffAxisPerspectiveProjectionMatrix(screenSize: CGSize, screenPosition: SCNVector3, screenOrientation: SCNQuaternion, cameraPosition: SCNVector3, near: Float, far: Float) -> SCNMatrix4 {
        
        let rightVector = GLKVector3Make(1, 0, 0)
        let upVector = GLKVector3Make(0, 1, 0)
        let forwardVector = GLKVector3Make(0, 0, -1)
        
        let translateWorldFromScreen = GLKMatrix4TranslateWithVector3(GLKMatrix4Identity, SCNVector3ToGLKVector3(screenPosition))
        let rotateWorldFromScreen = GLKMatrix4MakeWithQuaternion(GLKQuaternionMake(screenOrientation.x, screenOrientation.y, screenOrientation.z, screenOrientation.w))
        let transformWorldFromScreen = GLKMatrix4Multiply(translateWorldFromScreen, rotateWorldFromScreen)
        
        let screenRightVector = GLKMatrix4MultiplyVector3(transformWorldFromScreen, rightVector)
        let screenUpVector = GLKMatrix4MultiplyVector3(transformWorldFromScreen, upVector)
        let screenForwardVector = GLKMatrix4MultiplyVector3(transformWorldFromScreen, forwardVector)
        
        let cameraToScreen = GLKVector3Subtract(SCNVector3ToGLKVector3(screenPosition), SCNVector3ToGLKVector3(cameraPosition))
        let screenDistance = GLKVector3DotProduct(screenForwardVector, cameraToScreen)
        
        let screenLeft = GLKVector3DotProduct(screenRightVector, cameraToScreen)
        let screenBottom = GLKVector3DotProduct(screenUpVector, cameraToScreen)
        let screenRight = screenLeft + Float(screenSize.width)
        let screenTop = screenBottom + Float(screenSize.height)
        
        let left = screenLeft * near / screenDistance
        let right = screenRight * near / screenDistance
        let bottom = screenBottom * near / screenDistance
        let top = screenTop * near / screenDistance
        
        let projectionMatrix = GLKMatrix4MakeFrustum(left, right, bottom, top, near, far)
        
        return SCNMatrix4FromGLKMatrix4(projectionMatrix)
    }

GLKit already provides a helper function for generating a projection matrix, but Kooima’s paper gives more detail about how this function works if you are interested.

Let’s make sure our code is working. comment out the line sequence.append("interlace") above and change the color output of the viewTemplate pass in our technique dictionary PLIST to “COLOR”. This will disable the interlacing pass and draw the 25 views directly to the screen. When you run the app, you should see 25 spinning ships, all from slightly different perspectives. Make sure to run this on your iPhone, not in the simulator, as it needs the performance boost from using the Metal API in order smoothly draw so many views.

Views

Once you have it working, uncomment the line adding the interlace pass to the sequence and reset the viewTemplate’s output to “multiviewTexture”.


You’re ready for the final step: interlacing.

Comments are closed.