Continuing the ongoing series of interviews on fantasy user interfaces, it’s my pleasure to welcome Ryan Uhrich. In this interview he talks about what it takes to create engaging and compelling screen graphics for the imagined worlds of feature film and episodic productions, and what considerations go into finding the right balance between the their technical nature and the demands of the overall story around them. In between and around, Ryan dives deeper into his work on “Star Trek: Beyond” and “Altered Carbon”.

Kirill: Please tell us about yourself and the path that took you to do screen graphics for film and TV.

Ryan: I was one of those kids who daydreamed… a lot. I would doodle and draw during most of my classes. But, in high school, I started to realize that art class was something I was actually pretty good at. It was around this time that I saw a computer lab running a brand new copy of 3D Studio v3 for DOS (now called 3DS Max). I saw a computer generated sphere on the screen and I was awestruck. I remember thinking it looked so realistic! haha. This moment sparked a lifelong passion – but at the time I didn’t know how far it would take me. In the early nineties, computers were quite primitive and art class was where you went to avoid doing the “real” classes. Fortunately for me, I got to make a career out of it.

After high school, I taught myself how to make graphics with Photoshop and make web pages in HTML. The internet was brand spanking new and it was calling me in a big way. It satisfied my excitement for art and computers and I quickly found companies wanting to pay me to make stuff for them. From there, I got a job at a multimedia company converting old text manuals to interactive 2D & 3D lessons for Bombardier aircrafts. I started to learn Flash and I was a huge fan. I loved how it combined art & code and it gave me my first taste of animation. Eventually, by the mid 2000’s I enrolled in Grant MacEwan College for their Design Foundations program and subsequently went to Vancouver Film School for the Digital Design program, specializing in motion graphics.

Right after graduation, I moved to Copenhagen, Denmark to work at a studio called Thank You, where we created TV commercials. It was pretty nerve-racking to leave my home in Canada and venture off for my career, but I would definitely recommend it. It was an amazing opportunity and ended up providing a lot of experience, both personally and professionally. Another move to Sydney, Australia meant working for MTV, Z Space, and Collider. After three years abroad I somehow ended up back in Vancouver and was provided with the opportunity to work on an upcoming film called Ender’s Game. I instantly jumped on the offer because I was passionate about the aesthetics of UI and screen graphics and it felt like the right path.

Screen graphics for “Altered Carbon” under the creative direction of G Creative. Courtesy of Ryan Uhrich.

Kirill: Looking back at your first production, what was the most unexpected part of working in film?

Ryan: I was accustomed to tight deadlines from advertising and conceptualizing on the fly. However, the software pipeline was quite different and I found myself struggling to convert the 3D camera data from Maya to Cinema 4D. I remember spending a lot of time with those technicalities rather than designing and animating – which was what I was hired to do. Unfortunately, all these years later, VFX to motion graphics pipelines can still be problematic. I hope someday it will get easier. Haha.

Screen graphics for “Altered Carbon” under the creative direction of G Creative. Courtesy of Ryan Uhrich.

Kirill: How do you craft something compelling that keeps the viewer in the story?

Ryan: The largest component to screen graphics is storytelling. Our goal is to provide a tool for the director to convey important information to the audience at the right moment. It should communicate quickly and move the story forward without distracting the viewer. The second layer is the mood or feeling we are trying to elicit from the audience – it should harmonize with the narrative and not feel out of place. Once those are dialled we spend time thinking about all the fun stuff like plausibility and sexiness.

Screen graphics for “Altered Carbon” with Chris Cooper under the creative direction of G Creative. Courtesy of Ryan Uhrich.

Continue reading »

There’s a bunch of helper extension methods that the Kotlin standard library provides for working with collections. However, it would seem that at the present moment <a href="https://kotlinlang.org/api/latest/jvm/stdlib/kotlin.collections/java.util.-enumeration/index.html">java.util.Enumeration</a> has been left a bit behind. Here is a simple extension method to convert any List to a matching Enumeration:

/**
 * Extension function for converting a {@link List} to an {@link Enumeration}
 */
fun <T> List<T>.toEnumeration(): Enumeration<T> {
    return object : Enumeration<T> {
        var count = 0

        override fun hasMoreElements(): Boolean {
            return this.count < size
        }

        override fun nextElement(): T {
            if (this.count < size) {
                return get(this.count++)
            }
            throw NoSuchElementException("List enumeration asked for more elements than present")
        }
    }
}

And here is how you can use it to expose the local file system to the JTree component. First, we create a custom implementation of the TreeNode interface:

data class FileTreeNode(val file: File?, val children: Array<File>, val nodeParent: TreeNode?) : TreeNode {
    constructor(file: File, parent: TreeNode) : this(file, file.listFiles() ?: arrayOf(), parent)
    constructor(children: Array<File>) : this(null, children, null)

    init {
        children.sortWith(compareBy { it.name.toLowerCase() })
    }

    override fun children(): Enumeration<FileTreeNode> {
        return children.map { FileTreeNode(it, this) }.toEnumeration()
    }

    override fun getAllowsChildren(): Boolean {
        return true
    }

    override fun getChildAt(childIndex: Int): TreeNode {
        return FileTreeNode(children[childIndex], this)
    }

    override fun getChildCount(): Int {
        return children.size
    }

    override fun getIndex(node: TreeNode): Int {
        val ftn = node as FileTreeNode
        return children.indexOfFirst { it == ftn.file }
    }

    override fun getParent(): TreeNode? {
        return this.nodeParent
    }

    override fun isLeaf(): Boolean {
        val isNotFolder = (this.file != null) && (this.file.isFile)
        return this.childCount == 0 && isNotFolder
    }
}

Note a few language shortcuts that make the code more concise than its Java counterparts:

• Since File.listFiles() can return null, we wrap that call with a simple Elvis operator: file.listFiles() ?: arrayOf().
• The initializer block sorts the File children in place by name.
• To return tree node enumeration in children(), we first map each File child to the corresponding FileTreeNode and then use our extension function to convert the resulting List to Enumeration.
• Looking up the index of the specific node is done with the existing extension indexOfFirst function from the standard library.

Now all is left to do is to create our JTree:

val tree = JTree(FileTreeNode(File.listRoots()))
tree.rootVisible = false

Note: code samples in this post are kept in sync with the latest stable Kotlin coroutines release. All the samples have been verified to work with release 1.10.2. Also see the official guide to UI programming with coroutines.

Kotlin 1.3 has been released, and one of the major additions to the language is the official graduation of coroutines from experimental to stable. Let’s take a look at how we can replace the old and venerable SwingWorker with something that is a bit more modern.

In the simplest use case, you do some kind of long-running work in your doInBackground() and then process the result of that work in done():

object : SwingWorker<List<StringValuePair>, Void>() {
    @Throws(Exception::class)
    override fun doInBackground(): List<StringValuePair>? {
        return bar.callback.getLeafs(newPath)
    }

    override fun done() {
        try {
            filePanel.setFolder(get())
        } catch (exc: Exception) {
        }
    }
}.execute()

There’s a whole bunch of noise, so to speak, around the core of the logic – calling breadcrumb bar’s callback to get the leaf content for the newly set path, and then populating the folder based on the retrieved content. Let’s see how this code can look like with coroutines:

filePanel.setFolder(GlobalScope.async {
    bar.callback.getLeafs(newPath)
}.await())

In this case, we have distilled the core of the logic flow to its essence – asynchronous loading of the leaf content, followed by updating the UI. Another option is to collapse the async / await block into a single withContext to change the execution context away from the UI thread:

filePanel.setFolder(withContext(Dispatchers.Default) {
    bar.callback.getLeafs(newPath)
})

Note that once you switch to coroutines, you also need a larger context for proper synchronization. In Swing, it means wrapping the entire listener with GlobalScope.launch(Dispatchers.Swing):

// Configure the breadcrumb bar to update the file panel every time
// the path changes
bar.model.addPathListener {
    GlobalScope.launch(Dispatchers.Swing) {
        val newPath = bar.model.items
        if (newPath.size > 0) {
            // Use the Kotlin coroutines (experimental) to kick the
            // loading of the path leaf content off the UI thread and then
            // pipe it back to the UI thread in setFolder call.
            filePanel.setFolder(withContext(Dispatchers.Default) {
                bar.callback.getLeafs(newPath)
            })
        }
    }
}

Now let’s take a look at something a bit more interactive – updating the UI on the ongoing progress of a long-running background task.

Let’s add a button and a label to a sample frame, and kick off a 5-second task that updates the UI every second on the progress using SwingWorker.process():

val button = JButton("Start operation!")
val status = JLabel("Progress")

frame.add(button)
frame.add(status)

button.addActionListener {
    class MyWorker : SwingWorker<Unit, Int>() {
        override fun doInBackground() {
            for (i in 1..5) {
                publish(i)
                Thread.sleep(1000)
            }
        }

        override fun process(chunks: MutableList<Int>?) {
            status.text = "Progress " + chunks?.joinToString()
        }

        override fun done() {
            status.text = "Done!"
        }
    }

    val worker = MyWorker()
    worker.execute()
}

The first way to convert to coroutines would be with the help of channels:

button.addActionListener {
    GlobalScope.launch(Dispatchers.Swing) {
        val channel = Channel<Int>()
        GlobalScope.launch {
            for (x in 1..5) {
                println("Sending $x " + SwingUtilities.isEventDispatchThread())
                // This is happening off the main thread
                channel.send(x)
                // Emulating long-running background processing
                delay(1000L)
            }
            // Close the channel as we're done processing
            channel.close()
        }
        // The next loop keeps on going as long as the channel is not closed
        for (y in channel) {
            println("Processing $y " + SwingUtilities.isEventDispatchThread())

            status.text = "Progress $y"
        }
        status.text = "Done!"
    }
}

Note the usage of Dispatchers.Swing context that is passed to the GlobalScope.launch() function and the wrapping of the emulated long-running task in another GlobalScope.launch lambda. Then, as long as that lambda keeps on sending content into the channel, the for loop iterates over the channel content on the UI thread, and then relinquishes the UI thread so that it is no longer blocked.

Now let’s make it a little bit more structured. While code samples found in documentation and tutorials run on the lighter side of things (including this article), real-life apps would have more complexity on both sides of async processing. We’re going to split this logic into two parts – background processing of data and updating the UI on the main thread. Background processing becomes a separate function that returns a data channel:

fun process() : ReceiveChannel<Int> {
    val channel = Channel<Int>()
    GlobalScope.launch {
        for (x in 1..5) {
            println("Sending $x " + SwingUtilities.isEventDispatchThread())
            // This is happening off the main thread
            channel.send(x)
            // Emulating long-running background processing
            delay(1000L)
        }
        // Close the channel as we're done processing
        channel.close()
    }
    return channel
}

And UI code consumes the data posted to the channel and updates the relevant pieces:

button.addActionListener {
    GlobalScope.launch(Dispatchers.Swing) {
        // The next loop keeps on going as long as the channel is not closed
        for (y in process()) {
            println("Processing $y " + SwingUtilities.isEventDispatchThread())

            status.text = "Progress $y"
        }
        status.text = "Done!"
    }
}

Let’s make the background task cancellable so that the currently running operation can be safely canceled without subsequent erroneous UI updates. Background processing returns an object that has two parts – data channel and a job that can be canceled if needed:

class ProcessResult(val resultChannel: ReceiveChannel<Int>, val job: Job)

fun processCancelable() : ProcessResult {
    val channel = Channel()
    val job = GlobalScope.launch {
        for (x in 1..5) {
            if (!isActive) {
                // This async operation has been canceled
                break
            }
            println("Sending $x " + SwingUtilities.isEventDispatchThread())
            // This is happening off the main thread
            channel.send(x)
            // Emulating long-running background processing
            delay(1000L)    
        }
        // Close the channel as we're done processing
        channel.close()
    }
    return ProcessResult(channel, job)
}

And on the UI side of things, we keep track of the last job that was kicked off and cancel it:

var currJob: Job? = null

button.addActionListener {
    GlobalScope.launch(Dispatchers.Swing) {
        currJob?.cancel()

        val processResult = processCancelable()
        currJob = processResult.job

        // The next loop keeps on going as long as the channel is not closed
        for (y in processResult.resultChannel) {
            println("Processing $y " + SwingUtilities.isEventDispatchThread())

            status.text = "Progress $y"
        }
        status.text = "Done!"
    }
}

Finally, we can move away from the existing concept of data communication “pipe” between the two parts, and start thinking in terms of passing a lambda to be invoked by the data producer when it has completed processing the next chunk of data. In this last example, the producer marks itself with the suspend keyword and uses the parent context so that cancellation is properly propagated:

suspend fun processAlternative(job : Job, progress: (Int) -> Unit = {}) {
    for (x in 1..5) {
        // Emulating long-running background processing.
        // Use the parent job so that cancellation of the parent propagates in here.
        GlobalScope.async(context=job) {
            println("Running on " + SwingUtilities.isEventDispatchThread())
            delay(1000L)
        }.await()
        // And calling the callback on the UI thread
        println("Sending $x " + SwingUtilities.isEventDispatchThread())
        progress(x)
    }
}

And the UI side of things supplies a lambda that updates the relevant UI pieces:

var currJob: Job? = null

button.addActionListener {
    currJob?.cancel()

    currJob = GlobalScope.launch(Dispatchers.Swing) {
        // This will run until all the sequential async blocks are done
        processAlternative(currJob!!) { progress ->
            println("Processing $progress " + SwingUtilities.isEventDispatchThread())

            status.text = "Progress $progress"
        }
    status.text = "Done!"
    }
}

If you want start playing with replacing SwingWorker with coroutines, add org.jetbrains.kotlinx:kotlinx-coroutines-swing and org.jetbrains.kotlinx:kotlinx-coroutines-core to your dependencies. The latest version for both of the modules is 1.10.2.

Continuing the ongoing series of interviews on fantasy user interfaces, it’s my pleasure to welcome Gladys Tong. She started her company G Creative Productions about 20 years ago, and since then the studio’s work was featured in a wide array of productions such as I, Robot, Fantastic Four, 2012, Watchmen, Man of Steel, Ender’s Game, Elysium, Chappie, Godzilla, Altered Carbon, The Cloverfield Paradox, Batman v Superman and Star Trek Beyond.

Screen graphics for Batman v Superman: Dawn of Justice. Courtesy of G Creative and Warner Bros.

Kirill: Please tell us about yourself and your path so far

Gladys: My name is Gladys Tong and I run a small company called G Creative Productions. I have always loved art and science so when I discovered that I could make a living combining the two, I jumped in with both feet! When I started 20 years ago, there were not a lot of people doing what I do today so the path for me has been relatively uncharted. It has led me to an interesting career and some wonderful people whom I have had the good fortune to work with.

Kirill: What drew you into the field of interactive motion graphics, and how has that changed since you’ve started working professionally in this field?

Gladys: As I mentioned, I was always interested in art and science. From drawing at an early age to learning about multimedia with computers it was a natural evolution for me to be drawn to interactive motion graphics. Twenty years ago most people including my friends and family did not understand what I did. Now with more exposure through films and tv shows as well as online access to information it has exploded and now there are more people and companies in this field. It’s an exciting time.

Screen graphics for X-Men: Last Stand. Courtesy of G Creative and 20th Century Fox.

Kirill: What can you tell us about G Creative?

Gladys: I started the company in Canada and chose the name in a hurry. I wanted something short and simple – just a letter but wasn’t allowed to incorporate a company with just one letter of the alphabet so I had to come up with a longer name. Many people assume that the G stands for Gladys but it actually represents more than me – graphics, geek, gear, green (for green screen) to name a few. I like the ambiguity and endless possibilities. Much like what we do there are no right answers to everything but many possible answers.

G Creative’s mission statement is to fuse compelling imagery with targeted technology to present a story that inspires, entertains, and informs an audience. This represents a philosophy that I try very hard to achieve in practice through the work we do both in hardware and software services as well as the people I work with. I have a great collective of talented artists, designers, and technicians whom I am thankful for.

Screen graphics for The A-Team. Courtesy of G Creative and 20th Century Fox.

Kirill: When you meet a new person and they ask you what you do for a living, how do you describe it?

Gladys: When I meet a person that I don’t know I tend to describe what I do with words that are more general and easy to comprehend. So I tend to say computer graphics as that seems to be received with nods of understanding. If the conversation goes further, I usually elaborate with the film industry which yields similar nods of familiarity. If I actually describe what I do, I get perplexed facial expressions that suggest a need to decipher this cryptic job I have.

What I do for a living is not as common so people have a harder time grasping the fact that there is a job like this and that it involves a seemingly unusual combination of tech and creativity. Even people that I work with within the film industry sometimes find what we do strange or surprising. In a world that’s increasingly converging I think that will be less so.

Screen graphics for Robocop. Courtesy of G Creative and Columbia.

Continue reading »