Continuing the ongoing series of interviews with creative artists working on various aspects of movie and TV productions, it is my delight to welcome Derek Spears. Since joining the Rhythm & Hues Studios about fifteen years ago, Derek has worked on feature films such as “Red Riding Hood”, “The Mummy: Tomb of the Dragon Emperor,” “Superman Returns” and “X-Men: Days Of Future Past”. Most recently, his work as the Visual Effects Supervisor on HBO’s magnificent “Game of Thrones” has won back-to-back Emmy awards for Outstanding Visual Effects.
In this interview Derek talks about the early days of digital visual effects [VFX] and the evolution of the tools since then, the human-intensive parts of the VFX pipeline and his thoughts on what may happen in those areas in the next few years, the discussions that take place between the art department and the VFX department on building things physically, digitally or as hybrid, and the never-ending quest to take the audience on a wild ride of visual storytelling magic. The second part of the interview is all about the last two seasons of “Game of Thrones”, from creating the fire-breathing dragons to meticulously crafting the more “invisible” parts of Westeros and Essos, such as crowds and digital buildings.
Kirill: Please tell us about yourself and how you started in the field.
Derek: My background is in software engineering. I worked at Silicon Graphics for a few years in the apps development group. During my time there I worked with Kodak to help develop their compositing system, and after that I joined Cinesite to help them start their 3D graphics department. After that I worked at Digital Domain, and now I am a Visual Effects Supervisor at Rhythm & Hues on features and episodic shows. There I’ve worked on “Game of Thrones”, “Black Sails” and “Walking Dead” in the episodic world, and on features such as “R.I.P.D.”, “The Mummy: Tomb of the Dragon Emperor,” “Superman Returns” and others.
My background is in computer graphics and the technology world, but I’ve always enjoyed the intersection of art and technology that visual effects has provided.
Kirill: Would it be correct to say that you’ve joined the field when digital was starting, and there were a lot of special effects with animatronics and other physical approaches?
Derek: Digital was still in its ascendancy back then. People were still trying to understand how it worked. It was very much the Wild West, the frontier type of landscape at that point in time. There were no well-developed ideas that we have now, like pipelines, render cues and various approaches and methodologies. We were learning at the time, and it was very interesting to be a part of that.
Kirill: Do you remember how it felt on your first few productions, as you “lifted the veil” and saw how the magic of movies looks like on the inside?
Derek: It was interesting to learn how a production was done. Coming from an engineering background, filmmaking was a new thing for me. The interesting thing about those early days was that every new show you got to, you had to sit down and decide how to do it. There was no well established methodology.
We didn’t typically use ray tracing. RenderMan was very popular at the time, and there were some other choices – such as Softimage and Mental Ray. The choices then tended to be non-homogenous. The options now are more developed, but also more similar in their approach to solve the problem. It was a bit more interesting in the early days. You had to get out there and figure out how to do things.
It was a much bigger chance of failure because it hadn’t been done before. Now if somebody asks you to do something, the chances are that you’ve done something similar to it, and that it’s going to fit into the existing pipeline. We have tools for various situations, and that is not true for the early days.
Kirill: Looking at the evolution of hardware and software at your disposal, would you say that they’ve kept up with demands from the production side to keep on increasing the level of sophistication of the worlds you’re building?
Derek: There are a couple of interesting things here. There’s always the attempt to decrease the amount of human labour in any given problem. Lighting is the one that has made the most advances out of the many disciplines. It used to be a very difficult and time-consuming problem. Now you can capture light on set and then quickly recreate a very close approximation of it in render without a lot of experimentation. You still have to do a lot of work for modeling and texturing as the setup, but the actual lighting process is rather quick.
Animation and motion capture has helped, but I don’t think that it has taken a lot of labour out of animation. There are tools that are better at rigging, but it’s still an extremely labour-intensive process. The same goes for compositing. There’s a much larger toolset to make things work a lot better in a lot more difficult situations, but what we find is that complexity of the problem matches the efficiency of the tools. For everything that gets faster, the complexity is increasing. It used to take 12 hours to render a chrome sphere and now it takes 12 hours to render a dragon. It’s the same time to render, but you get a lot more for it.
I don’t think that things ever get simpler. It’s rather that the available resources increase the complexity.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: Now that there’s so much being done in the digital pipeline of VFX, is there any friction between the art department that lives in the world of physical sets, and the VFX department that is creating all these fantastical sets in pixels?
Derek: In many ways, the art department and the VFX department are quite complimentary. What the art department designs, the VFX department has to realize. In some places we’ve seen crossovers, like on “Oblivion” where the art director came from the VFX background. I haven’t seen any tension between the two departments on the productions I’ve been involved with. It’s always very cooperative, to the point of how can we involve the art department to help design and solve some of the VFX problems.
In fact, it gets so close on some of the shows. On one of the shows that we’re working on right now, we’re acting as part of the art department to help design creatures. I view it as a synergistic relationship.
If you’re talking about trade-offs, one of the things that you have to figure out is that everything costs money to do. When you design a big set of platform for your show to work on, one of the things that the two departments have to work out is what is the most cost-effective way to do that. How much are we going to do practically? How much are we going to do in VFX? In no way is that a combative relationship. Together we figure out the best way to not break everybody’s budget. Usually that works very nicely.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: Going back to the labour-intensive nature of doing VFX, would you say that the cost of doing VFX is not that much cheaper than building physical sets? How is the decision made about specific sets being physical, having digital extension or being completely green-screen?
Derek: It’s all about the cost these days. The big cost trade-off that you look at is scale and re-use. The bigger something gets, the cheaper it’s going to be able to build it digitally. However, the more often it gets re-used, the better you’d be able to amortize your physical set. If you have a set that is going to be used hundreds of times and it’s not that big, maybe it’s cheaper to build it on set. But if it’s a one-off and it’s massive, that’s probably cheaper to do digitally.
And sometimes it’s a hybrid. We may have a set where we take a lot of shots on a small section, and then as you go wider, you need to extend it and that becomes VFX. It’s an intersecting curve. You have to find where the physical cost vs. the VFX cost cross over. That’s a discussion that has been ongoing for years, and it’s something that every production works through between the art department and the VFX department. It’s a very cooperative process. Everybody wants to make sure that they maximize what they can get out of their budget. That’s something that you need to work on in every production.
Pre-visualizing the dragon landing for “Game of Thrones” by Rhythm & Hues.
Kirill: What’s your experience has been so far with doing pre-visualization? What does pre-vis bring to productions?
Derek: There are two different ways pre-vis is used. One is to help people to storyboard sequences, and the second one is technical pre-vis. The later helps answer the questions of what are we going to see if we shoot this, whether we need set extensions and where we are looking. It’s more on the technical side, and it helps quite a bit.
Going back to the discussions with the art department when you walk through all the different issues, you want to see what the costs are. You want to see where does it land to be cheaper. And sometimes the cost doesn’t matter, but you want to build something that is hugely improbable physically and it has to be VFX. It’s all about budgeting and costs, and pre-vis can help you find the answers. It can show you practically what you’re looking at.
Kirill: Do you see that productions are able to build much bigger worlds with the evolving capabilities of VFX tools?
Derek: Absolutely. Building virtual sets gives you unlimited freedom, and that’s a bit of a double edged sword. Before, where you had to build practical sets, there was a lot more planning that went into it. You had the budget for the art department, you knew what you had to shoot and what you could build, and those were the constraints.
With the advent of doing things digitally, those constraints are effectively gone. I can do whatever I want, and it’s too easy to forget that there’s the associated cost. And because you can do anything you want, people tend not to plan it out as well. Before we were very meticulous about laying it out, about what the construction would be and where the shots would be. With digital there is a freedom to not live within those constraints, and that can lead to less planning compared to doing things practically.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: On the other hand, particularly with “Game of Thrones”, you are constrained by the time you have to work on a specific season, unlike on a feature film where you can potentially squeeze in another few months.
Derek: Feature films have deadlines as well. If you’re doing a summer blockbuster, people plan those dates a year in advance so that they don’t collide with other films and maximize their profit potential. You are backed into a timeline, and just like everything, deadlines dictate all. What can be achieved with the money and the time says where you need to go.
“Game of Thrones” does a great job of planning out their work. They will pre-vis quite a bit of it to understand what they want to do, what the constraints are, what they need to build. Then they assemble all those bits and make sure they have the budget for it. They are a good example of using pre-vis of planning the season out and living in the constraints that we set out for ourselves.
Kirill: With the level of quality that rivals that of feature films, does it feel that you’re working on a television show on that production?
Derek: It definitely does more of a feature film world. It has a longer timeline, high production value and higher budgets. A lot of people in the VFX episodic TV community bemoan the amount of time and money that “Game of Thrones” has, because it is almost uncompetitive for the people that are working on smaller budgets and timelines. That’s the advantage, I guess, of having a hugely popular show. HBO is willing to put that time, money and effort into making it look good.
They are constrained, but they are definitely longer timelines than you’ll see in episodic television. They are trying to do something grand, and they are trying to do that on a cinematic level. I think that the work that we’ve done is definitely of that cinematic type.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: As Rhythm & Hues has deep expertise on doing creature animation in general, and dragons in particular, how much different was it working on the dragons in “Game of Thrones” for you?
Derek: The interesting thing about working on that show is that the dragons there are not standalone creatures. They are tightly tied to a primary character, and each of the seasons that we’ve worked on, Daenerys has been in some way interacting with them, and often riding on top of them. That makes for much more challenging problems. We’re not freeform-animating a dragon. We have to account how her film footage works with that.
We saw the evolution from Season 5 to Season 6 around that. If you go back to Season 5, all the shots of her riding a dragon are somewhat static. You don’t see a lot of camera movement. The production, in fact, wanted to use practical flames on set for all the fire-breathing parts. So we pre-animated the dragons, and that data was applied to motion control rig that held the flame-thrower. That was played back on the set, with the practical flame interacting with people, and we’ve built the animation around that to fit back with what everybody wanted to do.
We did the same for flying shots – pre-animating the dragons flying, and composing them in the way the production wanted to see as far as what the camera did in relation to Daenerys. Then they would take her and place her on a motion control setup that we call a “buck”. It’s kind of a green-screen saddle that she rides on top of Drogon. It’s powered by a hydraulic system that moves it around. It rotates, rolls and pitches to simulate what we’re doing with the dragon animation in the camera, so that the motion control rig that we’ve pre-animated moves in the same way as the camera.
Now we have these dynamic shots with a lot of movement in them. She’s responding to the movements of the dragon. As it’s flapping, there’s the up and down motion, and you can see how her body, hair and clothes respond to that. We went to static to much more dynamic shots. That pre-animation was a form of pre-vis that we took pretty deep in terms of the quality level, and that was used to playback with motion control on set. We applied the animation again, cleaned it up to make it look nicer, and that gave us the final shot with the rider.
That’s very different from what we’ve done in the past where, maybe, it’s just a dragon flying by itself.
Kirill: Do you have some kind of a guideline about the visual continuity of the dragons between seasons, or is each season its own thing?
Derek: The seasons are relatively standalone, but it’s still the same story and the same dragons. But between Season 5 and Season 6 they grew significantly, so we couldn’t just take the same animations that we had. The dragons are larger, and thematically they need to feel the same characters. There’s so much going on around them, and the main concern is to make them match within that context rather than staying true to the previous one.
Kirill: Is the dragon “screen time”, so to speak, limited by how much time you have to work on a single season, or is it more driven by the story itself?
Derek: I can’t speak to the storytelling standpoint, and perhaps those are just the moments they want to get to. There’s also cost implication. Each of these dragon sequences cost some amount of money, and that’s money that doesn’t go to other things. You have production value and sets and wardrobe and cast and crew. You take a look at the story and where you want to spend your money, and that’s how you get at where we are now.
Would they want more dragon sequences? I have no idea. I don’t know what their storytelling needs are. Would the audience want more dragon sequences? Yeah, probably. Everybody seems to like them. But like everything, we have to spend the money in best service of the story. They pick what they think is effective. In Season 5 and Season 6 the second-to-last episodes are usually their big action ones. They would do their big sequences in those. Everything else is built in service to that. That’s how it has been in the past.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: Like you said, the dragons are the most fantastical thing in those sequences. And then you have things like digital set extensions, creating complete digital sets, and doing extensive digital crowd replication. What about the software that controls these more “invisible” effects, and how has it evolved so far?
Derek: It’s all the same type of software. It’s several different types of packages. Maya modeling software has been around for many years, and its path has been evolutionary rather then revolutionary. There have been advances in how we paint and apply textures. Some years ago a lot of things were done with NURBS and implicit surfaces because it was easy to get texture space out of them. Modeling and texturing tools have gotten much better over the past ten or so years. Polygons are easier to deal with topologically, and the tools for that are much more robust.
The rigging tools have evolved as well. The way you set up bones, kinematics and muscle systems is much better than it was 20 years ago. They are providing much more realistic approaches to animation. The crowd tools have improved as well. Massive has been around since the early 2000’s, and now there are multiple tools out there, solving similar problems in different ways.
We went from setting up a bunch of lights in a non-ray traced environment with shadow maps and reflection maps, to ray traced rendering. Now we get much more realistic rendering than we used to. Compositing has evolved, and we’re much better at dealing with high dynamic range. We have much better tools for healing, tracking motion, and integrating everything. The tools used to be slow, and now the interaction is very quick.
One of the biggest advantages of digital projection is that we don’t have to wait a day to see what we did. Before you would print a 2K image, and wait until it was on film to look at it. Now we have 2K or 4K projection, and we can instantly look at it, at the right resolution, frame by frame. There’s a much finer tooth comb that goes over effects to make things work under a variety of different conditions.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: It’s amazing to see how much the field has advanced, especially compared to the level of animatronic effects that were seen in productions from a few decades ago. Do you worry how your work will be seen by the next couple of generations, in 40-50 years?
Derek: We always have a critical eye on the things we’re doing. Technology and artistry improves through the ages. If you go back and look at traditional films – not even the visual effects, but just how production values changed – you see that we have better cameras, better lighting, better sound. Everything improves, and that’s also true with visual effects.
If I’m still around in 40 or 50 years, I’ll look back at what I did and I’ll say “Wow, I thought that was good?” It would be graded on a much harsher curve. When the first film came out, people fled the cinema when they saw that moving train and they thought it was real. It was very crude and jittery black-and-white image, and since people had never seen that before, they believed it. There was a suspension of disbelief.
Our suspension of disbelief is very high now, and I think that it continues to climb higher. What we consider acceptable now, audiences will not consider acceptable in the future. The audiences will get more sophisticated in understanding what is real, and they will get much more critical. I believe that things in the future will become better.
Tools will get better. Process will get better. Automation will get better. I think the primary drive in the VFX for the next 5-10 years will be on the process of trying to do more for less. How do we take a lot of human effort out of a lot of things so that we can have machines fill more of the gaps at a better price point? How AI is going to play into that? How are characters going to drive the performance and interactive?
I believe that machine learning and AI will play an important role in VFX in the future, but interactive will probably drive that first. How do we build intelligent software that understands edging maps so that it looks at a frame and picks up any person from the background? Machines don’t do that very well right now, and they need help. How do we get that type of machine learning? How do we derive character-driven performances without any input from the animator on the details, with the AI can fill those gaps? I think those are the things that will happen in the near future.
Progressive layers in VFX for “Game of Thrones” by Rhythm & Hues.
Kirill: Is there any difference in the work you’re doing for “Game of Thrones” and your feature film work as far as the “target” screens for the audiences? These big summer blockbusters look so impressive on the big cinema screens.
Derek: I think they compare very favorably. The advent of HD has brought very similar resolution to the world of episodic television. That was the major difference where you worked at 480p in TV vs. 2K in film. Now you’re up to the same resolution, and they shoot on the same cameras. Rec. 2020 brings high dynamic range [HDR] and 4K televisions, and things that are broadcast and streamed are doing possibly higher resolution than feature films are.
The screens might not be bigger, but your relative distance to size ratio is very similar now. If you have a 60″ television in your home, and you sit close to it, it’s the same relative image area as when you go to the cinema and see it on a 40′ screen. What we have to do as far as the amount of effort and quality is extremely similar, if not the same, and the expectations are every bit as high.
Kirill: How was your field affected by 3D productions?
Derek: Stereo definitely brought an extra level of complexity. You have to do twice as many images. You have to worry about convergence. It takes things that used to be simple and makes them harder. Painting out a wire, for instance, might be simple on a single image, but in stereo you have to worry about stereo artifacts if you don’t paint it exactly the same in both eyes and you have a stereo shift. There’s the complexity of twice as much storage and twice as much work. There’s the interocular dependency that requires extra amount of effort.
There was resurgence in that, probably for studios to find a way to make films profitable, but it doesn’t seem to be as much push in that direction. Shooting things in stereo tends to be very expensive production process, including the cameras and the crew that go with that. But I haven’t really been tracking that trend. I know that a lot of films are converted to stereo afterwards, and that is more cost-effective. And it definitely requires more work in post-production and VFX.
Kirill: How do you describe what you do when people ask you what you do for a living?
Derek: I usually just say that I do visual effects. People think that it sounds really interesting, but I think that if they did it day-to-day for work, it wouldn’t be very interesting for very long [laughs]. I move the conversation somewhere else. Every job sounds very interesting to someone who doesn’t do that, but typically work is work. It’s not something I talk about in social setting.
Kirill: As you sit to watch a new movie or a new TV show, do you get to enjoy the stories and the worlds in them, or do you catch yourself thinking what was made in the physical world and what was made in the digital one?
Derek: Working in VFX is somewhat corrupting. When you see a magnificent spectacle that someone else has created, I’m instantly drawn to analyzing how they put it together. What were the techniques? How did they shoot it? How did they choose to do things? It’s not necessarily looking for the flaws, but rather admiring the underlying work and seeing how it’s done. It’s like a mechanic looking under the hood to see how the system is put together.
I wouldn’t say that it ruins the experience. It changes the experience. I no longer view it through a fresh set of eyes, but when people do it particularly well, I stop thinking about that. Afterwards I’d reflect on how effective that is, to disengage me from the process and have me enjoy the story.
Kirill: What about “Game of Thrones” when you do know the particular scenes that you’ve worked on? You might not have read the entire script, but you do know things. Do you get to enjoy that show when it’s out on HBO?
Derek: I definitely enjoy the episodes that I didn’t work on, because I don’t know what to expect. And on the episodes that I did work on, I find that I’m waiting for the moment when our work comes up and I want to see how it plays into the bigger story. It does change the anticipation of what you watch.
And here I’d like to thank Derek Spears for this great opportunity to hear about the ever-evolving art and craft of VFX. I’d also like to thank Charles Martin and Andrew Cohen of Impact24 PR for making this interview happen. Finally, if you want to know more about how films and TV shows are made, click here for additional in-depth interviews in this series.
