Screen graphics for "Blade II", 2002, courtesy of Mark Coleran

Continuing the ongoing series of interviews on fantasy user interfaces, it’s my honor to welcome Mark Coleran. His seminal work defined computer screens in film in early 2000s, influencing a whole generation of artists that do screen graphics and motion design, and he’s widely credited with coining the term FUI itself. In this interview Mark talks about the early days of doing interface design for film, his transition to the world of real-life software, application of VR and its potential future for storytelling, the difference between art and design, the slow pace of evolution of the software tools at his disposal, and how to not get burned out after a couple of decades in the design industry.

Kirill: Please tell us about yourself, and your path to where you are today.

Mark: I’m Mark Coleran, and I usually call myself a designer. To me being a designer means to cross a broad range of things, but I would say I am mostly in the field of communication design. Communication design is about telling people something. It’s about communicating an idea, a principle, a policy.

Back in the late ’80s I worked as a graphic designer in quick print shops, large print places and agencies, and I always had a penchant for the more technical side of things. I was fascinated by tools and their potential. I’ve been using Photoshop from version one, and started using AfterEffects early on. Video production was always there in the mix of things, and my first motion graphics piece was to do a quasi stop motion animation of a logo, all done on tape-to-tape, loading them one frame at a time onto a cassette.

As I was playing with motion graphics, I almost bankrupted myself buying machines and software things like Electric Image and After Effects which had just changed owners from Aldus to Adobe, originally Cosa. It was quite limited at the time, essentially Photoshop that could do multiple frames, and it was fantastic. I was playing with these bits and pieces, and as the market was evolving, it felt to me that I was perfectly positioned as an individual designer and production artist. It is production art because you are doing things for other people.

It was a good place to be, and it opened up certain opportunities. I worked with a prosthetics company in Pinewood that was doing a digital back end to the prosthetics people to extend that prosthesis out. So somebody might build a smaller part of a dinosaur as a prosthesis, and we would build the rest of the dinosaur digitally. I wasn’t too much into character animation, but it gave me an opportunity to work on a wide range of things like 3D modeling and compositing. While I was there I collaborated with special effects companies that were working on rendered wireframe sequences like a jump stunt off the top of a building. You add a lot of detail into that image to essentially reproduce what the stunt would be like in a cut, and that keeps the insurance people and the director happy and calms nerves as it gives a clearer idea of the “real” shot.

I always loved techy graphics, especially in movies like “Dr Strangelove” and “2001: A Space Odyssey”. The graphics there are geometric and mechanical, and they work to tell a story. So it just happened to be that on “Entrapment” they needed a screen to show how long this rope was, and I asked them if I could work on that screen. There was another company already working on it, and I ended up doing some visual effects and compositing on this little handheld computer. That was a good experience, and it got me in the door to work with Useful Companies, specialized in display work and screen graphics.

That would later be coined FUI, and I’m not sure I can take credit for that name, even though I’m given credit for it. Nobody knew what to call it and I was not quite happy with that name, but sadly it worked. Nobody outside of our little industry knows what that actually means, so maybe it’s a half success.

From there I learned a lot about doing screen graphics and got to work on some really good projects. It also came with the realization that I was a terrible designer. I don’t have any background or training for it specifically, but I do have a background in engineering drawing. I used to work with large, complex documents, and I started bringing that aesthetic and precision into my work on screen graphics. There two things married up quite well and worked for me rather than against me. I got to work with great people like Simon Staines. He’s a phenomenal designer, probably one of the best that you’ve never heard of as he prefers to keep his head way below the parapet.

People give me a lot of credit for this field, and a lot of it goes to him. I’m good because he’s good. He taught me a lot and turned me into a proper designer. He deserves a lot of credit in our field.

Screen graphics for “The Island”, 2005, courtesy of Mark Coleran.

Kirill: And then around 2007 you stepped away from the “fantasy” world into the “real” world projects.

Mark: That is true, but not quite as clear-cut. I love the software people create. You use it and you use it for a long time, and as they say, familiarity breeds contempt. And it’s absolutely not contempt for the people, believe me. People who make this software are amazing and talented, but you get frustrated with the limits of the software itself. You reach out, you talk to people, you find that sometimes those limits can’t be changed and sometimes they can. I got a chance to work with a lot of different people and contribute towards some of that as well. It was gradual, as it goes back and forth. I also had people reaching out to me asking to work on the interface part of their software.

All of this combined led to quite a few years of doing UI work, and that work is not as simple as people think. There’s a lot of factors involved in that. You have the technical capabilities of software and the systems they are running on. You have the user experience and the interaction part of it, which is massive. It took me a long time to learn this new craft and its considerations, going through the same process as I did with screen graphics. I had a couple of full-time jobs, and then joining GridIron who were known for their Nuclear Pro software.

We had discussions about some new ideas, and I joined them to contribute to the creation of a piece of software called Flow. It was designed specifically for artists like myself, and the idea was to try to improve how we do things. It was a great lesson and a great experience. I worked with phenomenally smart and brilliant people. It was a brilliant product and everybody loved it, but nobody bought it [laughs]. It was a lesson in so many aspects. It was an interesting experience to try to tie the film fantasy type of work to real products, because people love that aesthetic.

GridIron’s Flow, courtesy of Mark Coleran.

I recently did a conference talk with RGD in Canada, and it was about taking away the wrong lessons from FUI. I talked about the work I’ve done in films, and how I tried to apply it to the real world, and how I tried to apply the wrong things. We tend to fall in love with the aesthetic, and we think that the good looks are what makes it work. But the work we do in film is to create a diegetic prop. In film, you need to create a feeling about the person using it or the scene that it is in, or to articulate something about a piece of action or other plots.

It’s an illustration and a storytelling device, and there’s not much more to it. I say this not to dismiss it, as it is a really important thing. But it is not an interface. It was not designed to be used.

We spend so much time in film to make things look sophisticated and complex. It is sometimes horrendous how much time we put into the amount of detail we put into interfaces, and that is needed to impart a feeling of what it is supposed to be. And yet, real world software goes the opposite way of that. You try to make it less detailed, simpler, focused on the task that needs to be completed – and those things don’t work together. The film aesthetic fails here, in a sense.

But there are so many ways we can use the stuff from film. In film, you are telling a story, and in a sense, this part is quite under-utilized in real life software. When I work on interfaces for film, I use the same principles of composition as it’s used on the wider scene. I use light, shape, form, movement and sound to draw people’s attention to the right place. Each film screen has to have what we call a read. I see this thing, I have to understand it, and it has to tell the viewer exactly what is happening. And it has to happen instantly.

But I didn’t initially take these lessons back to the real world. It took time. It is not about the aesthetics. It is about how these cinematographic principles could make software better. The cinematic interface is not necessarily about how it looks like, but rather how it feels like. Adding animation to interfaces was one innovation that started in film. I was teaching it to myself, based on the same principles. Where am I? How did I get here? What can I do? Where can I go? Animation can assist with answering these questions. It was a great lesson on what to do and what not to do.

Screen graphics for “Deja Vu”, 2006, courtesy of Mark Coleran.

Kirill: Circling back to this concept of familiarity in film and in our real devices, do you see it as both enemy and friend? You create something new and different, and people get frustrated because they don’t understand it. And if it’s too familiar, it doesn’t perhaps feel as attractive, especially in film.

Mark: There’s a couple of different aspects to that idea, and they’ve also changed dramatically over time. You always start from where you are. The look and feel of those things will always be based on what exists now, and then you’re trying to push it ahead a little bit.

It’s hard to go fully radical, and do something odd and unique, purely because your own position isn’t familiar with that. It’s very hard to do abstract interfaces. You still have in your head that this is what this is supposed to be, and that pulls you back. But then, back when I started doing that work in film, there was a great benefit where you could almost get away with anything, because most people were not familiar with even using a computer. It was not common to have a computer at home, and people’s access to computers at work was quite limited. They were not used to seeing computers, and it worked for the suspension of disbelief. Of course, over the years that has changed, especially when the iPhone came out, which was right around the time I left.

The advances in consumer electronics made this job really hard. People’s familiarity with electronic devices and computing now is so deep. If you go for the real-world kind of thing, you have to get it absolutely spot on, because the slightest thing you get wrong destroys the idea completely. It’s harder to do the realistic stuff now than it is to do the fantasy stuff, and if you do the fantasy stuff, you have to push way ahead. Chris Noessel created a great book called “Make It So” in which he is looking at how these things were judged if they were going to be real systems.

One of the examples he brings is from black-and-white Buck Rogers films where they have video screens where they see each other, but they still pick up a handset like a telephone to talk to them, even though they could see them on the screen. That was the familiarity and the tech at the time. You had to have this hook, and it’s a brilliant observation from Chris. You look for those hooks where people are familiar with something, and then add to that.

My work now is a bit different from the average FUI work, and I describe it as pragmatic futurism. I’m looking for the realistic aspect of it. We’re trying to push it ahead, and add that detail and texture to it, but make it feel like its feet are still on the ground. There’s been a trend over the years to do beautiful, absolutely gorgeous work, but it is very different from everything else we would ever use. In some sense, that separation from reality gives you a lot of scope because you don’t have to make it feel as if this is a working thing. But it also makes it a lot harder to do things that feel like progressive interfaces. It makes it harder to see how good this could be, or where it could go wrong.

This is what I’m specializing in at the moment. I started in the FUI world, then I was doing a decade or so of UI / UX work trying to push things forward, and now this union of the two worlds is making it a bit hard for me to do FUI work. I think about it too much [laughs]. I had some feedback to stop thinking as much, but that’s what I do now. I’m looking for a genuine hook where it feels like it could be something real. It may be in the background, but it still has to feel like this was a real machine. It’s a bit better than we have, it works a little bit better, but it looks like it could have been a real object.

Sometimes you have to roll back the aesthetics. You can’t go too abstract with it, because you still need that familiarity. You’re pushing things, but not too far.

Screen graphics for “Deja Vu”, 2006, courtesy of Mark Coleran.

Kirill: Do you worry about how your work ages in film, as well as in real-life software? “The Net” was done in the early online days, and its portrayal of ordering pizza and plane tickets was excellent, even if the younger audiences today might not understand the impact of it back in mid-’90s. Do you look at the moment you are in, or do you try to somehow glance into the future?

Mark: To be honest, I never try to glance into the future. It’s interesting that you bring up “The Net”, as it was quite good in its predictions.

The things that don’t age in films tend to be around the display technology. It’s not what is on the screen, or even the idea and the plot of what is going on. It’s the technology itself that dates the film. We were using CRTs back in the day, and now nobody sees them. So the second you see one in a film, that is instantly dated. And sometimes you want that. I worked on “Spy Game” in 2001, and we did a whole lot of background stuff that was supposed to be in the late ’80s in a CIA control room. Anybody watching a movie who says “Oh, so what’s weird about ordering a pizza” is not actually getting the fact that this movie was made 25 years ago.

It’s the hardware around it that gives things away. One of the big things I’ve had to change is to move from 1280×720 screens to these massive 2560 and even 3840 ones. There’s so much more detail that is needed now.

Coming back to an earlier question about making things familiar to people, I worked on this little gesture system where I took phone gestures that people are familiar with, and broke them free from the glass. So you “zoom in” a thing using your line of sight, right over that thing instead of on screen. It adds a different dimension while still keeping things familiar. It’s not “Minority Report” where you’re waving your arms around with a radically different interface.

Maybe the conversation about things becoming dated in our field exists because we have seen a very specific transition in the scale, the resolution and the mechanics of display technology. It’s just a moment in time that we happened to have seen.

Screen graphics for “Bourne Ultimatum”, 2007, courtesy of Mark Coleran.

Kirill: Are you ever happy with the tools at your disposal today or in the past?

Mark: I am extraordinarily unhappy with the tools that we have available. This is not a personal criticism of anybody particular, but the fact that I came back to this job last summer and within two days I could do it as well as when I finished it 11 years earlier, means that the tools haven’t changed.

Kirill: Is that a bad thing?

Mark: Yes, because the tools at the time were limited. It’s a very old conceit in software that we build the tool to do a specific job, but it’s not just that. That tool is built to do that specific job for the largest and widest amount of users in the most basic way possible, and that’s the way corporate software works. It’s easier to add a little thing and sell it again, rather than do something radical that might change these things.

I’m not asking for specific things, or to have them cater to an individual as to how they work. But there’s been massive changes in how we all work. We use lots of different software to do our things, yet interchange is still a real problem. A lot of the work we do is still destructive and I hate that. I’m always looking for ways to build more procedural workflows where I’m not baking anything in, because I don’t know if I have to change something. The way that we work is destructive in the sense that it’s hard to keep and reuse assets for different projects. We’re constantly having to do the same job again.

We work with lots of different people, and yet this collaborative work is built on top of destructive tools. Two people can now work on a project at different times or occasionally together, but the work you’re doing is still the same. The tools are still the same, and it’s the real issue. There’s amazing people out there producing individual tools that are fantastic. And people who work on bigger tools like AfterEffects are phenomenal. But the nature of where these tools are positioned in the market means it is not as good as it could be. It fails on the edges of how people work, not necessarily the work they’re doing itself with the tool.

I’m trying to find new ways to do this stuff. One of the reasons I came back to this is because I’m interested in how can we do this better, and from multiple perspectives. And one of those perspectives is the tools. How can we get the basic stuff out the door? How can I have adaptive assets that I can change and build upon, which then gives me the time to focus on that finishing detail for that particular project? It’s a real huge issue, and I don’t think any single tool is solving this. They all live in their individual kingdom, despite some basic interchange capabilities.

They’re all still working as if they’re all completely separate workshops, whereas you need you need a workbench which you bring tools to. There are possible solutions and some discussions, but it’s a hard thing to get around. I think it needs a radically different perspective on how we approach the idea of a tool and what it is. We need a canvas, and we need to bring an object to work on that canvas, and when we take that object away, it doesn’t destroy the work and we can bring whatever we want to it to work on it. The way we’re working hasn’t changed much in the last 30-40 years. Some bits and pieces are radically better, but the workflow is still the same old thing that is not quite working anymore.

Screen graphics for “Mr. & Mrs Smith”, 2005, courtesy of Mark Coleran.

There’s a lot of pressure coming on artists from all different sides. Time gets shorter, budgets get tighter, the ability to leverage our assets and our work becomes much more great. I was hitting this when I left in 2007. I build this thing that I called FUI OS as a way of doing these things a lot quicker. I had a generic grid system which I could chop up to fill various monitors, a way of blocking things out which then I could apply styles to and explore, so within a day or two of a new project coming in I could have a basic look and feel to an entire thing. I had pre-rendered components for AfterEffects and expressions where I could control the components and drive it from something else.

And now I am exploring Figma for doing FUI. Some people complain about Figma not knowing what kind of software it wants to be. It was built as a UX prototyper, but it’s been built in such a strange way that it’s really open to being anything. It’s almost like a platform. Adam Plouff has built some amazing tools over at AEUX for Figma, Sketch and AfterEffects. His ideas about being able to move stuff around non-destructively gets at the heart of what I was talking about, and it’s phenomenal. With these ideas, 90% of his work doesn’t need heavier tools like Illustrator.

It’s a problem for all artists. How do we leverage everything that we’ve done and turn things into components, rather than you know copying and pasting expressions? It’s a big issue, and I’m not happy with the tools right now.

Kirill: Do you find it a little bit easier to design for fixed screens in movies as those screens are fixed, or do you still need a system that scales between all the screen sizes in there?

Mark: It is hard to do because nothing exists in isolation. Even if it’s just one screen on a set, there will be other things in that production where you may need to utilize those same assets, so you still have to think about it as a system. You also need the development time to create that idea so everything feels coherent.

Right now I’m trying to come up with the idea of how do we do that. How do we create a consistent grid system with pre-done, coherent layouts, and then apply aesthetic style choices on top of that for the specific production?

Screen graphics for “Mr. & Mrs Smith”, 2005, courtesy of Mark Coleran.

Kirill: Going back to screen graphics as a storytelling device, is it painful sometimes to see weeks or even months of your work condensed to a few seconds in the final product? Is this just the reality of the business?

Mark: It’s the reality of it. And it’s also a general problem with design itself. We care about what we do, and it’s easy for us to become obsessively caring about something. We think we own it.

People will hate me for this, but for me it boils down to this. If you’re doing art, you’re doing it for yourself. If you’re doing design, you’re doing it for somebody else. You don’t get to make those decisions. We really do care about this, but you do have to have a very high level of pragmatism, almost down to a cynical level of detaching yourself from it. Somebody’s paying me to do this job, so let’s get the job done. But it’s not mine.

You don’t have to be aloof. I care about what I do. I want it to be good, and I want it to work well and to do what it was meant to do. But I focus more on my craft now than on the individual objects themselves. Did I do this well? Have I learned something new by doing this? Did I save some time by trying a different approach? It’s the craft that I now get the pleasure out of, and not necessarily the end result. It becomes a hard lesson. When I left the work, it was because I did actually care about that. I was getting so annoyed by what I was doing and how it was looking like on screen. We do this work, and then it’s there for a couple of seconds on screen. It doesn’t feel like it’s worth it.

That’s why I went into the world of real software. In that world, every decision you make affects how that thing works. It affects the experience somebody has. It’s continuous as well, as you refine it, iterate on it, and build upon it. And my experience in real software is the same. Nobody cares. Nobody sees what you’ve done. They all complain about the stuff you did badly, but in the end it doesn’t really matter. It’s out there, and it’s being used, and you don’t own it. A hundred other people had their fingers on there, and you did your part, and it’s exactly the same as movies.

I’ve done good work in bad films, and yet I didn’t know that when I was working on that film. There’s so many different aspects and so many different people touch a film, and any one of them could ruin it or make it great. You never know, but it’s not you alone making something. You’re part of a big creative machine. Sometimes it’s amazing, and sometimes it’s horrible. You focus on the craft of what you’re doing, on doing it right and doing it well, and not necessarily the end result. You have no control over that. You own what you can do, and then ignore the rest. It sounds aloof, but you have your own sanity to protect. Otherwise you’ll burn out. You’ll get bitter and upset because other people are going to make decisions about your work. Some of them may be bad decisions, but you have no control over that. You can’t care too much.

Screen graphics for “Domino”, 2005, courtesy of Mark Coleran.

Kirill: We’ve seen quite a few films, some set in more distant future, but some also set in the today’s timeline, that explore how graphics and interfaces can look like on screens that are not these “hard” rectangular surfaces. Sometimes it’s a piece of curved glass, sometimes it doesn’t have fixed boundaries, sometimes it goes into the third dimension, sometimes there’s no screen at all and it floats as a hologram in the middle of the room. Do you see this as storytellers in search of a “wow” factor, something to impress the audiences?

Mark: There’s a couple of different aspects to that, and some of them are just practical. For one, it’s what we were talking about earlier around dating stuff. It lets you have as neutral a surface as possible. You get rid of the hardware and it’s just a piece of glass, or even nothing around that thing in the space. You end up with no indication on what has generated that interface. You remove the specific technological reference from it, and it becomes almost a neutral piece of technology that doesn’t exist.

Another part of it, and I don’t have hard research behind it, apart from questions I’ve been asking people, is that a lot of it comes down to directors and cinematographers not liking displays and machines in general. It’s the same reason they look so bad in an office. You look into any office, and all you see is a sea of monitors. We don’t even stop to ask how would that office look and feel like if you didn’t ever see a monitor. The monitor itself, and the display technology that goes with it, are disruptive in a space. From the cinematography standpoint, you get a much better composition if you can see the actors. It’s easier to shoot, and it looks better. They want to see the performance, not the displays.

If that screen is big enough to show you something well, then it’s obstructing something else. You remove that screen, and it opens up the scene. I can see things happening in other places. “Avatar” did a beautiful job in the control room where you see what is going on, what those individual people are doing. Of course, “Avatar” is far ahead into the future, and we’d be talking about why would they actually be using what looks like a Dell monitor had it been there.

Contrary to popular belief, there’s not as much thought that goes into the futurism and interaction aspects of it. A lot of things are general ideas that might be getting pulled off the cuff. It tends to be more practical, in the moment, of how can we make this look good. In some ways, that is a really interesting perspective in that they’re doing what might feel natural to people. They’re not overthinking it, so maybe they’re coming up with some great ideas that people who do design technology should actually pay attention to. But then again, we also need to pay attention to these intuitive calls that might not be good themselves.

Those aerial gestures look good for a few seconds, and I don’t need to touch that screen, but I’m also going to be tired as hell after half an hour doing that. What looks good for a movie doesn’t necessarily feel good in real life. How do I deal with privacy when I have a translucent screen? How do I not get distracted with all the background noise? How do I keep this work surface clean? There’s things we need to consider with that if we take lessons from it.

Screen graphics for “The Island”, 2005, courtesy of Mark Coleran.

Kirill: “Minority Report” was this huge thing that inspired so many explorations that aimed to bring those interactions into the real world, and yet it didn’t seem to bear anything tangible.

Mark: There’s are perhaps couple of reasons for that. Visually, as a performance, it looks fantastic. There’s this concept of something being cinegenic, something that looks good on camera – and certainly “Minority Report” still looks amazing.

But what people also don’t consider is that “Minority Report”-type interface and interaction might only be fantastic for a certain thing in a certain context. People tend to think in absolutes about the way they do something. You can come up with a really good idea, but it’s only good for a specific thing in a specific place. It’s not so much about the waving of the arms, but more about the content on screen. It’s abstract, it’s mixed, it’s blended. That kind of surface and that kind of content might help when you’re moving things out of the way and sifting through something with your hands.

That idea works fantastically for that kind of stuff, but maybe you need something more specific for something else. Maybe I need to touch that thing and move it over here because now I want more control over it. It’s a real mistake a lot of people make trying to draw lessons from these films. You need to ask yourself what is this thing good for, and when it could be good. For the film stuff, you don’t have time to articulate or get that idea across. You have to make this obvious and quick, so you’re only going to use one thing and make it really obvious. If we’re going to draw real lessons from it, we need to take it from several sources.

The easy idea to have in your head is that something is the right or the wrong answer to a problem. Black and white. But the fact is everything is gray. It could be anything in between and all of them. What is this good for and when? And what is that good for and when? And we mix those things together, and we get something interesting.

Android shell concept explorations for Samsung, 2013, courtesy of Mark Coleran.

Kirill: There’s been so much money and effort invested in the world of virtual reality (VR) in the last few years, and yet nothing transformational came out of it so far for our “boring”, everyday office lives. Or maybe that is not the application of it.

Mark: Ironically, you mentioned the application of it. I’ve touched on this stuff on and off for about 35 years now. I was around 18 when I first experienced it, and it was cool for its time. It’s fantastic when done right, but then the people again are not understanding what it is and what is good about it, and why it is good and also when it is good.

The technology has caught up in such a way we can actually have something really special now. It’s caught up with the ideas that people had 30 years ago of what we could achieve, but it’s still a technology in search of a problem. It’s also changed radically in how technology companies approach problems.

There’s some interesting stuff that we could do with VR and AR, and we need to draw a sharp distinction between those two things. They may use similar technologies, but they’re for radically different things. VR is great for close personal experiences, and it works intimately in that digital personal space. And AR is in your environment, and that’s a different type of experience. VR has a phenomenal potential in games, but it’s going to be a different kind of game. It’s not a format shift, which is how a lot of people treat it. It opens up new opportunities, but they have to be approached as new things.

But given the nature of how people approach technology now, and especially companies, it has to be massive. It can no longer be a really good idea for a specialized vertical market or a niche. They can’t sell 100,000 or 500,000 of these things. They have to sell 50 million or 100 million of them, and it’s absurd. I think this is what is killing this field, this idea that it has to be massive. But instead, it should go to a hundred different companies that will find interesting and cool little things as a way to apply this. And again, it’s not about finding just one way to apply it. It could be lots of different aspects, but this is not how it is being allowed to be developed nowadays. Nowadays it is about having to have this one massive thing, and that it’s going to be done by these people who we no longer trust and no longer care about.

Kirill: Do you think this can be easily incorporated in the visual storytelling in film, maybe for VR and AR, or maybe for neural interfaces? How do you do it in a way that makes what is going on clear to the viewer?

Mark: I don’t think it is, at least not the way we think about it right now. The way you just said comes down to implying a format shift, and this is not a format shift. This is a radically different way of doing something, and when I say “doing something”, I don’t mean “presenting something”.

When we talk about presenting something, there’s a degree of passivity involved. If you look at how stories are told today, you see the effort and the work that goes into the composition, the timing, the editing. If you interfere with that with a virtual device, you may ruin that experience completely. You do not get to build tension the same way. You don’t get the build pace the same way. But then, maybe there’s a radically different way of telling stories that these devices will open. Certainly it’s not about telling the same stories in the same way on a different device, and that’s the danger a lot of people are getting into.

Explore this thing, come up with something new, but don’t do the same thing on a new device. That’s where the opportunities are. That could be radical.

Explorations of a unified single interface for all consumer electronics for LeEco, 2016, courtesy of Mark Coleran.

Kirill: It’s an interesting thing to look at this aspect of humankind and our need for storytelling. There’s the narrator who is active, and there are listeners who are passive. Maybe that’s our nature, to partition ourselves into the one telling the story and the ones listening.

Mark: But later on, the person listening tends to become a storyteller themselves. And they might get the story wrong, by accident or deliberately. Everybody adds their own thing to the story, and maybe that’s where these opportunities are. You’re going to see a story, and maybe you get to start to define part of that story by your own experiences, or you want to contribute to it. I have no idea how that would work [laughs], but that’s a very different experience from going to a Cameron or a Villeneuve movie which has been thought through from beginning to end.

In social context, some stories do invite participation. Back in 2010, I had a small talk at a motion conference on burnout for artists. The usual format for these talks is to say you’re going to talk about this thing, this is how this happens, this is what you should do, etc – but nobody likes being lectured to or talked at. What people want is a story. So I told a story about an artist, their great success, and the subsequent burnout, and them scrapping everything. And if you tell a story well, you invite association and identification with the protagonist in the story from the person hearing the story as they start to imagine themselves in that role. That talk went two hours longer than it was supposed to, and extra people kept coming in and participate in it.

It was a huge thing because nobody ever talked about that kind of thing, but it worked because of the story. And it wasn’t just one person’s story, but rather how other people identify with that story. It works for different types of human stories where we associate and could be those people. Maybe that sort of a participatory experience can work for VR.

Kirill: Back at the very beginning, you defined yourself as a designer. Do you ever think about how your life could have been if you were born 500 years ago and what you would be doing?

Mark: I would have probably been burned at the stake by now. I’ve been fired more times I can mention now because I don’t keep my mouth shut. Sometimes it’s difficult for me to read people and to understand what’s going on around me in the human context. I’m not always attuned to that, and yet sometimes hypersensitive to that. If I see something and I think it’s wrong, I will say so. And that ends up upsetting people.

For some reason, when people do hypnosis to reveal their past lives, say 500 years ago, they’re always somebody famous. Nobody is ever a pig farmer [laughs].

Kirill: Well, anywhere between 80 and 90% of Europe’s population worked the land in the Middle Ages, so chances are most of us would be pig farmers.

Mark: Right, but nobody sees themselves as a pig farmer. They’re always a member of royalty. Very likely I would have been a nobody. I would love the idea that I was an artisan or a journeyman of some kind, contributing to building something. It would have been a nice idea, but my own rationality is almost preventing me from exploring that idea very well.

Pattern development work for a round mobile device, 2014, courtesy of Mark Coleran.

Kirill: Do you get bored working with the keyboard and the mouse? Do you sometimes wish for a different way of interacting with information?

Mark: Absolutely I do. I have the keyboard, the tablet, the Mac on the side, but the problem is that there’s no one mode to how I work. There’s the ideation where I’m getting notes, grabbing things from here and there, and creating a shape of something. And that is quite different from starting to refine that, building the thing using specific and detailed movements. I have the mix of touching the screen to move something, and doing more precise mouse movements, and maybe using voice dictation to make a quick note. But the software hasn’t caught up with a lot of these possible ways of interacting. Sometimes I wish I could wave my hand for my Mac to go to the next space.

There’s possibilities with it, but nobody ever seems to think about the idea that you’re going to mix interactions. It’s always a specific – you’re using a mouse, you’re using a keyboard, you’re using a tablet. People are still thinking in single modes and it’s frustrating. I wish it was different, but I lack the technical ability to change that.

Kirill: Do you think it’s connected to that mass appeal, that what this industry does has to be for hundreds of millions, and can no longer be for tens of thousands?

Mark: We have been relying on the traditional vendors for the major software for so long. But then you so see some unusual software like Figma which is open to anybody to interfere with, and change and challenge, and add to, things like Cavalry and even things like Houdini. Real innovation comes from individuals who contribute to things. A lot of amazing tools in AfterEffects are third-party that people brought to the market.

The big players have a corporate imperative to how they do things. But smaller people can always bring something new to the game. And sometimes you need that middle ground where they need more assets and more access, and it can be difficult. Flow was a great environment to see where you were, and see the nature and structure of a project, but it was a separate place from the system in the Finder. And people couldn’t mix the two. You had to go here for some things, and there for others, and that was one of the barriers to the adoption. We didn’t have access to make it a different type of system file view, and nobody does.

It’s an issue with big companies. There’s a lot of gatekeepers keeping out people who contribute something interesting and good. I think it’s the small people who are going to come up with the big stuff. It’s these little things that are going to bring something interesting to how we do stuff. As to whether people have the room to thrive and prosper from, that’s a separate issue.

I can guarantee that for every thing we want or need, some brilliant developer or some fantastic artist has made that or came up with that idea. The hard part is realizing that idea and getting it out in front of people. I hope we have a way of doing that somehow sometime soon.

Concept explorations of a 3D interface for an Android shell for Sony mobile devices, 2010, courtesy of Mark Coleran.

Kirill: Do you find that technology is intruding into your life too much?

Mark: It does, but that’s part of a problem of my own nature that I allow it to do that. I don’t do a job once, I do it twice almost every time. I get the job done, and then I start pulling it apart, thinking how could I have done this job differently and what other things can I get from doing this thing. So I let it interfere with me continuously. It’s an issue for me, but that’s part of my nature. I am obsessive about what I do.

But generally, when I’m functioning normally, it’s not an issue at all. I will work my eight hours and I will walk away from the machine to read a book. It depends on where you are, and what you get excited about, and sometimes that exploration is what you’re getting excited about. Some people talk in absolutes, like you should do this, or you should work that way, or you need to do a job that you love. I don’t agree with absolutes. You find what works for you. Sometimes I work crazy amount of hours to get something done, and I get a lot of satisfaction from it. And then other times I will not touch it. It’s two different sides of the same coin. Find what works and when it works, and if it feels obtrusive then turn it off.

These days, that feeling doesn’t come from me doing my job and the way I do my job. It comes from the way we are forced to interact with friends on the services we have to use.

Kirill: On a related topic, what do you do to recharge?

Mark: Unfortunately, the pandemic in the last two years has broken things, so leaving the house is a challenge in itself these days. In contradiction to my answer to the last question, I also got sick of sitting in front of the computer. I didn’t like the work that I was doing and I couldn’t bring myself to do creative work on my own things anymore.

I have lots of ideas, like 50 a day. You make a note of them, and then you either don’t know what to do with that idea, or it’s too big a thing to even think about approaching. What I did was find a way of doing something with those ideas, which was to create a little canvas and it’s where I got into sticker art. At the same time I started to see a lot of sticker art around me, some beautiful work that people make but nobody will ever part with. They exchange with other artists but they don’t sell it. They trade it.

So I thought that I needed to make some stickers that I can trade with. I created a set of rules for doing it. Right now, when I have an idea, I will spend then next ten minutes to draw it, get it done, print it out on a piece of sticky sheet, and that’s it. Later on I might go back, edit it and refine it, but the first time I touch it, I have ten minutes. Whatever happens after that, that’s done. This got me back into being able to do creative work on a machine again without hating just being on the machine. It got me excited about creating again.

The idea of having these tiny little canvases is to not be overwhelmed. You get away from the perfectionism. You don’t need to fill this big area. And it really worked. Within a couple of months I had a small collection I could start trading with artists. And within a year I’m recognized as a street artist [laughs] with my own collection and style. There’s one gallery, Egoiste in Afflecks, Manchester calls me a Dystopian Sticker Artists, commentary on life, politics and even FUI. It worked out, and I now use it as an outlet to switch gears, to escape to when it gets overwhelming. It also opens things back up for everything else.

Doing sticker art gets you involved in a different type of creativity and a different group of creative people. It makes you radically aware of how many people use art as a way of dealing with their own mentality. Perhaps we never even realized that maybe half of us end up in these specific jobs because of the way we think about things and the way we approach stuff, and yet very few seem to acknowledge that our obsession with the look and feel and the detail of something is telling you something about your nature. There’s a huge group of people you find and talk to, and understand that you’re not alone.

I also get on a bicycle or go for a walk. I moved away from the cities into a small town in the north of England. It’s easy to be around green if you need to be, especially under these current circumstances.

Kirill: Do you ever see yourself retiring, not from the point of view of having enough money to be able to afford to, but rather retiring creatively?

Mark: No. It’s a rather strange process, because unlike a lot of people involved in this, I’m into my 50s now. I can still do this job, and I can still do free stuff. I can’t work the long hours all the time, that’s for sure.

I don’t think I want to retire. What I am doing is looking for different ways to do it. I’d like to build a way of doing this where I get to start defining how we do this, and helping people do that. Maybe it’s working with filmmakers on how to use this technology well, rather than just using it. It’d be nice to build out that idea, which would work better for me, because working straight day-to-day production is getting difficult. I can be up for hours and hours, days and days where I need to be, but I don’t have the stamina for just staring and doing the same thing.

I’m very proud of the insult that Alfonso Cuarón gave me during “Children of Men” when he called me Design Taliban. I knew who he was and how specific he wanted stuff to be, so I took that and ran with it. Turned out there was a limit to how much of my accuracy he wanted to have on that film. And that’s fine, you learn to find where those boundaries are in a movie.

I want to get stuff right, and I’d like filmmakers to think about this as a cohesive and key part of the story. It’s never a central part, and I have no illusions about where this stuff stands. It’s not really important, and we need to understand that. We can get so obsessed and upset about this, and we can lose perspective on that. But it is part of something, and it can be a really good part of something. It’s up to us how well we do that, and we can do a lot better than we are doing. There’s a real danger for a lot of us when filmmakers don’t really know how to deal with this, so sometimes they’ll be referential to other films, and we get a circular action going on.

Certain things become common and generic. Certain looks become common, and it’s because most people don’t know how to deal with it. They don’t know how to describe it, how to articulate what they’re really looking for. That’s the big area I’d like to be able to work more and get more done in for the future.

A sketch to evaluate the visual capabilities of Sketch App, courtesy of Mark Coleran.

Kirill: That was going to be my next question actually. Do you find yourself looking more into the past, or do you find yourself looking more into the future at this point in life?

Mark: Everything we do is defined by what we’ve done, and that is good for the most part. You build on top of things. You can’t come up with fresh things, it’s never worked that way, especially for creatives. Everything that you do is an accumulation of your previous work and it’s an accumulation of everybody else’s work as well.

You don’t avoid it, but you need to understand why you’re doing that and where it came from. You need to understand why did somebody do something in a certain way for a certain thing. You’re always looking at context in the past for example. Somebody else may have solved a problem that you need to solve, so don’t go through it again. Use their solution to this thing, but then bring your own thing to add.

The past is important, but you also think about where could this go and what could this be. I’ve done a mix of product design for real stuff and FUIs, and there’s this idea of the future and what it could be for us, how we would be part of it, and how we would interact with machines. Remember how about 10 years ago companies were producing crazy and sometimes outrageously stupid “future vision” videos. And then they disappeared, and nobody’s doing it anymore. I find that absence quite odd, because they may have been outrageous and they may have been silly, but some of them were absolutely brilliant. Some of them were beautifully made and had some really good ideas in them. They were great little inspirations but they just vanished. People had this idea of what we could do, and now it’s gone.

It would be nice to be able to bring things back like that, and find ways of doing that. We had these ideas of what could be, sometimes just showing people for the sake of it, and it seems to have stopped. It would be nice to be able to look forwards – in a pragmatic sense. I don’t want to create the neural interface. Whatever happens, I’m not getting that bloody chip stuck in my head. I don’t trust companies in any sense whatsoever to have my best interests at heart. I will buy that product, I may use their products, but that’s it. They don’t get anything else from me.

And everybody was so happy in those videos. One friend of mine even directed one of them, and it was a nice piece of work. The strange thing is it seems to have been replaced by a million people making HUDs on Youtube and Behance and Dribbble, and they’re HUDs with no context, and no reason, and no users, and no backgrounds. People suddenly became obsessed with the aesthetics of it, and that’s what seems to replace that idea. They do deserve credit, and it’s beautiful and sometimes absolutely gorgeous work, but it also deserves some criticism because it’s design work without context.

There used to be a trend around 15 years ago of people doing a complete makeover of the Facebook interface or the Craigslist interface. Some were better than others, but they were always judged and criticized based on why they had done it. Some people did really well, and they produced great ideas, especially when it came as a design exercise based on a self-created brief. That’s the key part. It has to be carefully thought through. We can create these visions ourselves, but if you don’t take into consideration the same problems or issues that Facebook designers are facing, instead of starting a conversation it turns into this generic copying and reproduction until everything seems the same. It was an odd trend which I didn’t understand, and maybe I’m unfairly critical of it.

Kirill: If you could go back 35 years and tell your younger self to not worry about X too much, what would that X be?

Mark: I would say that nobody else cares about the things that you care about. You like what you do, you love what you do, it’s important to you, but it may not be important to anybody else.

That difference can be the source of a lot of frustration for a lot of designers and creative people. We’re not necessarily doing what we think we’re doing. We’re always doing something for somebody else. And half the time, we’re just going to be doing the thing that they want and not having any input at all. Understand that you’re not as important as you think you are.

It’s a strange thing for me, because there’s this myth that has grown up around me [laughs]. I’m this mythical FUI beast that did that work, but I’m just a person. I’m just a designer. I was working with somebody recently, and they said that they though I’d be quicker and get this stuff done in a day. But it was two weeks worth of work. So the idea of who you are and what you are can be different to other people as well. Be clear about what you’re doing, who you are, and why you’re doing it.

Understand it’s all just a job. Everybody wants to treat design and creative work like it’s something special, and it is to you. But it is just a job as well. Do not forget that you are just doing something as a job. I’ve worked way too hard and way too long on stuff that nobody cares about it, because I forgot that. It didn’t matter if I done 8 hours a day or 16, but because I did 16, other things suffered. So don’t forget it’s just a job, and you’re not as important as you ever think you are. That’s the one thing I really wish I’d learned a long time ago.

Screen graphics for “Bourne Ultimatum”, 2007, courtesy of Mark Coleran.

And here I’d like to thank Mark Coleran for taking the time to talk with me about the art and craft of screen graphics, and for sharing the supporting materials for the interview. You can also find him on Twitter and Instagram. And if you’re interested to read additional interviews about the wonderful world of screen graphics and user interfaces for film and TV, click here for more. Happy holidays, and I’ll see you in 2022!