It is an often-shared ‘fact’ that modern smartphones have more computational power than NASA did when they landed a man on the moon. This is clear from Moore’s law: that computational density doubles every two years. This relentless, radical change has affected all aspects of society, including the role of filmmakers (we’re already seeing the slow decline in practical effects).
Technological advances ranging from Machine Learning to Computational Photography, are poised to revolutionise filmmaking. We’ve already witnessed this in past revolutions spawned from new technology in the film industry, ranging from 3D cameras to CGI. Currently, the future points to Computational Photography (CP) (creating a single photo by integrating multiple images) and to machine learning techniques called “Generative Adversarial Networks” (GAN). Perhaps we will soon discover a new form of filmmaking that equally balances photography and digital image creation, which some are labelling “Computational Filmmaking”.
How does this differ from ‘ordinary’ filmmaking? Simply put, in ordinary filmmaking there’s always one phase for live-action photographed and another distinct phase for digital effects. In the Computational Filmmaking all of this happens at once, on the fly as it were.
So over time how has technology changed the film industry and where are we…
The iPhone has been able to do CP with its HDR camera function for a while now. With the iPhone 7, Apple also added a second camera that can be used for optical zoom and for a Computational Photography technique called “Portrait Mode”. This simulates, in some way, having a larger sensor and a bigger, more expensive lens, hence allowing the iPhone to do what your DSLR does, but with much less hardware!
In the foreseeable future:
At Adobe MAX 2016, they unveiled ‘SkyReplace’ – in a technology demonstration that showed just how many post-production jobs could be replaced by AI. Sky replacement is nothing particularly new, it’s been done by artists on every single print ad, film, commercial and TV show you’ve ever seen, and for a long time, frame by frame, by hand. SkyReplace is poised to automate this usually manual job through Machine Learning.
In the unforeseeable future:
GAN (Generative Adversarial Networks) is a statistical probability model that can generate realistic looking images – sounds amazing right?
This means you show a neural network millions of images of a tree, and then it can recognize a tree when it sees one, but more interestingly, it can generate original tree images all on its own – let that sink in – original images. This is still in early days, so the images are small, but as Moore’s law has shown, it’s only a matter of time before this becomes powerful new software and widely available. How long until a photorealistic image can be generated just by typing a sentence, or by writing a script?
Adobe’s ‘Voco’ and DeepMind’s ‘WaveNet’ are both ‘neural network’ systems that use text and audio samples (just like GAN uses images) to produce speech indistinguishable from actual human voices. One possible application to filmmakers might be that the software could compose its own, original music!
Dubbed ‘Photoshop for audio’, whilst this is still a demo, it will likely be available to consumers soon. Maybe you’ll be able to use Morgan Freeman or David Attenborough to narrate your own movies.
In the near future, filmmakers might be able to tell their computers what they want their movie to look like, what mood it seeks to portray, and the plot. The computer, using all the newest CP and GAN technology, will generate a watchable result.
The real world that cameras are capturing will begin to be merely a starting point – soon computational filmmaking could really revolutionise the way we think about and approach film and the whole creative process.