Our vision is to build a robot that is able to paint with real color on a real canvas. Our machine called e-David (Drawing Apparatus for Vivid Interactive Display) is an ordinary welding robot, which typically is used for manufacturing cars. We combine the machine with sensors, a camera and control computer. A computer program produced drawing commands that are executed by the machine. Since we work with real materials and brushes, the output of the machine is never perfect. Therefore we establish a control loop using the camera in which the output of the robot is captured and new drawing commands are computed that correct the errors of the preceding step. This way we gradually approach a given goal function (a given image).
Our hypothesis is that painting – at least the technical part of painting – can be seen as optimization processes in which color is manually distributed on a canvas until the painter is able to recognize the content – regardless if it is a representational painting or not. Optimization happens intuitively during the drawing process and is highly dependent on the medium and its restrictions. Centuries of so-called academic art have been shown to what extent algorithms for spatial division as well as color and content composition can be used for creating art works.
On the other side computer graphics, and especially in so-called non-photorealistic rendering made a lot of progress in imitating painting styles using the computer. By simulating media and stroke composition a lot of painting-like images were produced, typically by computing pixel information that later was printed on a conventional printer. E-David will substitute this by distributing real color on real canvas and thus will enable us to enclose the whole process of drawing production into an optimization framework. This will allow us to investigate human “optimization schemes” and to find out to what extent such schemes can be formulated using algorithms. We will use iterated optimization schemes to produce color distributions on the canvas which are constrained by given styles. Semantic information can be integrated to optimize representational paintings; a tree will be painted in a different way than a face even if the colors are similar. Such semantic information can be obtained from semantic image analysis of the given target image. A number of other questions arise immediately:
It is of general interest to find out to what extent a machine is able to learn the whole process of creating a painting. The results might even influence our perception of what art is - besides the imitation of existing drawing styles (imagine what Seurat would have been thinking about a robot that helped him with the millions of dots) the machine might enable new techniques since labor plays no role any more. This way very complex visual art works can be created. The software and hardware platform will be open to artists, which can realize art works on a Java-based/processing environment.
Analysis of drawing styles
The setup can be used for the analysis of existing art works and painting styles. The robot enables us to execute brush strokes in a precisely timed sequence. This will allow us to set up experiments for color mixture and ordering of brush strokes. Accomplished by appropriate image analysis techniques this might allow us investigating the temporal development of artworks (and even the detection of forgeries).
Algorithms for visual feedback
Algorithms for learning the usage of drawing tools and their visual control are prototypical for a more general class of optimization methods that can be applied in different areas of CIM. Examples are welding and painting robots, especially in small businesses in which efficient reprogramming and adaptation of robot programs is necessary.
Extension and validation of simulation methods for computer graphics
Computer models for color description, their mixture and interaction with the canvas are still very limited. Existing approaches allow this only to a limited extent, especially if complex materials are used. There are no methods to compare computer simulations and real artifacts, this also hinders a thorough evaluation. Within the e-David project we will continuously supervise the paint application. However, this supervision is not trivial since real paint has a number of 3D effects, specular reflection and other disturbing properties. A special setup with polarized light from small angles is used to overcome these problems, we will calibrate the images by using spectroscopic methods.
e-David's Google+ profile.