“Can we train a machine to take a 3D model and create 2D perspective sketches of it that look as if they were drawn by a human?”

cGAN model created as part of research investigating the use of synthetic data for training sketch-to-3D ML algorithms. Given as input an image of a 3D structure and its normal map, the model outputs a sketch with realistic errors and deviation that appears to be drawn by a human. The algorithm avoids sketching fine details of the structure, to allow for any structure to be used to create a simplistic sketch.

To generate the training data, I used the parametric modeling program Rhino and its visual scripting language, Grasshopper. I generated random structures with both “complex” and “simple” details, then created two images; one of the “complex” model, and one of the “simple” sketch. The ground truth sketches are generated using a mathematical approach to approximating human sketches (see bottom right), referenced from this paper. With this method, I could generate thousands of training pairs in a few hours, vastly speeding up the training process for the model.

The model was then built using pix2pix, a common network architecture for image-to-image translation. The generated sketches could reliably identify “major” edges, while avoiding fine details that were not critical to the overall shape.