This is a script to generate new images of human faces using the technique of generative adversarial networks (GAN), as described in the paper by Ian J. Goodfellow. GANs train two networks at the same time: A Generator (G) that draws/creates new images and a Discriminator (D) that distinguishes between real and fake images. G learns to trick D into thinking that his images are real (i.e. learns to produce good looking images). D learns to prevent getting tricked (i.e. learns what real images look like). Ideally you end up with a G that produces beautiful images that look like real ones. On human faces that works reasonably well, probably because they contain a lot of structure (autoencoders work well on them too).
The code in this repository is a modified version of facebook's eyescream project.
The following images were generated by a model trained with th train.lua --D_L1=0 --D_L2=0 --D_iterations=2.
1024 randomly generated 32x32 face images.
64 generated 32x32 images, rated by D as the best images among 1024 randomly generated ones.
16 generated images (each pair left) and their nearest neighbours from the training set (each pair right). Distance was measured by 2-Norm (torch.dist()). The 16 selected images were the "best" ones among 1024 images according to the rating by D, hence some similarity with the training set is expected.
To generate the dataset:
- Labeled Faces in the Wild (original dataset without funneling)
- Python 2.7 (only tested with that version)
- Scipy
- Numpy
- scikit-image
To run the GAN part:
- Torch with the following libraries (most of them are probably already installed by default):
pl(luarocks install pl)nn(luarocks install nn)paths(luarocks install paths)image(luarocks install image)optim(luarocks install optim)cutorch(luarocks install cutorch)cunn(luarocks install cunn)cudnn(luarocks install cudnn)dpnn(luarocks install dpnn)
- display
- Nvidia GPU with >= 4 GB memory
- cudnn3
Building the dataset:
- Download Labeled Faces in the Wild and extract it somewhere
- In
dataset/runpython generate_dataset.py --path="/foo/bar/lfw", where/foo/bar/lfwis the path to your LFW dataset
To train a new model, follow these steps:
- Start
displaywith~/.display/run.js & - Open
http://localhost:8000to see the training progress - Train a 32x32 color generator with
th train.lua(add--grayscalefor grayscale images) - Sample images with
th sample.lua. Add--neighboursto sample nearest neighbours (takes long). Add e.g.--runs=10to generate 10 times the amount of images.
You might have to work with the command line parameters --D_iterations and --G_iterations to get decent performance.
Sometimes you also might have to change --D_L2 (D's L2 norm) or --G_L2 (G's L2 norm). (Similar parameters are available for L1.)
G's architecture is mostly copied from the blog post by Anders Boesen Lindbo Larsen and Søren Kaae Sønderby. It is basically a full laplacian pyramid in one network. The network starts with a small linear layer, which roughly generates 8x8 images. That is followed by upsampling layers, which increase the image size to 16x16 and then 32x32 pixels.
local model = nn.Sequential()
model:add(nn.Linear(noiseDim, 128*8*8))
model:add(nn.View(128, 8, 8))
model:add(nn.PReLU(nil, nil, true))
model:add(nn.SpatialUpSamplingNearest(2))
model:add(cudnn.SpatialConvolution(128, 256, 5, 5, 1, 1, (5-1)/2, (5-1)/2))
model:add(nn.SpatialBatchNormalization(256))
model:add(nn.PReLU(nil, nil, true))
model:add(nn.SpatialUpSamplingNearest(2))
model:add(cudnn.SpatialConvolution(256, 128, 5, 5, 1, 1, (5-1)/2, (5-1)/2))
model:add(nn.SpatialBatchNormalization(128))
model:add(nn.PReLU(nil, nil, true))
model:add(cudnn.SpatialConvolution(128, dimensions[1], 3, 3, 1, 1, (3-1)/2, (3-1)/2))
model:add(nn.Sigmoid())where noiseDim is 100 and dimensions[1] is 3 (color mode) or 1 (grayscale mode).
D is a standard convolutional neural net.
local conv = nn.Sequential()
conv:add(nn.SpatialConvolution(dimensions[1], 64, 3, 3, 1, 1, (3-1)/2))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.SpatialDropout(0.2))
conv:add(nn.SpatialAveragePooling(2, 2, 2, 2))
conv:add(nn.SpatialConvolution(64, 128, 3, 3, 1, 1, (3-1)/2))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.SpatialDropout(0.2))
conv:add(nn.SpatialAveragePooling(2, 2, 2, 2))
conv:add(nn.SpatialConvolution(128, 256, 3, 3, 1, 1, (3-1)/2))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.SpatialDropout(0.2))
conv:add(nn.SpatialAveragePooling(2, 2, 2, 2))
conv:add(nn.SpatialConvolution(256, 512, 3, 3, 1, 1, (3-1)/2))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.SpatialDropout(0.2))
conv:add(nn.SpatialAveragePooling(2, 2, 2, 2))
conv:add(nn.View(512 * 0.25 * 0.25 * 0.25 * 0.25 * dimensions[2] * dimensions[3]))
conv:add(nn.Linear(512 * 0.25 * 0.25 * 0.25 * 0.25 * dimensions[2] * dimensions[3], 512))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.Dropout())
conv:add(nn.Linear(512, 512))
conv:add(nn.PReLU(nil, nil, true))
conv:add(nn.Dropout())
conv:add(nn.Linear(512, 1))
conv:add(nn.Sigmoid())where dimensions[1] is 3 (color) or 1 (grayscale), and dimensions[2] is the height of 32 (same as dimensions[3]).
Training is done with Adam (by default).
The train.lua script has the following parameters:
--batchSize(default 16): The size of each batch, which will be split in two parts for G and D, making each one of them half-size. So a setting of 4 will create a batch of size of 2 for D (one image fake, one real) and another batch of size 2 for G. Because of that, the minimum size is 4 (and batches must be even sized).--save(default "logs"): Directory to save the weights to.--saveFreq(default 30): Save weights every N epochs.--network(default ""): Name of a weights file in the save directory to load.--noplot: Whether to NOT plot during training.--N_epoch(default 1000): How many examples to use during each epoch (-1 means "use the whole dataset").--G_SGD_lr(default 0.02): Learning rate for G's SGD, if SGD is used as the optimizer. (Note: There is no decay. You should use Adam or Adagrad.)--G_SGD_momentum(default 0): Momentum for G's SGD.--D_SGD_lr(default 0.02): Learning rate for D's SGD, if SGD is used as the optimizer. (Note: There is no decay. You should use Adam or Adagrad.)--D_SGD_momentum(default 0): Momentum for D's SGD.--G_adam_lr(default -1): Adam learning rate for G (-1 is automatic).--D_adam_lr(default -1): Adam learning rate for D (-1 is automatic).--G_L1(default 0): L1 penalty on the weights of G.--G_L2(default 0): L2 penalty on the weights of G.--D_L1(default 0): L1 penalty on the weights of D.--D_L2(default 1e-4): L2 penalty on the weights of D.--D_iterations(default 1): How often to optimize D per batch (e.g. 2 for D and 1 for G means that D will be trained twice as much).--G_iterations(default 1): How often to optimize G per batch.--D_maxAcc(default 1.01): Stop training of D roughly around that accuracy level until G has catched up. (Sounds good in theory, doesn't produce good results in practice.)--D_clamp(default 1): To which value to clamp D's gradients (e.g. 5 means -5 to +5, 0 is off).--G_clamp(default 5): To which value to clamp G's gradients (e.g. 5 means -5 to +5, 0 is off).--D_optmethod(default "adam"): Optimizer to use for D, either "sgd" or "adam" or "adagrad".--G_optmethod(default "adam"): Optimizer to use for D, either "sgd" or "adam" or "adagrad".--threads(default 8): Number of threads.--gpu(default 0): Index of the GPU to train on (0-4 or -1 for cpu). Nothing is optimized for CPU.--noiseDim(default 100): Dimensionality of noise vector that will be fed into G.--window(default 3): ID of the first plotting window (in display), will also use about 3 window-ids beyond that.--scale(default 32): Scale of the images to train on (height, width). Loaded images will be converted to that size. Only optimized for 32.--seed(default 1): Seed to use for the RNG.--weightsVisFreq(default 0): How often to update the windows showing the activity of the network (only if >0; implies starting withqluainstead ofthif set to >0).--grayscale: Whether to activate grayscale mode on the images, i.e. training will happen on grayscale images.--denoise: If added as parameter, the script will try to load a denoising autoencoder fromlogs/denoiser_CxHxW.net, where C is the number of image channels (1 or 3), H is the height of the images (see--scale) and W is the width. A denoiser can be trained usingtrain_denoiser.lua.
- Training was done with Adam.
- Batch size was 32.
- The file
train_c2f.luais used to train a coarse to fine network of the laplacian pyramid. (Deprecated)