This is a few lightweight classes to apply a trained neural net. The main design principles are:
- Minimal dependencies: The core class should only depend on C++11 and Eigen. The JSON parser to read in NNs also requires boost PropertyTree.
- Flat structure: Each layer in the NN inherits from the
ILayerorIRecurrentLayerabstract base class, the NN itself is just a stack of derived classes. - Easy to extend: Should cover 95% of deep network architectures we would realistically consider.
- Hard to break: The NN constructor checks the serialized NN for
consistency. To prevent bugs with incorrectly ordered variables,
all inputs and outputs are stored in
std::maps.
We also include converters from several popular formats to the lwtnn
JSON format. Currently the following formats are supported:
- AGILEPack
- Keras (most popular, see supported layers)
- Julian's format, based on Numpy and JSON
In particular, the following layers are implemented in the Keras sequential and functional models:
| K sequential | K functional | |
|---|---|---|
| Dense | yes | yes |
| Normalization | yes | yes |
| Maxout | yes | yes |
| Highway | yes | yes |
| LSTM | yes | soon |
| GRU | yes | soon |
| Embedding | sorta | soon |
| Concatenate | no | yes |
The converter scripts can be found in converters/. Run them with
-h for more information.
After running make, just run ./tests/test-agilepack.sh. If nothing
goes wrong you should see something like:
Running conversion ./convert/agile2json.py data/agile-in.yml
Testing with ./bin/lwtnn-test-arbitrary-net
all outputs within thresholds!
cleaning up
There may be some problems if you don't have python 3 or don't have
pyyaml installed, but these should be limited to the YAML ->
JSON converter. At the very least calling ./bin/lwtag-test-hlwrapper
with no arguments (which doesn't depend on the converter) should work.
Take a look inside the test routine, it does a few things:
- Runs
./converters/agile2json.py. This takes an AGILEPack output and write a JSON file to standard out. - Sends the output to
./bin/lwtag-test-arbitrary-net. This will construct a NN from the resulting JSON and run a single test pattern.
Of course this isn't very useful, to do more you have to understand...
Code is intentionally organized into only a few files to make it easier to copy into existing packages. The main files are:
Stackfiles: contain the low level NN classes, and any code that relies on Eigen.LightweightNeuralNetworkfiles: contain the high-level wrappers, which implement STL (rather than Eigen) interfaces. To speed up compilation the header file can be included without including Eigen.NNLayerConfigheader: defines the structures to initialize networks.parse_jsonfiles: contain functions to build the config structures from JSON.
There are a few other less important files that contain debugging code and utilities.
Open include/LightweightNeuralNetwork.hh and find the class
declaration for LightweightNeuralNetwork. The constructor takes
three arguments:
- A vector of
Inputs: these structures give the variablename,offset, andscale. Note that these are applied asv = (input + offset) * scale, so if you're normalizing inputs with somemeanandstandard_deviation, these are given byoffset = - meanandscale = 1 / standard_deviation. - A vector of
Layer 763B Configstructures. See the below section for an explanation of this class. - A vector of output names.
The constructor should check to make sure everything makes sense
internally. If anything goes wrong it will throw a
NNConfigurationException.
After the class is constructed, it has one method, compute, which
takes a map<string, double> as an input and returns a map of named
outputs (of the same type). It's fine to give compute a map with
more arguments than the NN requires, but if some argument is missing
it will throw an NNEvaluationException. All the exceptions inherit
from LightweightNNException.
The Stack class is initialized with two parameters: the number of
input parameters, and a std::vector<LayerConfig> to specify the
layers. Each LayerConfig structure contains:
- A vector of
weights. This can be zero-length, in which case no matrix is inserted (but the bias and activation layers are). - A
biasvector. Again, it can be zero length for no bias in this layer. - An
activationfunction. Defaults toLINEAR(i.e. no activation function).
Note that the dimensions of the matrices aren't specified after the
n_inputs in the Stack constructor, because this should be
constrained by the dimensions of the weight vectors. If something
doesn't make sense the constructor should throw an
NNConfigurationException.
The Stack::compute(VectorXd) method will return a VectorXd of
outputs.
The lwtnn-test-arbitrary-net executable takes in a JSON file along
with two text files, one to specify the variable names and another to
give the input values. Run with no arguments to get help.
You can also use lwtnn-test-keras-arbitrary-net.py to test the
corresponding model saved in the Keras format.
Currently we support LSTMs and GRUs in sequential models. The low
level interface is implemented as RecurrentStack. See
lwtnn-test-rnn for a working example.
Again, the corresponding model in Keras can be tested with
lwtnn-test-keras-rnn.py.
If you find a bug in this code, or have any ideas, criticisms,
etc, please email me at dguest@cern.ch.
