This fork extends LWTNN to somehow support training of networks inside the library. This was achieved by using the autodiff-library under the hood. For the parameter-update, a very simple SGD is used. The changes are fully experimental, and at the moment only enabled for networks, not for graphs.
-
Most classes are now template classes, so the default numeric type
doublecan now be changed toautodiff::var. To not break the library, the templating is done with the following scheme:Before:
class Stack { // something with double... };
Now:
template<typename T> class StackT { // something generic }; using Stack = StackT<double>;
As a consequence, some
.cxx-files are now empty, and replaced by.txx-files in theinclude-folder. -
The Fitting is encapsulated in the class
FittableLWTNN, which inherits the classLightweightNeuralNetworkT<autodiff::var>. To enable this, the formerprivatesection is nowprotected. Additionally, this class is friend with the layer classes and the stack-class (is this hacky?). -
Currently,
dynamic_cast<>is used to get access to the layer data. This is probably hacky, but also easily to solve. -
Because of the autodiff-library, C++17 is now required.
The interface resembles somehow the keras interface. Until now, it was only tested for simple regressions with small networks, consisting of a few dense layers.
double learning_rate = 0.01;
std::size_t epochs = 250;
std::size_t batch_size = 16;
auto history = nn.fit(x_train, y_train, x_valid, y_valid, learning_rate, batch_size, epochs);The training data can be either provided as std::vector<ValueMap> or as std::vector<Eigen::VectorXvar>.
There is a new test-binary called "test-fittable-networks", which demonstrates the use of FittableLWTNN.
- no graphs supported
- many compiler warnings (but also because of autodiff...)
- very experimental state and poor testing
- a bit hacky approach with
friendanddynamic_cast<>(need of proper getters/setters...) - performance for larger networks not very good. This is not much investigated, just some attempts with OpenMP were tried...
What follows now is the original readme.
The code comes in two parts:
- A set of scripts to convert saved neural networks to a standard JSON format
- A set of classes which reconstruct the neural network for application in a C++ production environment
The main design principles are:
-
Minimal dependencies: The C++ code depends on C++11, Eigen, and boost PropertyTree. The converters have additional requirements (Python3 and h5py) but these can be run outside the C++ production environment.
-
Easy to extend: Should cover 95% of deep network architectures we would realistically consider.
-
Hard to break: The NN constructor checks the input NN for consistency and fails loudly if anything goes wrong.
We also include converters from several popular formats to the lwtnn
JSON format. Currently the following formats are supported:
- Scikit Learn
- Keras (most popular, see below)
Our underlying assumption is that training and inference happen in very different environments: we assume that the training environment is flexible enough to support modern and frequently-changing libraries, and that the inference environment is much less flexible.
If you have the flexibility to run any framework in your production environment, this package is not for you. If you want to apply a network you've trained with Keras in a 6M line C++ production framework that's only updated twice a year, you'll find this package very useful.
Clone the project from github:
git clone git@github.com:lwtnn/lwtnn.gitThen compile with make. If you have access to a relatively new
version of Eigen and Boost everything should work without errors.
If you have CMake, you can build with no other dependencies:
mkdir build
cd build
cmake -DBUILTIN_BOOST=true -DBUILTIN_EIGEN=true ..
make -j 4If you have Python 3 and h5py installed you can run a test. Starting from the directory where you built the project, run
./tests/test-GRU.sh
(note that if you ran cmake this is ../tests/test-GRU.sh)
You should see some printouts that end with *** Success! ***.
The following instructions apply to the model/functional API in Keras. To see the instructions relevant to the sequential API, go to Quick Start With sequential API.
After building, there are some required steps:
Make sure you saved your architecture and weights file from Keras, and created your input variable file. See the lwtnn Keras Converter wiki page for the correct procedure in doing all of this.
Then
lwtnn/converters/kerasfunc2json.py architecture.json weights.h5 inputs.json > neural_net.json
Helpful hint: if you do lwtnn/converters/kerasfunc2json.py architecture.json weights.h5 it creates a skeleton of an input file for you, which can be used in the above command!
A good idea is to test your converted network:
./lwtnn-test-lightweight-graph neural_net.json
A basic regression test is performed with a bunch of random numbers. This test just ensures that lwtnn can in fact read your NN.
// Include several headers. See the files for more documentation.
// First include the class that does the computation
#include "lwtnn/LightweightGraph.hh"
// Then include the json parsing functions
#include "lwtnn/parse_json.hh"
...
// get your saved JSON file as an std::istream object
std::ifstream input("path-to-file.json");
// build the graph
LightweightGraph graph(parse_json_graph(input));
...
// fill a map of input nodes
std::map<std::string, std::map<std::string, double> > inputs;
inputs["input_node"] = {{"value", value}, {"value_2", value_2}};
inputs["another_input_node"] = {{"another_value", another_value}};
// compute the output values
std::map<std::string, double> outputs = graph.compute(inputs);After the constructor for the class LightweightNeuralNetwork is
constructed, it has one method, compute, which takes a map<string, double> as an input and returns a map of named outputs (of the same
type). It's fine to give compute a map with more arguments than the
NN requires, but if some argument is missing it will throw an
NNEvaluationException.
All inputs and outputs are stored in std::maps to prevent bugs with
incorrectly ordered inputs and outputs. The strings used as keys in
the map are specified by the network configuration.
In particular, the following layers are supported as implemented in the Keras sequential and functional models:
| K sequential | K functional | |
|---|---|---|
| Dense | yes | yes |
| Normalization | See Note 1 | See Note 1 |
| Maxout | yes | yes |
| Highway | yes | yes |
| LSTM | yes | yes |
| GRU | yes | yes |
| Embedding | sorta | issue |
| Concatenate | no | yes |
| TimeDistributed | no | yes |
| Sum | no | yes |
Note 1: Normalization layers (i.e. Batch Normalization) are only supported for Keras 1.0.8 and higher.
| Function | Implemented? |
|---|---|
| ReLU | Yes |
| Sigmoid | Yes |
| Hard Sigmoid | Yes |
| Tanh | Yes |
| Softmax | Yes |
| ELU | Yes |
| LeakyReLU | Yes |
| Swish | Yes |
The converter scripts can be found in converters/. Run them with
-h for more information.
For more in-depth documentation please see the lwtnn wiki.
If you find a bug in this code, or have any ideas, criticisms,
etc, please email me at dguest@cern.ch.