ANNiML is an neural network written in Objective Caml.
See also OCaml
The code is mainly inpired from the following book [Bishop96]: "Neural Networks for Pattern Recognition", Christopher M. Bishop, Oxford University Press, 1996. ISBN 0198538642.
ANNiML can be used for regression (interpolation of an unknown function), or classification (assign an input vector to a class). The network is trained on patterns provided as input of the program.
A neural network can be seen as a function
F of an input vector
It computes an output
y = F(x,w_0,w_1,w_2,...,w_p), where the
are the weights assigned to the connections between the units.
The biases of the network are considered as additionnal connections
from bias units with a constant output value
The weights must be tuned by training the neural network on a set of
t is the target vector, so as to minimize an
error depending on the difference between the output
y and the target
To do that, we need to
w*which minimizes the error on the training set.
The training methods iteratively search the weights space, and
stop either when the error is small enough (below an absolute tolerance), or
when it cannot be improved (relative tolerance), or when a maximum
number of iterations is reached.
For minimization methods with a constant step (gradient descent), it may be
better not to use the relative tolerance (use
reltol=0) as the
step may accidentally lead to a point where the relative improvement of the
objective function is less than
reltol, stopping the training too early.
The stop criterion is also an important issue in avoiding overfitting problems: the training should stop early enough to avoid overfitting the training data, to the detriment of the generalization on fresh input data.
As only local optimization methods are used, the results highly depend on the initial weights, randomly chosen before the training starts. Several runs with different initial weights should be performed.
ANNiML is not in the public domain yet. The source code is under CVS.
Retrieve the MATH and ANNiML modules under CVS:
> cvs checkout MATH > cvs checkout ANNiML
Compile the MATH libraries:
> cd MATH > make
Go to the ANNiML directory and compile:
> cd ../ANNiML > make
anniml [layers] -train <fpatterns> [other_options]or
anniml -run <input_vector> [other options]or
anniml -test <fpatterns> [other options]or
anniml -predict <finputs> [other options]
-train <fpatterns> train the network on a patterns file.
-test <fpatterns> test the trained network on a patterns file.
-run <input_vector> run the trained network on a new input vector passed as argument.
-predict <finputs> make predictions on a new input data file.
-p option is not used, results are saved by default in a new file
with a .pred extension.
-dir <dirname> working directory.
-n <net_file> file describing the network's topology. Default is the
patterns file basename with a '.net' extension.
-w <wts_file> weights file. Default is the patterns file basename with
a '.wts' extension.
-p <fpredict> save the results of the
-predict option in another file
than the input file with a .pred extension.
-out <string> output function name:
-act <string> activation function name:
-err <string> error function name
sum of squares),
-c <columns_file> file listing the columns to select in the patterns file.
-norm normalize the patterns: xi:= (xi-avg(xi))/sigma(xi)
-max <int> maximum number of iterations.
-abstol <float> absolute tolerance used in the stop condition
-reltol <float> relative tolerance used in the stop condition
-bfgs BFGS quasi-newton optimization.
-g <float> , basic gradient descent with constant step
-gm <float> <float> , gradient descent with constant step
(default 0.20), and with momentum
mu (taken between 0 and 1,
-batch batch learning for gradient descent (chunk size = nb_patterns).
-online on-line learning for gradient descent (chunk size = 1).
-chunk <int> size of the patterns blocks used to update the gradient.
Possible options for gradient descent methods are:
-fv frequency of the verbose output on stdout (default 10).
-prc print classification results.
-rand root for the random generator (default 0).
-help display this list of options.
--help display this list of options.
Each line of the patterns file must contain first an input vector
x, and second a target vector
Be cautious to use a network's topology that is consistent with your
patterns file (dimension of
x must be equal to the number of input
units, and dimension of target vector
t must be equal to the number
of output units).
There is an option (
-c) allowing to select the columns that you really
want to use in your patterns file.
When performing classification, the dimension of your target vector must be equal to the number of classes. For example, if you have three classes, then
t= (1,0,0)will mean that the vector
xbelongs to class 0,
t= (0,1,0)will mean that the vector
xbelongs to class 1,
t= (0,0,1)will mean that the vector
xbelongs to class 2.
y_ican be seen as the probability to belong to class
As already said at the beginning of this page, you must have a sufficiently large number of patterns (compared to the number of weights in the network) if you want to avoid overfitting (see [Bishop96]).
It is recommended to use only a part of your data to train the neural network, and to test the trained network on the rest of the data. Overfitting can be observed when the network fits very well on the training data, and badly on the test data.
Ideally, a cross-validation would even be better, where only a part of the training data is actually used to train the network, and the rest (validation set) is used to evaluate its performance and stop the training before overfitting arises. Several different splits of training and validation sets should be tried, as well as several random initial values for the weights, before selecting the network with best average performance. Cross-validation is not implemented yet.
Be careful to normalize the inputs of the neural network before
There is a
-norm option to do that, which removes the average value and
divides each input by the standard deviation.
However, it operates only on the pattern file given as argument of the
command line. So you may not normalize the same way on your training set
and your test set.
You should better pre-process your initial data set, and normalize the
inputs before splitting it into a training set and a test set.
A few examples of patterns files can be found in the
Ann_patterns for the functions that read, write, and
anniml, the user must choose the error function being minimized
during the training, and also the transfer functions of the different units.
These functions are described in module
The default options are set to perform regression, with:
As an illustration, here is a regression on noisy data that was produced
using the following function:
y(x)= 0.5 + 0.4 sin(2*pi*x)
to which was added a gaussian noise (standard deviation: 0.05)
> anniml 1 5 1 -train sinus_train.pat -dir examples -g 0.6 -max 20000 -reltol 0.Train a network with one hidden layer (1 input unit, 5 hidden units, and 1 unit in the output layer) on the pattern file 'sinus_train.pat', using simple gradient descent with step 0.6, with 20000 iterations at most and default absolute tolerance 1E-7. The resulting weights are saved in 'examples/sinus_train.wts'. The network's topology (fully connected) is saved in 'examples/sinus.net'.
> anniml -test sinus_test.pat -dir examples -n sinus_train.net -w sinus_train.wtsTest the trained network on the pattern file 'sinus_test.pat'. The test sample should be different from the one used to train the net.
> anniml -run 0.7 -dir examples -n sinus_train.net -w sinus_train.wtsCompute the neural network's output for an input x=0.7
> anniml -predict sinus_plot.new -dir examples -n sinus_train.net -w sinus_train.wtsRead new inputs from file 'sinus_plot.new', compute the network's outputs, and save the inputs and outputs in file 'sinus_plot.pred'
> cd examples > gnuplotGo to the examples directory and launch gnuplot
gnuplot> plot 'sinus_train.pat', 'sinus_plot.pred' w l, 0.5+0.4*sin(2*pi*x) w lPlot the noisy train data, the fitted curve, and the true function.
> anniml 1 5 1 -train sinus_train.pat -dir examples -w sinus_bfgs.wts -bfgs -max 1000Train the network, using BFGS to tune the weights. Save the resulting weights in 'examples/sinus_bfgs.wts'.
> anniml -predict sinus_plot.new -p sinus_bfgs.pred -dir examples -n sinus_train.net -w sinus_bfgs.wtsRead new inputs from 'sinus_plot.new', compute the network's outputs and save them in 'sinus_bfgs.pred'.
You can plot 'sinus_bfgs.pred' and compare with the results obtained with the simple gradient descent.
> anniml 2 5 2 -train sinclass_train.pat -dir examples -err ce -out softmax -act tanh -bfgs -max 1000 -rand 2 -prcTrain the network for classification, with a cross-entropy error function, a softmax transfer function for the output units. For a change, we chose the hyperbolic tangent activation function for the hidden units.
> anniml -test sinclass_test.pat -dir examples -n sinclass_train.net -w sinclass_train.wts -err ce -out softmax -act tanh -prcTest the trained network on the pattern file 'sinus_test.pat'.
Here are the ANNiML modules (see also file
Forward and backward propagation through the neural network.
Parsing the program parameters from the command line.
Transfer functions and error functions.
Patterns for neural networks.
Random generation of the network's weights
Compute and print a few indicators of the neural network's performance.
Neural networks topology: units, layers, and connections.
Read weights from files, and write weights to files
ANNiML main functions.
You may also refer to the following indexes:
Index of types
Index of values
Index of modules
This documentation was produced with
ocamldoc -html -t ANNiML -intro Readme -d ../../doc/ANNiML/ -hide Pervasives *.mli