Jump to content

Neural Network from first principles

Recommended Posts

  • Moderators
These are exciting times.

Hardly any day goes by without an announcement of another victory in artificial intelligence, deep learning or machine learning.


As machine learning becomes more mainsteam, chances are that at some point in time it will become important to be able to incorporate this type of problem solving in projects, so I did some read-up on the subject. 


There is no lack of available tools. Google has opensourced TensorFlow, Facebooks's FAIR team has added amazing pre-trained networks on github, Microsoft has its Cortana Intelligence Suite on Azure and Amazon offers machine learning services on AWS.

Just to name a few


On the plus-side most of these services can be used on a freemium basis and given the companies involved these services will be cutting edge, highly performance optimised and scalable.


On the down-side and unless you're an expert in these fields (which I'm not), these services are to a large extent black boxes.

How they work internally is largely hidden and how to judge the relevance of its outputs can be difficult.


Personally I like to at least have an idea of the internals of tools like this.

There have been posts on using Brain and Synaptic as neural networks in SMS in this forum but those posts focus more on the technical implementation rather than on working principles of neural networks.


So this post is on implementing a neural network in SMS from first principles.

Maybe not the most optimised solution but code should be readable.


There are many different flavours of neural networks but the usual structure is a set of neurons arranged in layers.

The first layer is the input layer, the last layer hosts one or more output neurons and inbetween are zero to many hidden layers.

Neurons in these layers are connected to each other and the 'intelligence' of a network is expressed internally as a weight given to each and every connection.


Neurons 'fire' when the input is higher than a certain threshold. This threshold is defined by a mathematical function, usually called the activation function. All neurons in a particular layer must have the same activation function and usually, but not always, a network has the same activation function for all layers.


  TLayer = Class(TObject)
    ActivationType : String;           // ['Sigmoid','Linear','Tanh']
    LayerType      : String;           // ['Input','Hidden','Output']
    LayerName      : String;
  TNeuron = Class(TObject)
    NeuronType     : String;           // ['Normal','Bias','Memory'];
    NeuronOutput   : Float;
    NeuronError    : Float;
    Layer          : TLayer;
  TWeight = Class(TObject)
    NeuronFrom     : TNeuron;
    NeuronTo       : TNeuron;
    Weight         : float;
  TNeuralNetwork = class(TW3CustomControl)
    procedure InitializeObject; override;
    procedure FinalizeObject; override;
    procedure Resize; override;
    Layers         : array of TLayer;
    Neurons        : array of TNeuron;
    Weights        : array of TWeight;


TNeuralNetwork is derived as a visual component here so the internal structure can be made visible (drawn on the canvas of a paintbox) later on.


The topology of a network is defined by adding layers, adding neurons and assigning each neuron to a specific layer


procedure TForm1.InitializeForm;
  // this is a good place to initialize components
  MyNetwork := TNeuralNetwork.Create(Self);
  MyNetwork.AddLayer  ('InputLayer', 'Input',  '');            // input layer. input layers have no activation function
  MyNetwork.AddLayer  ('Hidden-1',   'Hidden', 'Sigmoid');     // first hidden layer, with sigmoid activation
  MyNetwork.AddLayer  ('Hidden-2',   'Hidden', 'Sigmoid');     // second hidden layer, sigmoid activation
  MyNetwork.AddLayer  ('OutputLayer','Output', 'Sigmoid');     // output layer, with sigmoid activation

  MyNetwork.AddNeurons(2, 'Input', 'InputLayer');    // 2 input neurons in the input layer

  MyNetwork.AddNeurons(1, 'Bias',  'InputLayer');    // plus 1 bias neuron

  MyNetwork.AddNeurons(3, 'Normal','Hidden-1');      // 3 normal neurons first hidden layer
  MyNetwork.AddNeurons(1, 'Bias',  'Hidden-1');      // plus 1 bias neuron

  MyNetwork.AddNeurons(3, 'Normal','Hidden-2');      // 3 normal neurons second hidden layer
  MyNetwork.AddNeurons(1, 'Bias',  'Hidden-2');      // plus 1 bias neuron

  MyNetwork.AddNeurons(1, 'Output','OutputLayer');   // 1 output neuron


Neurons in the input layer don't have an activation function. Input values of these neurons will just be copied to its output. 

Bias neurons are optional. They are a special type of neuron and are sometimes added to give a gentle push to the learning process. Bias neurons always have a standard output of (usually) 1.


Procedure TNeuralNetwork.AddLayer (LayerName: String; LayerType: String; ActivationType: String);
  Layer := TLayer.Create;
  Layer.LayerName := LayerName;
  Layer.LayerType := LayerType;
  Layer.ActivationType := ActivationType;
Procedure TNeuralNetwork.AddNeurons (&Number: Integer; NeuronType: String; LayerName: String);
  For var j := 0 to &Number -1 do begin
    Neuron := TNeuron.Create;
    Neuron.NeuronType := NeuronType;
    Neuron.NeuronOutput := 0;
    If Neuron.NeuronType = 'Bias' then
      Neuron.NeuronOutput := 1;
    Neuron.NeuronError := 0;
    For var i := 0 to Layers.Count -1 do begin
      If Layers[i].LayerName = LayerName then begin
        Neuron.Layer := Layers[i];


Once the structure is established, the network can be initialised.

This involves setting up all connections between the neurons and providing them with an initial random weight valued between 0 and +1.

Randomize sets the random generator seed to a random value.


Procedure TNeuralNetwork.Initialise;
  For var i := 0 to Layers.Count -2 do begin                  //Weight objects live in between layers so don't go past last layer
    For var j := 0 to Neurons.Count -1 do begin
      If Neurons[j].Layer = Layers[i] then begin
        For var k := 0 to Neurons.Count -1 do begin
          If Neurons[k].Layer = Layers[i+1] then begin
            If Neurons[k].NeuronType <> 'Bias' then begin    //bias neurons are never linked to, only linked from
              Weight := TWeight.Create;
              Weight.NeuronFrom := Neurons[j];     //showmessage('from ' + inttostr(j));
              Weight.NeuronTo   := Neurons[k];     //showmessage('to   ' + inttostr(k));
              Weight.Weight := RandomInt(1) + Random;


Next post feedforward and backpropagation training

Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

  • Create New...