Marcin Drobik

software journeyman notes

Introduction to Neural Networks - Part 1

Before we start talking about Neural Networks, lets talk about specific implementation of simple logic functions. By using them we'll learn basic math and graphical representations of Neural Networks.

Sigmoid

When we implemented the Logistic Regression we used the sigmoid function. This function is defined as:

... and looks like that:

The important properties for us is that for large values of t the S(t) approaches 1 and for small (negative) values of t the S(t) approaches 0.

And

Logic "and" is a function of two parameters - a and b. To use it in arithmetic calculations we'll assume that values close to 0 represents logical false, and values close to 1 represent logical true.

Let's look at following function (S is sigmoid):

f = S(-30 + 20a + 20b)

When looking at the various values of a and b we see that it indeed implements the logical And - it only outputs value close to 1 when both a and b are 1:

a b f
0 0 S(-30) ~ 0
1 0 S(-10) ~ 0
0 1 S(-10) ~ 0
1 1 S(10) ~ 1

Neuron representation

Our function f can be represented as a single neuron on following graph:

In neural networks terminology, the a and b are Inputs, the 1 is special input called Bias Input, the orange circle is a Neuron (with activation function S(t)) and the arrows in between represent Weights of the inputs for particular Neuron. The arrow on the right represents Output of the neuron - in this case, our function f = S(-30 + 20a + 20b)

Vectorized version

It's very easy to represent this single neuron using matrix operations in Stratosphere.NET. First let's define our inputs as vector x:

var x = Matrix.Vector(1, 0, 1); // a = 0; b = 1

... and the weights as vector theta:

var theta = Matrix.Vector(-30, 20, 20);

Now the neuron output is simply the multiplication:

var f = Sigmoid(x*theta.T);

In next post we'll see how to create simple neural networks.

comments powered by Disqus