Neural Networks with TensorFlow

In this workshop we will be trainin a neural network, using tensorflow, to classify different clothing pieces in the fashion-mnist dataset.

What is Tensorflow?

Tensorflow(TF) is an open source library for numerical computation and machine learning. Initially when Tensorflow 1.0 was released, it had a high learning curve do to its unconventional structure and un-pythonic syntax. This led to the release of Tensorflow 2.0 which incorporated Keras; a simple to use deep learning framework. Due to this change Tensorflow is now intuitive and easy to use.

How does Tensorflow work?

Tensorflow uses tensors for their computation. A tensor is an immutable multidimensional array with a uniform datatype. This can be see as similar to a Numpy array.

Let’s see how we can implement a tensor using Tensorflow

import tensorflow as tf

t_one = tf.constant([1,2,3])
print("[1]", t_one)

t_two = tf.constant([5])
t_three = tf.multiply(t_one, t_two)
print("[2]", t_three)

print log

[1] tf.Tensor([1 2 3], shape=(3,), dtype=int32)
[2] tf.Tensor([ 5 10 15], shape=(3,), dtype=int32)

As you can see above, we used the tf.constant() class to create an instance of a tensor. We were also able to use functions such as tf.multiply() in order to perform an operation on our tensors. For the more information on tensors in tensorflow you can check out the documentation here.

What are Neural Networks?

Neural Networks(NN) are sets of interconnected neurons that take in a large set of data and aim to discover an underlying pattern. While neural networks don’t exactly emulate our biological neural networks, they are loosely inspired by how our brains learn.

Our brain doesn’t learn concepts instantly, it usually takes different iterations of explanations and even trial and error to solidify our understanding of an idea. Imagine if you had no understanding of what cats and dogs were. If I showed you thousands of different pictures of cats and dogs, you would start to form an understanding and eventually you would become more confident in telling me which pictures were cats and which were dogs. While oversimplified, this is the basis of how neural networks learn.

Now that you have a high level overview of how neural networks learn, let’s dive deeper and take a look at their structure.

Above, we can see an example image. The neural network is composed of 3 types of layers:

• Input Layer—This is the input of our network where we pass in our data(This could be text, images, sound, etc.)
• Hidden Layer—This layer falls in between the input and output layers. It does the mapping between the input and output layers by performing a series of mathematical operations
• Output Layer—This can be seen as the results of our network

The term “deep learning” comes from neural networks with more than 1 hidden layer. These types of neural networks are called deep neural networks. The neural network figure above displays a deep neural network because it contains 2 hidden layers. Neural networks with only 1 hidden layer can be referred to as simple neural networks.

What is the function of the neuron?

Each neuron performs a set of mathematical operations to derive an output.

Above is an example of a randomly selected neuron. Each neuron will have a connection with every neuron in the previous layer. The way each neuron obtains its value is simple. It begins by summing up every input. The input consists of the input value(x) as well as an associated weight(w). When the model is trained, these weights shift around in order to get the optimal model. The weights carry influence as to how strong a connection between two neurons are. The higher the weight the greater the influence of the neuron.

Above we summed up every input which consisted of the input multiplied by the weight. We can further simplify this as the summation equals the dot product of the vectors x and w.

Furthermore, we also have to add the bias(b) term to transpose the constant value to obtain the output values.

Finally, to obtain the neurons output value, we pass it through an activation function in order to introduce non-linearity into the neurons output. Without the activation function, our neural network is essentially a linear regression model. With the activation function, our model is able to learn more complex tasks. There are several activation functions, but in this post we will be going over the ReLU activation function.

Rectified Linear Unit(ReLU) is a standard activation function. It is widely used because of how well it works. Typically when you are unsure of which activation function to use, ReLU should be your go to.

The steps outlined above occur through every neuron in the neural network until the network reaches its end. This process of passing data through the neural network is called forward propogation. Once this process is completed we move on to backpropogation.

What is backpropogation?

Back-propagation fine tunes the model based on the results of forward propogation every iteration. After a forward pass, the neural network calculates the loss function and propogates those results backwards to tune the weights and biases. Since the model is updating its weights and biases every iteration, the model should become more optimized. The goal is to lower this error rate.

The loss function is highly related to the cost function however, the loss function refers to the error of one training sample, while the cost function is the average of the entire training set

For our purposes, we’ll be using Sparse Categorical Crossentropy for the live coding workshop. While I won’t be going over the math for this loss function, it is important to note that sparse categorical crossentropy is a good loss function to use when you are dealing with multiclass classification problems like the one we’ll see in our workshop.

For all the loss functions that can be found on keras, you can check out their docs page on losses

Once we have our loss value, we backpropogate those results through an optimization technique. For our workshop, we’ll be utilizing the Adam optimizer. Similar to prevous algorithms we’ve covered, this optimizer is the standard one to use when you are unsure which to use. It is popular because of how well it works for a lot of general problems.

For all the optimizers on keras, check out their docs page on optimizers

Getting started with the workshop

For this live-coding portion, we’ll be training our own neural network using tensorflow! Our goal is to classify different clothing items from the fashion-mnist dataset.

To begin the live coding workshop click here

Let’s Begin

As usual, we begin by importing our libraries

import tensorflow as tf
from tensorflow import keras
from tensorflow.keras import layers
import matplotlib.pyplot as plt
import numpy as np

As expected we have keras and tensorflow for creating our model. We’ll also be using matplotlib in order to print out the images from the dataset. Now that our imports are in place, we also need to load our dataset.

fashion_mnist = tf.keras.datasets.fashion_mnist
(x_train, y_train), (x_test, y_test) = fashion_mnist.load_data()

This dataset comes standard with tensorflow, so we can directly import it from tensorflow. Typically, we like to seperate our data when training our model. We call this a train-test split. The reason for this is because we don’t want to test how well our model performed on our training data. Our model could just memorize the dataset and overfit to it. If we have something to compare the models accuracy compared to the accuracy of data it has never seen before, we can better judge how well our model is actually performing.

The data with the x, represents images while data with a y represented their labels. Our goal is to train a model that given x image can predict y label. Our model will learn by taking in images and making a prediction. It will then be given the answer and will make changes to the network in order to optimize the model.

x_train, x_test = x_train / 255, x_test / 255 

Before we begin training our model we have a little more preprocessing to do. We normalize our image data to be between 0 and 1. Since we are dealing with image data, we get values from 0 to 255. By dividing each image by 255, this normalizes our data to be between 0 and 1. This helps to speed up our training process since smaller numbers are being computed.

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

We also declare a list with all the labels. As mentioned we’ll be classifying between 10 different objects. Those objects are listed above.

class_names = ['T-shirt/top', 'Trouser', 'Pullover', 'Dress', 'Coat',
               'Sandal', 'Shirt', 'Sneaker', 'Bag', 'Ankle boot']

plt.figure(figsize=(10,10))
for i in range(25):
    plt.subplot(5,5,i+1)
    plt.xticks([])
    plt.yticks([])
    plt.grid(False)
    plt.imshow(x_train[i], cmap=plt.cm.binary)
    plt.xlabel(class_names[y_train[i]])
plt.show()

We then create a simple grid plot to show a couple example of the images we’re working with.

Now that we have all of our data preprocessed, we can begin by creating our model. We will use Keras to first define the structure of our model.

model = keras.Sequential() # Initializing model

model.add(layers.Flatten(input_shape=(28, 28))) # Input layer
model.add(layers.Dense(128, activation='relu')) # Hidden layer
model.add(layers.Dense(10)) # Output layer

We begin by defining our model. We create a Sequential() class that will contain all the layers of our model. This class is the standard class for creating simple neural networks. It is typically what you’ll be using for creating models.

We then use the add() function to add layers to our model. We first define our input layer by flattening the image. Neural networks take 1-dimensional data, so we flatten our image. Essentially we take all the rows and stack them vertically to create a 1-dimensional shape.

Afterwards, we add our first hidden layer. This is a standard layer with 128 neurons and it uses the relu acitvation function.

Lastly, we declare our output layer. The neurons in this layer should correspond to the amount of objects you are trying to classify. In this example we are classifying 10 different objects so we’ll have our output layer include 10 neurons.

model.compile(
    optimizer='adam',
    loss=tf.keras.losses.SparseCategoricalCrossentropy,
    metrics=['accuracy']
    )

Now we specify a couple more metrics for our model. We include the optimizer we’ll want to use as well as the loss function. We also include a metric to display the accuracy of our model as we’re training it.

model.fit(x_train, y_train, epochs=10)

Finally, we use the fit() method to begin training our model. We specify our training data as well as how many times we want to iterate through our dataset. This iteration is called an epoch. We specify 10 epochs which means it will complete a full training cycle 10 times.

test_loss, test_acc = model.evaluate(x_test,  y_test, verbose=2)

Now that our model is trained, we pair it up against our testing data to see how well it actually did.

The entire code for this workshop, including the portions not mentioned here can be found here