What is deep learning?

Applications

Google Street-View (and ReCaptchas)

Better than Human

Machine learning vs. Classical programming

Machine learning: a new programming paradigm

Why Computer Vision is difficult?

How Computer see the above picture?

Deep Learning

• Neural Networks
• Multiple layers
• Fed with lots of Data

History

• 1980+ : Lots of enthusiasm for NNs
• 1995+ : Disillusionment = A.I. Winter (v2+)
• 2005+ : Stepwise improvement : Depth
• 2010+ : GPU revolution : Data

Who is involved

Google	Hinton (Toronto)
Facebook	LeCun (NYC)
Universities	Bengio (Montreal)
Baidu	Ng (Stanford)

Andrew Ng:

“AI is the new electricity.”

2011, Image Classification

Deep Learning started to beat other approaches...

What makes deep learning different?

Why deep learning? Why now?

In general, three technical forces are driving advances:

Before we begin: the mathematical building blocks of neural networks

We will discuss:

• A first example of a neural network
• Tensors and tensor operations
• How neural networks learn via backpropagation and gradient descent

We will use Python in examples

Python Data Science Handbook. Essential Tools for Working with Data by: Jake VanderPlas

• Read the book in its entirety online at https://jakevdp.github.io/PythonDataScienceHandbook/
• The book's Jupyter notebooks: https://github.com/jakevdp/PythonDataScienceHandbook

A first look at a neural network

	 
		import keras
		from keras.datasets import mnist
		(train_images, train_labels), (test_images, test_labels) = mnist.load_data()
		from keras import models
		from keras import layers

		network = models.Sequential()
		network.add(layers.Dense(512, activation='sigmiod', input_shape=(28 * 28,)))
		network.add(layers.Dense(10, activation='sigmiod'))
		
		network.compile(optimizer='sgd',
					loss='mean_squared_error',
					metrics=['accuracy'])
		train_images = train_images.reshape((60000, 28 * 28))
		train_images = train_images.astype('float32') / 255

		test_images = test_images.reshape((10000, 28 * 28))
		test_images = test_images.astype('float32') / 255
		from keras.utils import to_categorical

		train_labels = to_categorical(train_labels)
		test_labels = to_categorical(test_labels)
		network.fit(train_images, train_labels, epochs=5, batch_size=128)
	 
	 

Compilation step

• An optimizer—The mechanism through which the network will update itself based on the data it sees and its loss function.
• A loss function—How the network will be able to measure its performance on the training data, and thus how it will be able to steer itself in the right direction.
• Metrics to monitor during training and testing—Here, we’ll only care about accuracy (the fraction of the images that were correctly classified)

Data representations for neural networks

Tensors

Don’t confuse a 5D vector with a 5D tensor! A 5D vector has only one axis and has five dimensions along its axis, whereas a 5D tensor has five axes (and may have any number of dimensions along each axis).

Dimensionality can denote either the number of entries along a specific axis (as in the case of our 5D vector) or the number of axes in a tensor (such as a 5D tensor), which can be confusing at times. In the latter case, it’s technically more correct to talk about a tensor of rank 5 (the rank of a tensor being the number of axes), but the ambiguous notation 5D tensor is common regardless.

2.2.6 Manipulating tensors in Numpy

2.2.8 Real-world examples of data tensors

1. Vector data—2D tensors of shape
(samples, features)
2. Timeseries data or sequence data—3D tensors of shape
(samples, timesteps, features)
3. Images—4D tensors of shape
(samples, height, width, channels) or
(samples, channels, height, width)
4. Video—5D tensors of shape
(samples, frames, height, width, channels) or
(samples, frames, channels, height, width)

The gears of neural networks: tensor operations

1. Element-wise operations
2. Broadcasting
3. Tensor dot
4. Tensor reshaping

Tensor Operations

	 
import numpy as np
x = np.random.random((3, 2))
print(x)
y = np.ones((2,))/2
print(y)
z = np.maximum(x, y)
print(z.shape)
print(z)
z = x+y
print(z)
z = x*y
print(z)
	 
	 

A geometric interpretation of deep learning

The engine of neural networks: gradient-based optimization

1. What’s a derivative?
2. Derivative of a tensor operation: the gradient
3. Stochastic gradient descent
4. Chaining derivatives: the Backpropagation algorithm

Intro to optimization in deep learning

1. Intro to optimization in deep learning: Gradient Descent
2. Intro to optimization in deep learning: Momentum, RMSProp and Adam
3. Intro to optimization in deep learning: Busting the myth about batch normalization
4. Adam — latest trends in deep learning optimization

Various Gradient Descent Algorithms

Stochastic Gradient Descent

TensorFlow Operations

	 
		import tensorflow as tf
		x = tf.constant(3.0)
		with tf.GradientTape(persistent=True) as g:
		  g.watch(x)
		  y = x * x
		  z = y * y
		dy_dx = g.gradient(y, x)  # 6.0
		dz_dx = g.gradient(z, x)  # 108.0 (4*x^3 at x = 3)
		dz_dy = g.gradient(z, y)  # 18.0 (2*y at y = 9)
		del g  # Drop the reference to the tape
		print(dy_dx)
		print(dz_dx)
		print(dz_dy)
	 
tf.Tensor(6.0, shape=(), dtype=float32)
tf.Tensor(108.0, shape=(), dtype=float32)
tf.Tensor(18.0, shape=(), dtype=float32)
	 

Understanding convolutional neural networks

1. Convolution arithmetic tutorial
2. Machine Learning and AI - Bangalore Chapter
3. Counting No. of Parameters in Deep Learning Models by Hand

		 
import keras
from keras import layers
from keras import models

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))
		 
		 

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 26, 26, 32)        320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 13, 13, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 11, 11, 64)        18496     
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 5, 5, 64)          0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 3, 3, 64)          36928     
_________________________________________________________________
flatten_1 (Flatten)          (None, 576)               0         
_________________________________________________________________
dense_1 (Dense)              (None, 64)                36928     
_________________________________________________________________
dense_2 (Dense)              (None, 10)                650       
=================================================================
Total params: 93,322
		 

		 
import keras
from keras import layers
from keras import models

model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.Flatten())
model.add(layers.Dense(64, activation='relu'))
model.add(layers.Dense(10, activation='softmax'))	
		 
		 

• Download Images from Kaggle
• Download Images from fastai 845MB, (need some manipulation)

		 
model = models.Sequential()
model.add(layers.Conv2D(32, (3, 3), activation='relu',
                        input_shape=(150, 150, 3)))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(64, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Conv2D(128, (3, 3), activation='relu'))
model.add(layers.MaxPooling2D((2, 2)))
model.add(layers.Flatten())
model.add(layers.Dense(512, activation='relu'))
model.add(layers.Dense(1, activation='sigmoid'))			
		 
		 

Neural style transfer

1. My Pages at: mamintoosi.ir: Neural Style Transfer, Fast Style Transfer
2. Additional outputs on my Github: Foxes
3. TensorFlow documentation

Generative Adversarial Networks