درباره‌ی کارگاه یادگیری عمیق

Topics

1. Introduction
   • Machine Learning, Deep Learning
   • History, Applications
2. Practice
   • Tensors
   • Auto Gradient
   • Regression
   • MLP for Digit Classification
   • CNN for Image Classification
3. Some Further Notes
   • More Applications
   • Some Limitations on Real World Problems

Source books

Source books

کاربرد بسط تیلور در کاهش حجم شبکه‌های عصبی پیچشی برای طبقه‌بندی نقاشی‌های سبک امپرسیونیسم و مینیاتور

انتقال سبک برای افزایش داده‌های آموزشی شبکه‌های کانولوشنی در شناسایی شعله‌ی آتش

Machine Learning and Deep Learning

Almost all problems related to machine learning is a version of:

Some related concepts:

ML vs DL for Face Recognition

What is deep learning?

Deep learning is a class of machine learning algorithms that  uses multiple layers to progressively extract higher-level features from the raw input. For example, in image processing, lower layers may identify edges, while higher layers may identify the concepts relevant to a human such as digits or letters or faces. [Wikipedia]

Types of deep learning

1. Convolutional Neural Networks, VGG, AlexNet, ResNet,...
2. RNNs, ELMO, LSTM, BERT, GPT, ...
3. Graph Convolutional Networks (GNNs), GraphSAGE, Cluster-GCN, Heterogeneous GCNs
• Laplacian matrix
• Spectral Clustering

Why Computer Vision is difficult?

How Computer see the above picture?

History of Deep Learning

• 1980+ : Lots of enthusiasm for NNs
• 1995+ : Disillusionment = A.I. Winter (v2+)
• 2005+ : Stepwise improvement : Depth
• 2010+ : GPU revolution : Data

Who is involved

Google	Hinton (Toronto)
Facebook	LeCun (NYC)
Universities	Bengio (Montreal)
Baidu	Ng (Stanford)

Andrew Ng:

“AI is the new electricity.”

Why deep learning? Why now?

In general, three technical forces are driving advances:

1. Hardware
   NVIDIA GPUs, Google TPUs, AMD Radeon
2. Algorithmic advances
   Better activation functions
   Better weight-initialization schemes
   Better optimization schemes
3. Datasets and benchmarks
   Flickr, YouTube videos and Wikipedia

Before we begin: the mathematical building blocks of neural networks

We will discuss:

• Tensors and tensor operations
• How neural networks learn via backpropagation and gradient descent
• A first example of a neural network
• And then CNNs

We will use Python in examples

Python Data Science Handbook. Essential Tools for Working with Data by: Jake VanderPlas

• Read the book in its entirety online at https://jakevdp.github.io/PythonDataScienceHandbook/
• The book's Jupyter notebooks: https://github.com/jakevdp/PythonDataScienceHandbook

Data representations for neural networks

Tensors

Don’t confuse a 5D vector with a 5D tensor! A 5D vector has only one axis and has five dimensions along its axis, whereas a 5D tensor has five axes (and may have any number of dimensions along each axis).

Dimensionality can denote either the number of entries along a specific axis (as in the case of our 5D vector) or the number of axes in a tensor (such as a 5D tensor), which can be confusing at times. In the latter case, it’s technically more correct to talk about a tensor of rank 5 (the rank of a tensor being the number of axes), but the ambiguous notation 5D tensor is common regardless.

Manipulating tensors in Numpy

Real-world examples of data tensors

1. Vector data—2D tensors of shape
(samples, features)
2. Timeseries data or sequence data—3D tensors of shape
(samples, timesteps, features)
3. Images—4D tensors of shape
(samples, height, width, channels) or
(samples, channels, height, width)
4. Video—5D tensors of shape
(samples, frames, height, width, channels) or
(samples, frames, channels, height, width)

The engine of neural networks: gradient-based optimization

1. What’s a derivative?
2. Derivative of a tensor operation: the gradient
3. Stochastic gradient descent
4. Chaining derivatives: the Backpropagation algorithm

Intro to optimization in deep learning

1. Intro to optimization in deep learning: Gradient Descent
2. Intro to optimization in deep learning: Momentum, RMSProp and Adam
3. Intro to optimization in deep learning: Busting the myth about batch normalization
4. Adam — latest trends in deep learning optimization

Stochastic Gradient Descent

Auto Gradient

• A Gentle Introduction to torch.autograd - PyTorch
• pytorch-101-understanding-graphs-and-automatic-differentiation
• pytorch-autograd-explained

Auto Gradient in PyTorch

					
						from torch import autograd
						x = torch.Tensor([4])
						x.requires_grad_()
						y = torch.Tensor([0])
						y.requires_grad_()

						f = 5 * x**2 + 3*torch.sin(y)

						df_dx = autograd.grad(f, x)[0]
						print("∂f/∂x| x=4 :", df_dx.item())

						df_dy = autograd.grad(f, y)[0]
						print("∂f/∂y| y=0 :", df_dy.item())
					
	∂f/∂x| x=4 : 40.0
	∂f/∂y| y=0 : 3.0
					

Auto Gradient for Regression

Auto Gradient for Regression

									
									for epoch in range(num_epochs):
										y_hat = b + w * xTrain
										error = (y_hat - yTrain)
										loss = (error ** 2).mean()
										loss.backward()
										with torch.no_grad():
											b -= lr * b.grad
											w -= lr * w.grad
											b.grad = None
											w.grad = None
									
									

Error Back Propagation

Error Back Propagation

A first look at a neural network

	 
	class MyMultilayerPerceptron(nn.Module):
		def __init__(self, input_size, h_dim, num_classes):
			super(MyMultilayerPerceptron, self).__init__()
			
			self.input_size = input_size
			self.num_classes = num_classes
			
			self.linear_1 = nn.Linear(input_size, h_dim)
			self.linear_2 = nn.Linear(h_dim, num_classes)
		
		def forward(self, x):
			x = F.relu(self.linear_1(x))
			x = F.relu(self.linear_2(x))
			return F.softmax(x)
	
	 

	 
		# model is a regression or classification
		optimizer = torch.optim.SGD(model.parameters(), lr=1e-3)  
		
		for epoch in range(num_epochs):
		  for (inputs, labels) in data_loader:
			# Do Forward -> Loss Computation -> Backward -> Optimization
			optimizer.zero_grad()
			outputs = model(inputs)
			# loss is prediction error, such as: Sum(outputs-labels)^2
			loss.backward()
			optimizer.step()
	
	 

Compilation step

• An optimizer—The mechanism through which the network will update itself based on the data it sees and its loss function.
• A loss function—How the network will be able to measure its performance on the training data, and thus how it will be able to steer itself in the right direction.
• Metrics to monitor during training and testing—Here, we’ll only care about accuracy (the fraction of the images that were correctly classified)

Image Classification with CNNs

Understanding convolutional neural networks

1. Convolution arithmetic tutorial
2. Machine Learning and AI - Bangalore Chapter
3. Counting No. of Parameters in Deep Learning Models by Hand

Mathematical definition of Convolution

Differentiation, Inner Product and Convolution

Edge extraction is a simple derivative:
One of the edge detection operators is Prewitt filter:

-1	0	+1
-1	0	+1
-1	0	+1

Sobel Filter

Laplacian

Laplacian

Laplacian

						 
	for (inputs, labels) in data_loader:
		inputs = inputs.to(device)
		labels = labels.to(device)
		
		# original shape is [batch_size, 28, 28] because it's an image of size 28x28
		inputs = inputs.view(-1, 28*28)

		# Do Forward -> Loss Computation -> Backward -> Optimization
		optimizer.zero_grad()
		outputs = model(inputs)
		loss = criterion(outputs, labels)
		loss.backward()
		optimizer.step()
							
						 

Language Models

Neural style transfer

1. My Pages at: mamintoosi.ir: Neural Style Transfer, Fast Style Transfer
2. Additional outputs on my Github: Foxes
3. My paper about fire detection

Generative Adversarial Networks

Andrew Ng:

“Data is food for AI”

“The model and the code for many applications are basically a solved problem”

دنیای واقعی

آشنایی با موضوع کاری

،برنامه‌نویسی

زبان‌های مختلف، مخصوصاً پایتون

SQL

لینوکس و یک سری ابزار همچون گیت و داکر

آماده‌سازی داده‌ها ۸۰ درصد کار است، فقط ۲۰ درصد مدلسازی است
که عموما راه حلش موجود است

Mike West:
Machine learning engineers do NOT create models, they call them.
Data cleansing is 80% of my job.
Modeling is about 10% and it’s the easy part.

More data is better than of tweaking algorithms in practice

Mike West answer to: Should a machine learning beginner go straight for deep learning?