Deep Learning Step5 Back Propagation and Composition Function

Deep Learning Step5

  Step 5 Back Propagation

  Why We Need Back Propagation?

In supervised learning systems, Back Propagation gets minimum value of loss function. Loss function is used to decide optimized hyper parameters which will be described later.

  Differentiation of Composition Function

 First, we need to know how to do differentiation of composition. 
For example, the following functions are ready.  

Input Data : x 
Function A(), B() ,C() 
Output each : a = A(X) , b = B(a) , c = C(b)
 
Fig.5.1 Forward Propagation

 If x is put into A(), you will get a. As well as a is put into B and b into C. This way is called Forward Propagation(Fig.5.1).
 If we want value of dC(b)/dx (c = C(b) = C(B(A(X))) => dc/dx = dC(b)/dx = dC(B(A(X)))/dx), we can solve this equation like the below.

Fig.5.2 Backward Propagation

 This data flow in Fig.5.2 is called back propagation because the value of differentiation of dc/dc is going to dc/dx in the rule of differentiation of composition function. The code below shows us forward propagation and back propagation by numerical differentiation from deep-learning-from-scratch-3/step09.py at master · oreilly-japan/deep-learning-from-scratch-3 · GitHub. 

import numpy as np

class Variable:
    def __init__(self,data):
        if data is not None:
            if not isinstance(data,np.ndarray):
                raise TypeError('{} is not supported'.format(type(data)))

        self.data = data
        self.grad = None
        self.creator = None
    
    def set_creator(self,func):
        self.creator = func

    def backward(self):
        if self.grad is None:
            self.grad = np.ones_like(self.data)

        funcs = [self.creator]
        while funcs:
            #funcs=[1,2] => a = funcs.pop() =>a=2 , funcs[1]
            f = funcs.pop()
            #A(x) x=x , y=A(x)
            x,y = f.input, f.output
            x.grad = f.backward(y.grad)
            if x.creator is not None:
                funcs.append(x.creator)


class Function:
    def __call__(self, input):
        x = input.data
        y = self.forward(x)
        output = Variable(as_array(y))
        output.set_creator(self)
        self.input = input
        self.output = output
        return output

    def forward(self,x):
        raise NotImplementedError()

    def backward(self,gy):
        raise NotImplementedError()

def as_array(x):
    if np.isscalar(x):
        return np.array(x)
    return x

class Square(Function):
    def forward(self,x):
        return x ** 2

    def backward(self,gy):
        x = self.input.data
        gx = 2 * x * gy
        return gx

class Exp(Function):
    def forward(self,x):
        return np.exp(x)

    def backward(self, gy):
        x = self.input.data
        gx = np.exp(x)*gy
        return gx

def numerical_diff(f,x,eps=1e-4):
    x0 = Variable(x.data-eps)
    x1 = Variable(x.data+eps)
    return (f(x1).data-f(x0).data)/(2*eps)

def f(x):
    A  = Square()
    B = Exp()
    C = Square()
    return A(B(C(x)))

def square(x):
    return Square()(x)


def exp(x):
    return Exp()(x)


x = Variable(np.array(0.5))
x = Variable(1)
y = square(exp(square(x)))
print(x.grad)
y.backward()
print(x.grad)

Comments

Post a Comment