Deep Learning Step6 Computational Graphs

Deep Learning Step6

  Step 6 Computational Graphs

About Computational Graphs

 There are many computational graphs in AI. They make it easy for us to understand visually how AI model is made. Computational graphs below are shown in this step. 

・Additional Nodes

・Multiplicated Nodes

・Separated Nodes

・Repeated Nodes

・Sum Nodes

・MatMul Nodes

About  additional Nodes

 Additional nodes add up 2 nodes like X and Y in Fig.6.1. Where L means Loss function. 

Fig.6.1 Forward Propagation of additional nodes

 On the other hand, in back propagation of it, additional nodes doesn't change the differentiation value from the upper value of the differentiation.

Fig.6.2 Back Propagation of additional nodes

About  Multiplicated Nodes

  Multiplicated nodes make multiplication of X and Y like in Fig.6.3.

Fig.6.3 Forward Propagation of multiplicated nodes

 The back propagation will be like in Fig.6.4. ∂Z/∂X is equal to Y if Z is X × Y, ∂Z/∂Y is equal to X as well.　 

Fig.6.4 Back Propagation of multiplicated nodes

About  Separated Nodes

 In Separated nodes, you just copy X as input to 2 Xs like in Fig.6.5. 

Fig.6.5 Forward Propagation of Separated nodes

  The back propagation will be like in Fig.6.6. First X is a function of second X in the middle layer. So differentiation of first X will be 2*∂L/∂X. 

Fig.6.6 Back Propagation of Separated nodes

About  Repeated Nodes

 In Repeated nodes, you just copy the input X into many Xs (Fig.6.7). 

Fig.6.7 Forward Propagation of Repeated nodes

 The differentiation of first X will be N *∂L/∂X, where N is any positive integer.

Fig.6.8 Back Propagation of Repeated nodes

About  Sum Nodes

 The Sum nodes add up input data. In Fig.6.9, input data are a ~ z. So these data will be a + b + … + z through the sum nodes. 

Fig.6.9 Back Propagation of Sum nodes

 In the back propagation, Sum nodes doesn't change the value of differentiation from upper side(Fig.6.10).  

 

Fig.6.10 Back Propagation of Sum nodes

About  MatMul Nodes

　In forward propagation of MatMul nodes, it calculates matrix product of 2 matrices like X and W in Fig.6.11. Where X is N×D matrix, W is D×H matrix and Y as output of X*W is N×H matrix. 

Fig.6.11 Forward Propagation of MatMul nodes

 The differentiation of ∂L/∂X will be ∂L/∂Y * WT (transposed matrix of W). The solution is from calculating the differentiation of composition function of X. ∂L/∂W will be XT(transposed matrix of X)*∂L/∂Y which is the same reason as ∂L/∂X. 

Fig.6.12 Back Propagation of MatMul nodes

Example of computational graph

 How will X * W + b be shown in a computational graph? This equation is often used in neural networks. In Fig.6.13, First X and W go through MatMul Node and they will be X * W. X * W and b go through Sum node as well, and we will get X * W + b.

Fig.6.13 Forward Propagation of computational nodes

  The back propagation begins with Y (= X * W + b) in Fig.6.14.  ∂L/∂(X * W) will be ∂L/∂Y because the sum node won't change the value of differentiation from upper side. So ∂L/∂X will be ∂L/∂Y*WT and ∂L/∂W will be XT*∂L/∂Y.

Fig.6.14 Back Propagation of computational nodes