Before reading this article, I recommend to have a look at my previous article on TensorFlowKit and like my repository.

When I started working in the field of machine learning, it was quite difficult to move to vectors and spaces from objects and their behavior. At first it was rather complicated to wrap my head around all that, and most processes did not seem obvious and clear at once. That’s the reason why I did my best to visualize everything I did in my groundwork:  I used to create 3D models, graphs, diagrams, figures, etc.

When speaking about efficient development of machine learning systems, usually such problems as learning speed control, learning process analysis, gathering various learning metrics, and others are mentioned. The major difficulty is that we (people) use 2D and 3D spaces to describe various processes that take place around us. However, processes within neural networks lay in multidimensional spaces, and that makes them rather difficult to understand. Engineers all around the world understand this problem and try to develop various approaches to the visualization or conversion of multidimensional data into simpler and more understandable forms.

There are separate communities dedicated to solving such problems, for example, Distill, Welch Labs, 3Blue1Brown.

TensorBoard

        Before I started working with TensorFlow, I used the TensorBoard package. It turned out to be a handy cross-platform solution for visualizing different kinds of data. I spent a couple of days “teaching” the Swift application to create reports in the TensorBoard format and integrate them into my neural network.

        Development of TensorBoard started in the middle of 2015 in one of the Google laboratories. In the end of 2015 Google opened the source code and the project became an open source one.

        The current version of TensorBoard is a Python package created using TensorFlow, and it allows visualization of the following kinds of data:

  •     Scalar data in time stack with the smoothing option
  •     Images in case you can represent your data in 2D, for example, convolutional network weights (filters)
  •     Actual computational graph (as an interactive view)
  •     2D modifications of tensor values over time
  •     3D histogram-modification of data allocation within tensor over time
  •     Text
  •     Audio

Besides, there is a projector and a possibility to extend TensorBoard using plugins, but that is a topic for another article.

You need to install TensorBoard on your computer (Ubuntu or Mac) to get started.

Also, you need to install Python 3. I recommend to install TensorBoard as a part of the TensorFlow package for Python.

Now run TensorBoard after specifying a directory for storing reports:

Let’s open http://localhost:6006/.

TensorFlowKit

Example with GitHub. Remember to like my repository.

Now let’s get a view of some cases using an example. Reports (summaries) in the TensorBoard format are created when the computational graph is being built. In TensorFlowKit, I did my best to duplicate Python approaches and interface, so that it would be possible to use shared documentation in the future. As I mentioned earlier, each report is added into a summary. It’s a container holding a value array, each of which representing an event we need to visualize. Later on, summary is saved to a file in the file system, where TensorBoard is able to read it.

 

So, we need to create FileWriter after specifying the graph we are going to visualize, and create a summary that will hold our values.

After running the application and refreshing the page we will see the graph we’ve built in the code. It will be interactive, so we can navigate it.   

Also, we want to see changes of some scalar value over time, for example, the value of the loss function and the accuracy of our neural network. To do that, let’s add output of the operations to the summary:

So, after each step of calculations of our session, TensorFlow automatically subtracts values of our operations and passes them to the input of the resulting summary that will be saved in FileWriter (I will tell you how to do that a bit later).

There is a lot of weights and biases in our neural network. Usually these are various high dimension matrices, and it is quite difficult to analyze their values by displaying them (printing out). It’s better to create a distribution diagram. Also, let’s add information on the weights change value that is made by our network during the learning process into our Summary.

Now we have a visualization of weights changes and the changes during the learning process.  

However, that is not all. Let’s take a look at the organization of our neural network. Each figure with handwriting received on input finds some reflection in the corresponding weights. That means the input figure can activate certain neurons, and this way leaves some mark in our network. Let me remind you that we have 784 weights for each neuron out of 10. So, we’ve got 7840 weights. All of them are represented as a 784×10 matrix. Let’s try to evolve the matrix into a vector and after that extract the weights that correspond to each class.

To do that, let’s add a couple of operations to our graph: stridedSlice and reshape.

Now let’s add each vector we get into the Summary as an image.

In the Images section of TensorBoard, we can see the weights’ “marks” as they used to be during the learning process.

Now let’s process our Summary. To do that, we need to join all created Summaries into one, and process it while exercising the network.

While the network works:

Please keep in mind that I did not consider the problem of the accuracy calculation, as it is calculated basing on the learning data. It is not correct to calculate it basing on the data for exercising.

In the next article I will explain how to build one neural network and launch it on Ubuntu, MacOS, iOS from one source.  

 

Author: Volodymyr Pavliukevych

Senior Software Engineer, Data Scientist.

Senior Software Engineer, Data Scientist.

No Comments so far.

Leave a Reply