Dept. Colloquiua

CS 495  Special Topics in Computer Science:  DEEP LEARNING,  Instructor:  Jeffrey Horn

COURSE ANNOUNCEMENTS (Sunday, March 20, 2017)

What's REALLY New:

• A6 is assigned.   See below.
• Start reading Chapter 5 of Michael Nielsen's online book, Neural Networks and Deep Learning, here.  the next assignment, A7, will probably be based on the exercises in this chapter.  (But do not worry about following all of the algorithmic details or mathematical models and derivations; we will be using these methods, not implementing them!)
  
• You should consider submitting a project to The Celebration of Student Scholarship.   At the very least, you should check it out by attending some sessions.   Usually there are refreshments!

Not so New (Older Announcements):

• A5 is updated with latest results, plus a spreadsheet for recording your data and calculating your four averages.   See below.
• A5 is assigned.   See below.
• A4 is assigned.   See below.
• A3 is assigned.   See below.
• A2 is assigned.   See below.
• Article assigned.   See below.
• A1 is assigned.   Due this week.   See below.
•  This is where old announcements will go, so that you won't miss any news!
• Welcome to class!

• Syllabus

• Lecture Notes

• Assignments

• Programming
• Tests and Quizzes

• Assignment 1:   The Perceptron

• Assignment 2:  Training the Perceptron
• DUE:   FRIDAY, February 3, 2017
  
• Here is the data file:  A2.pda
• First, run the PDA32 perceptron on the data file.   It should learn the concept, which means finding a line that separates the 10 positive examples from the ten negatives. 
  
• Second, try to find a smaller set of the twenty examples which, when used to train the PDA32 perceptron, results in a learned concept (line) that successfully classifies ALL TWENTY examples.   Note:  the learned concept does NOT have to be the same as the one learned on ALL TWENTY examples, but it MUST correctly classify all twenty.   Try to find as small a subset of the examples as possible.   The smaller the subset used, the higher your A2 grade, but I will be generous/lenient!   Do make sure that the smaller set you submit is indeed less than twenty!
• Email me the smaller data file (.pda).   We'll see who finds the smallest training set (that properly generalizes)!

• Assignment 3:  Backpropagation - 2 Hidden Units
  
• DUE: 
  
• Code:  BP with two hidden units
• Follow instructions (in-class) to download and install BOTH the TextPad (or TextWrangler) editor and the Java Developers' Kit (JDK) from the internet:
  • TextPad
  • Java
• TASK:  Simple and short.   I want you to become familiar with a small back-propagation network.   This one has only two-input nodes, two hidden layer nodes, and one output node.  And yet is capable of learning "very non-linear" concepts!   And not just XOR.   So please try running it on a training set that you choose by clicking in the box.   Try to get the network to learn a concept represented by a "very curvey" line.   And then send me a screenshot.  The only hard constraint I have on the learned concept is that it be 100% accurate on the training examples (i.e., no red circles in the green areas nor green in the red!).   Backpropagation is NOT guaranteed to converge to a concept that correctly classfifies its training set!
  
• SUBMISSION:   Email me the screenshot.   Be sure to put "A3" in the SUBJECT line somewhere (so I can filter my email for your submission).
• Assignment 4:  Backpropagation - 3 Hidden Units
  
• DUE: 
  
• Code:  BP with three hidden units
• TASK:  Simple and short.   Try your best to get the network with three hidden layer nodes to learn the SPHERE concept (which is a circle in two dimensions, and since we have only two (real-valued) inputs, we are in a two-dimensional instance space).   Imagine a real-world example where, e.g., the positive examples are all within a certain radius of some coordinate that represents a "target" (e.g., our new work locations) and positives represent other locations that are close to the target (e.g., apartments that are within walking distance of our new job location)).   Negative instances are the ones too far away!   Use as many examples as you like.   Unlike A3 your concept in A4 does NOT have to correctly classify all of the training examples.   I just want to see a green "disk" within a red field (or red disk within a green field!).
  
• SUBMISSION:   Email me the screenshot.   Be sure to put "A4" in the SUBJECT line somewhere (so I can filter my email for your submission).
• Assignment 5:  Backpropagation - k Hidden Units
  
• DUE:  Before Spring Break
  
• Code: (updated on Feb. 21, 2017)   BP with k hidden units
  
• Data:  (not needed for A5, just FYI for now...)  Concepts.zip
• Results so far:
• Here are the class results that I have received as of March 1, 2017:   Excel file, PDF file.
  
• Here is a spreadsheet with the results of MY runs (k=40 hidden layer nodes):   Excel file.  You might find it useful for recording the numbers from your own runs (that is, the ten different random seed runs).   Note that it is the numbers that I want you to send me, for the Class Results spreadsheet, are the averages from the bottom row for the FIRST FOUR data columns (that is, average accuracy on (1) positive training examples, (2) negative training examples, (3) positive testing examples, (4) negative testing examples.
• Assignment 6:  Analysis of Learned Concepts
  
• DUE:  End of next week (Friday, March 24, 2017)
• Use this worksheet: (handed out in class Tuesday, March 14, 2017)
• Instructions:
• As we have done in class, analyze the network that successfully learned the 40-bit PALINDROME concept.
• We are following up on the A5, where we showed that Back-prop could train a single-hidden-layer network to about 98% accuracy (over all examples, positive and negative, training set and testing set) WHEN the size of the hidden layer was approximately HALF the size of the input (i.e, approximately 20 neurons in the hidden layer for a 40-bit problem).  
  
• So train up your own network but first add this code to your file "BackpropA5.java": 
  
1. Add this method to the class "BackpropA5": 
   
• public void examineHiddenLayerNode(int j)
  {
      System.out.println("  OUTPUT LAYER NODE bias weight:  " + weightsOutputUnitsBias[0] + "\n\n");
      System.out.println("  HIDDEN LAYER NODE " + j + ":         OUTPUT weight:  " + weightsLayerTwo[j][0]);
      System.out.println("    INPUTS WEIGHTS:   ");
      System.out.println("           bias weight:  " + weightsHiddenUnitsBias[j]);
      System.out.println("           input node      weights:    ");
      for(int i=0; i< numInputUnits; i++)
      System.out.println("                  "+i+"   "+weightsLayerOne[i][j]);
  }
  
2. Call the above method at the end of the method "main" with this loop (as last line of "main"): 
   
• for(int j=0; j<numHiddenUnits; j++) bp.examineHiddenLayerNode(j);
  
• Or just substitute this new Backprop class file "BackpropA6.java" for the file "BackPropA5.java"
  
• Configure your network to have anywhere from 17 to 23 hidden layer nodes, then choose a random seed.   Record these two data on the worksheet (link above).
• Train it and make sure you get close to 98% accuracy (overall).   If you cannot get high accuracy, try different random seeds until you do.
  
• Then take a look at the output in the console window.   Find a hidden layer node that has a high weight (for its connection to the output node). 
  
• On the worksheet record the following (use the labeled blanks and create your own if the worksheet is missing any, for the following data):
1. Number of hidden layer nodes.
2. Random seed
3. (learned) Output layer node's bias weight
4. Index J of the hidden layer node that you have picked for the details below (e.g., one of 0-17 for a network with hidden layer of size 18)
5. (learned) Weight on hidden layer node j's output connection (which goes TO the output node).
6. (learned) The forty weights on hidden layer node j's input connections (coming from the input layer).
7. Write a short commentary on any patterns that you notice.   Feel free to speculate as to any ideas you might have as to any subconcepts (a.k.a., features) that might be evident, or any correlation at all to the PALINDROME concept.
• Feel free to use your own worksheet or spreadsheet, especially if you can up with a neater or dearer one than my hastily made worksheet!   But be sure to label ALL of your values.
• Assignment 7:  Deep Data
  
• DUE: 
  
• Links:
• CIFAR-10:  https://www.cs.toronto.edu/~kriz/cifar.html
• Downloading VLC player:  http://www.videolan.org/vlc/index.html
  
• Extracting images from video with VLC player:  https://www.isimonbrown.co.uk/vlc-export-frames/
• PNG to CIFAR-10:  https://github.com/gskielian/PNG-2-CIFAR10
• Tech. Report:  https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf
  
• Instructions:
• Pick your category (i.e., concept) (e.g., airplane, car, wildcatwillie, etc.).   Pick a short, one-word name for it (e.g., "waterbottle","mantix")
  
• Find or make video of category.
• To get an idea of what the images should look like, see the examples on the CIFAR-10 page.
• Also look at these guidelines, in Appendix C (pp. 52-53) of this report.
  
• Generate 6000 images as 32x32 pixel png files by doing this:
• Install VLC player.
• Follow the instructions here.
• If you choose to grab an image every frame (set "recording ratio" to "1") then you should be able to generate approximately 20 images per second of video, which is about 1000 images per minute!   (I'm not sure that is good though, since the images might be very similar to each other, perhaps "too" similar?)
  
• Separate the 6000 images into six folders of 1000 images each, named "Batch1", "Batch2", etc.  Put these six folders into another folder, named using your category name (e.g., "dog").   Send me this file, compressed (e.g., "dog.zip") as your A7 submission.
  

TESTS AND QUIZZES

Quizzes

• Quiz 1: