Jan 17: Detection theory, ROC and AUC

Today we released the competition. This year the task is to predict the surface ty on which a mobile robot is rolling on its wheels. This should be done using inertial measurement unit data (accelerometers etc.). The competition is open for everyone, and there is already a baseline, that all teams should exceed. The exact requirements (e.g., the form of the report, where to return etc) to pass the course will be defined later.

Next, looked at an opencv human detection example code. This links nicely the detection theory with machine learning. The code uses Histogram of Oriented Gradients (HOG) representation of the image and classifies each image location as "person" or "not person" using a support vector machine. The code is available at our github, and the essential lines are below.

    # Initialize HOG detector.
    
    hog = cv2.HOGDescriptor()
    detectorCoefficients = cv2.HOGDescriptor_getDefaultPeopleDetector()
    hog.setSVMDetector( detectorCoefficients )

    # Load test image
    
    filename = 'person1.jpg'    
    img = cv2.imread(filename)
    
    # Detect humans
    
    found, w = hog.detectMultiScale(img, 
                                    winStride=(8,8), 
                                    padding=(32,32), 
                                    scale=1.05,
                                    hitThreshold = -1)

    draw_detections(img, found)

During the example, a phone was circulating in the audience demonstrating in-device detection. If you dare, you may test an image detection/recognition demo using deep learning from Google's Tensorflow project. It is available for Android phones (just search Play store for "tensorflow object detection"). Below is one detection example from last weekend.

We also saw that the sensitivity of detector can be tuned to balance between different kinds of errors: missed detections (we did not find all persons in the picture) and false alarms (we found a ghost in the picture). More formal treatise of error types can be found from Wikipedia.

After the example, we studied the model-based approach for detecting a sinusoid embedded under the Gaussian noise. Since there is a model of the signal, the detection can be formulated mathematically, and detection scores are obtained accurately using integration. The same applies also for discrete case (e.g. human detection), but counting is used in place of integrals.

At the end of the lecture, we studied receiver operating characteristic curve and the derived AUC accuracy metric. We also showed how to manually solve AUC tasks such as Question 5a in this exam.

[Edit] In some exercise groups, the solution of task 2 was a bit unclear. Therefore, the "official" solution is shown below.