Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs Roozbeh

Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs Roozbeh

Analyzing Semantic Segmentation Using Hybrid Human-Machine CRFs
Roozbeh Mottaghi1, Sanja Fidler2, Jian Yao2, Raquel Urtasun2, Devi Parikh3
1
2
3
UCLA
TTI Chicago
Virginia Tech

Remove

holistic scene understanding models for
semantic segmentation.
We compute CRF potentials based on:
- Responses of human subjects
- Ground Truth
Segmentation
- Machine
- Removing components

Building
Grass
Tree
Cow
Sheep
Sky
Aeroplane

Context
boat

sky
water

Book

Flower
Sign

Chair
Road
Cat
Dog
Body

Bird

Boat

90.4%

Scene recognition

Machine Accuracy: 77.4

CRF

Object detection

Object shape

Holistic CRF Model
Which scene is depicted?

Plug-and-Play architecture

Which classes are present?

No restrictions on the form
of the potentials

True/False detection?

>

Class Unary: Likelihood of presence of a certain class
Machine: Frequency in training data
Humans: Given a pair of categories, we asked Which category is
more likely?
Class-class Co-occurrence:
Machine: Co-occurrence matrix from training data
Humans: We asked Which scenario is more likely to occur in an
image? Observing (cow and grass) or (cow and airplane)?
grass

sky

aeroplane

tree

water

cat

face

cow sheep

book

flower

dog

bird

chair

road

bicycle

body

sign

car

face

book

tree building

body

sky

cow sheep

flower

bird

car

bicycle

chair

dog

cat

boat
boat

Machine

Human

Super-segment labeling
J. Yao, S. Fidler and R. Urtasun,
Describing the Scene as a Whole: Joint
Object Detection, Scene Classification and
Semantic Segmentation, CVPR 2012.

Human & Machine Potentials
We used Amazon Mechanical Turk to produce human potentials.

Object Detection:
Machine: DPM (Felzenszwalb et al.)
Humans: Ground truth which is provided by humans.

nsi

ste
n

t

We tried several hypotheses including incorporating scale, handling
overfitting, predicting human potentials from machine potentials,
etc., but none of them explained the boost.

H

H
S,

SS

M

M
S,

H SS

SS

H

M
S,

SS

M

H
S,

SS

S+

H

M
S,

M SS H SS
M
M

SS

+
SSM S

H

H
S,

SS

H SS

Human and machine errors become similar when we reduce the
window size for TextonBoost:

Shape Prior:
Machine: Per component average mask of examples. (78.2%)
Humans: We asked them to draw object contour along the
boundary of superpixels (80.2%)
Machine: 200x200 windows

10 subjects participated in our study for each task (total of 500
subjects).

car

Bird

Cow

face

Chair

In total, we had ~300K tasks.
We used MSRC dataset for our experiments.

inc
o

Humans and machines make 84complementary mistakes:
82
Breaking the connection
80
H: Human
78
M: Machine
between layers causes a
76
S: Segment
significant decrease in accuracy 74
SS: Supersegment
72
when we have human at one
70
MS
w/o
level and machine at another.
HS
HS
M S (M+H) S(M+H) S
M SS

road

sign
water

85.3%

86.8%

The CRF model performs better with the less accurate human segment
potential. Goal of subsequent analysis is to explain the boost.

grass

building

89.8%

Experiments with Human-Machine CRFs

Human Accuracy: 72.2

aeroplane

cow, sheep, aeroplane, face, car, bicycle,
flower, sign, bird, book, chair, cat, dog,
body, boat

Segment labeling

Water
Human Face
Car
Bicycle

bird

Findings from our human studies inspire

an approach that results in state-of-theart accuracy on MSRC.

Recognize this

Ground
Truth

interface used by
human subjects
Car, cmp. 1

Car, cmp. 3

A v e r a g e p e r -c la s s
r e c a ll

Machine

average per class recall

Human

Average Recall

Summary
Our goal is to identify bottlenecks in

Scene Potential:
Machine: Spatial Pyramid Match over SIFT, RGB, GIST, (81.8%)
Humans: We asked the human subjects to classify the images into
one of 21 scene classes.

Segment & Super-segment Potentials:
Machine: TextonBoost classifier
Humans:

Machine: 30x30 windows

Human

sheep

A simple change of using multiple
window sizes in the segment classifier
provides a significant improvement.
The type of mistakes are more
important than the number of mistakes.

2.4%
80
79
78
77
76

Window size

Recently Viewed Presentations

  • Legalized Marijuana & Workplace implications

    Legalized Marijuana & Workplace implications

    Colorado law repeatedly states that substances "present in the system" qualify for positive drug tests. While marijuana proponents decry this as unfair because it may not necessarily mean "impairment" at the time of the test - remember 3 important issues:...
  • The life and times of Coretta Scott King

    The life and times of Coretta Scott King

    Program Evaluation Within program mastery tests Formal reading tests Questionnaires Program fidelity problems: Departures from the program such as: omitting specific exercises or tasks failing to detect and correct errors failing to Repeat until firm providing fewer than 4-5 lessons...
  • Gravity anomaly due to a simple-shape buried body

    Gravity anomaly due to a simple-shape buried body

    Geoid anomaly and corrections The Bouguer anomaly is the geoid anomaly, with the free-air and Bouguer corrections applied: Geoid anomaly and corrections Terrain correction, gT: This correction accounts for the deviation of the surface from an infinite horizontal plane. The...
  • Rigidity Detection in Parkinson's Disease

    Rigidity Detection in Parkinson's Disease

    Quantitative Detection of Parkinson's Disease Symptoms Advisor: Dr. Chris Kao ... that carries information from neuron to neuron and eventually to the muscles Brain no longer able to direct or control muscle movement in a normal manner. Half a million...
  • Overview of behavioral health

    Overview of behavioral health

    Image from clipart-library.com. Easily angered, irritability. May withdraw from others, avoid school. ... Don't walk alone, avoid isolated places, find allies. Block the cyberbully through email, phone, social media. Report any threats or violence. Stick up for others who are...
  • Types of Rock - MR. FORD'S EARTH SCIENCE CLASS

    Types of Rock - MR. FORD'S EARTH SCIENCE CLASS

    It is composed primarily of calcium carbonate. Determine if the following rock samples are foliated or non-foliated: Amphibolite Quartzite Phyllite Foliated Be sure to complete your "Types of Rocks" notes as you view this presentation.
  • Social Crisis, War, and Revolution

    Social Crisis, War, and Revolution

    Social Crisis, War, and Revolution. Chapter 7, Section 2. Section 2 notes. ... The word defenestrate comes from Latin word for "window" and means "out the window" ... The treaties upheld the Peace of Augsburg, added Calvinism to the list...
  • Don't Be Mad at ME!

    Don't Be Mad at ME!

    Predictability Trust Wages and benefits Role as worker and provider Control over one's life Structure of daily life The work family Anger is an emotion that often propels us to move toward the source Perceived unfairness Inability to reach our...