Convolutional Deep Belief Networks for Scalable Unsupervised ...

Convolutional Deep Belief Networks for Scalable Unsupervised ...

Convolutional Deep Belief Networks for Scalable Unsupervised Learning of Hierarchical Representations Honglak Lee, Roger Grosse, Rajesh Ranganath, and Andrew Y. Ng ICML 2009 Presented by: Mingyuan Zhou Duke University, ECE

September 18, 2009 Outline

Motivations Contributions Backgrounds Algorithms Experiment results

Deep Vs Shallow Conclusions Motivations To Learn hierarchical models which simultaneously represent multiple levels, e.g., pixel intensities, edges, object parts, objects, and beyond can be represented by layers from low to high.

Combining top-down and bottom-up processing of an image. Limitations of deep belief networks (DBNs) Scaling DBNs to realistic-size images remains challenging: images are high-dimentional and objects can appear at arbitrary locations in images. Contributions

Convolutional RBM: feature detectors are shared among all locations in an image. Probabilistic max-pooling: in a probabilistic sound way allowing higher-layer units to cover larger areas of the input. The first translation invariant hierarchical generative model supporting both top-down and bottom-up probabilistic inference and sales to

realistic image sizes. Backgrounds: Restricted Boltzmann Machine (RBM) (binary v) (real-value v) Giving the visible layer, the hidden units are conditionally

independent, and vise versa. Efficient block Gibbs sampling can be performed by alternately sampling each layers units. Computing the exact gradient of the log-likelihood is intractable, so the contrastive divergence approximation is commonly used. Backgrounds: Deep belief network (DBN)

In a DBN, two adjacent layers have a full set of connections between them, but no two units in the same layer are connected. A DBN can be formed by stacking RBMs. An efficient algorithm for training DBNs (Hinton et al., 2006): greedily training each layer, from lowest to highest, as an RBM using the previous layer's activations as inputs. Algorithms: Convolutional RBM (CRBM)

Algorithms: Probabilistic max-pooling Algorithms: Probabilistic max-pooling Each unit in a pooling layer computes the maximum activation of the units in a small region of the detection layer. Shrinking the representation with max-pooling allows higher-layer representations to be invariant to small translations of the input and reduces the computational

burden. Max-pooling was intended only for feed-forward architectures. A generative model of images which supports both top-down and bottom-up inference is of interest. Algorithms: Sparsity regulations Only a tiny fraction of the units should be active in relation to a given stimulus. Regularizing the objective function to encourage each of

the hidden units to have a mean activation close to some small constant . Algorithms: Convolutional DBN (CDBN) CDBN consists of several max-pooling-CRBMs stacked on top of one another. Once a given layer is trained, its weights are frozen, and its activations are used as input to the next layer.

Hierarchical probabilistic inference Experimental Results: natural images Experimental Results: image classification Experimental Results: unsupervised learning of

object parts Experimental Results: Hierarchical probabilistic inference Deep Vs Shallow .

From Jason Westons slides: DEEP LEARNING VIA SEMI-SUPERVISED EMBEDDING, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES From Francis Bachs slides:

Convex sparse methods for feature hierarchies, ICML 2009 WORKSHOP ON LEARNING FEATURE HIERARCHIES Conclusions Convolutional deep belief network: A scalable generative model for learning hierarchical representations from unlabeled

images. Performing well in a variety of visual recognition tasks.

Recently Viewed Presentations

  • Wisconsin Child Care Summit - Wisconsin Department of Public ...

    Wisconsin Child Care Summit - Wisconsin Department of Public ...

    One important component for success will be to fully utilize the power of CACFP to support quality in subsidized child care. As the Office of Child Care recognizes in the rule, " Another key partner in ensuring health, safety and...
  • FRP vs. FIRE

    FRP vs. FIRE

    Therefore, for Pcr = 1261 kN; f = 133.9 MPa. The calculated value of f is within the elastic range for a 344 MPa yield stress material. However, if the unsupported length was only 3 m, Pcr =would be calculated...
  • Programming Languages & Software Engineering

    Programming Languages & Software Engineering

    Have so far relied on an informal notion of, "does a module type-check given a signature?" As usual, there are precise rules… structure Foo :> BAR. is allowed if: Every non-abstract type in . BAR. is provided in . Foo,...
  • Question Bank 1- English IV  Which intellectual and

    Question Bank 1- English IV Which intellectual and

    Wordsworth believed that poetry should express… The spontaneous overflow of power emotions. An objective truth about observed reality. Concepts of superficial beauty, not necessarily with deeper meaning. The collective "will" of the people.
  • The Impact of Inter-Center Competition on Liver Transplant

    The Impact of Inter-Center Competition on Liver Transplant

    Fa ct ors a ffecting graft survi val af te r liver transplant ation fr om donation after cardiac death donors. Transplantation 2006; 82: 1683-8 . Organ Acceptance Decisions in Liver Transplantation: Alagosz Model . MELD. 10. 20. 30. 40....
  • Design for Manufacture (DFM)

    Design for Manufacture (DFM)

    The profiles are then traced by an optics system that is mounted to an X-Y stage. ... Selective Laser Sintering (SLS) A laser beam is then traced over the surface of this tightly compacted powder to selectively melt and bond...
  • Balancing Chemical Equations Why do you need to

    Balancing Chemical Equations Why do you need to

    1) Check the equation by completing an atom inventory. 2) Begin to balance one atom at a time by changing/adding . coefficients, so that amounts of each element are the same on both sides of equation * Do not change...
  • PHYS 1443 - Section 501 Lecture #1

    PHYS 1443 - Section 501 Lecture #1

    Since the total mechanical energy is conserved In order for the object to escape Earth's gravitational field completely, the initial speed needs to be RE m h ME vi vf=0 at h=rmax This is called the escape speed. This formula...