Work

Scene parsing, or semantic segmentation, consists in labeling each pixel in an image with the category of the object it belongs to. It is a challenging task that involves the simultaneous detection, segmentation and recognition of all the objects in the image.

Back in 2011, I had been working on a model (with Laurent Najman and Yann LeCun) that could parse a wide variety of scenes in close to real-time on a regular computer. I keep videos of this work here for posterity :) as the work was very early then.

In the following clips, recorded in random locations in the NYC area, we demonstrate the generalization capabilities of our model. This model was trained on the rather small Stanford Background dataset, which contains 715 images (520 for training, the rest for testing), labeled into 8 classes.

The next clip demonstrates generalization of our larger model, trained on the SiftFlow dataset, made of 2,688 images and 33 classes. The video was stitched from 4 GoPro cameras, and shot around the NYU campus. Our parse is far from perfect, but remember: our model doesn't perform any global inference, to maintain a high frame rate.

neuFlow is a custom dataflow computer that I designed to optimally compute dense, filter-based vision models. Dataflow processors like this one are well suited for high speed processing of video streams, can be extremely compact, and could give vision capabilities to any light-weight unmanned vehicle with low power constraints.

Back then I was very interested in the low-power, efficient computation of convolutional networks and the like. Yann LeCun et al. have demonstrated the efficiency of such architectures in pattern/object recognition of images [LeCun 1998, LeCun 2010, Kavukcuoglu 2010, Jarrett 2009].

neuFlow is based on a reconfigurable grid of simple processing units. Such a grid provides a way of dynamically reconfiguring the hardware to run diff. operations on streams of data, in parallel. The arch is inspired by the old ideas of dataflow computing [Adams 1968, Dennis 1974, Hicks Arvind 1993].

This figure shows an overview of the neuFlow architecture.

This figure shows our grid configured for a particular operation: a non-linear mapping of a sum of 2D convolutions, performed in one pass.

Our latest platform for neuFlow was sitting on an 8-layer PCB I routed... The platform embedded two cameras, 8GB of DDR3 memory, and one of the largest Xilinx FPGAs. It was able to run my scene parsing model at 10fps+ on HD streams.

One of the key questions of Vision Science (natural and artificial) is how to produce good internal representations of the visual world. What sort of representation would allow an artificial vision system to detect and classify objects into categories, independently of pose, scale, illumination, conformation, and clutter? More interestingly, how could an artificial vision system learn surch representations automatically, the way animals and human seem to learn by simply looking at the world?

In the time-honored approach to computer vision (and to pattern recognition in general), the question is avoided: internal representations are produced by a hand-crafted feature extractor, whose output is fed to a trainable classifier. While the issue of learning features has been a topic of interest for many years, considerable progress has been achieved in the last few years with the development of so-called deep learning methods.

Good internal representations are hierarchical. In vision, pixels are assembled into edglets, edglets into motifs, motifs into parts, parts into objects, and objects into scenes. This suggests that recognition architectures for vision (and for other modalities such as audio and natural language) should have multiple trainable stages stacked on top of each other, one for each level in the feature hierarchy. This raises two new questions: what to put in each stage? and how to train such deep, multi-stage architectures?

Convolutional Networks (ConvNets) are an answer to the first question. Until recently, the answer to the second question was to use gradient-based supervised learning, but recent research in deep learning has produced a number of unsupervised methods which greatly reduce the need for labeled samples.

A typical convolutional network for images: