Jitendra Malik

Visual Recognition

The late 1990s marked a transition from geometric to learning techniques for visual recognition in computer vision. At this time, the techniques were from the statistical machine learning tradition – nearest neighbor, random forests and support vector machines. To apply these to visual data, handdesigned features were necessary and the Malik group pioneered features such as textons (vector quantized filter outputs)^[10] , shape contexts (relative arrangements of points) as well as novel machine learning techniques such as fast intersection kernels, SVM-KNN etc. This enabled them to achieve world record performance numbers for various tasks - handwritten digit recognition (2001), breaking CAPTCHAs (2002), Caltech101 categories (2005-07), people detection (2009). The shape context method [8] received the Helmholtz test-of-time award, and has more than 9000 citations.^[11] In 2012 a major paradigm shift occurred with the “AlexNet” work from Geoff Hinton’s group launching the deep learning revolution.

Malik played a catalytic role in this by encouraging Hinton to prove that deep learning worked better by competing on standard visual recognition benchmarks, specifically ImageNet. However, even after AlexNet, the broader computer vision community was still skeptical about the generality of the approach, since the ImageNet challenge was for classification and did not require object localization, for which at the time, PASCAL VOC was the accepted benchmark. Girshick et al invented the R-CNN method which proved that indeed this could be done, with a stage of pre-training on ImageNet classification followed by fine-tuning for object detection.^[12] This paper has more than 44000 citations for its CVPR 2014 version and received the Longuet-Higgins test-of-time award. A subsequent paper defined the problem of object instance segmentation and presented a model for its solution.^[13] R-CNN variants dominated object recognition for the better part of a decade until finally being superseded by VLMs.

Segmentation

Visual grouping and figure-ground discrimination were first studied by the Gestalt school of visual perception more than a century ago. The key insight, translated into modern terminology, is that we do not perceive an image as just a set of pixels, rather organize it into “regions” or “segments” corresponding to objects in the world. Operationalizing this computationally had been a central problem in computer vision from its early days. Malik’s group, over a period of two decades, 1995- 2015, made fundamental contributions which transformed the area from a miscellaneous collection of techniques to an empirically based science, and with the latest tools from deep learning, now largely a solved problem (cf. the Segment Anything system from Meta). Before Malik’s group’s work, contour detection and image segmentation research in computer vision was evaluated in an ad hoc manner. Authors showed their algorithms results without any measure of what is the right answer, unlike the case for mathematically well posed problems like structure from motion. Malik’s group changed the culture. They created the “Berkeley Segmentation Data Set” using multiple human observers to mark the boundaries they perceived in the image, and showing that the observers were consistent with a hierarchical model of segmentation. The creation of this dataset itself was noteworthy-using human observers to annotate images had previously been done for very niche categories like digits and faces, whereas here natural images were being segmented at scale. The BSDS paper has more than 10,000 citations and received the Helmholtz test-of-time award.^[14]

Armed with this dataset, Malik et al developed a rationalization for the various Gestalt cues as “optimal” if we assume that human vision evolved to be adaptive to the statistics of objects in the natural world. In computer vision, this enabled the development and comparison of different segmentation algorithms on a quantitative basis, rather than just by the display of results on a few cherry-picked examples. The idea of benchmarking on standard datasets itself became a norm, particularly in object recognition led by Perona (Caltech 101), Everingham et al (PASCAL VOC) and Fei-Fei Li (ImageNet). Finally, Malik’s group developed several algorithms and models for image and video segmentation.^[15][16][17] The most important of these was the normalized cut algorithm developed by Shi and Malik which used the eigenvectors of the Laplacian of the graph of pixel affinities to segment images. Normalized cuts is a general clustering algorithm widely used in settings other than images and has inspired much work on graph partitioning. The normalized cut paper has nearly 23,000 citations and received the Longuet-Higgins test-of-time award. ^[15]

3D Reconstruction

Malik’s group developed several different approaches which capitalize on this insight. The field of image based modeling and rendering was launched in the mid-1990s with a principal contribution being the FAC¸ ADE system for building 3D architectural models from images, with a key insight being that such models are composed of primitives such as cuboids, pyramids and surfaces of revolution.^[18] Tulsiani/Kar et al^[19] pioneered category-specific object reconstruction from a single image, and received the best student paper prize at CVPR 2015. To analyze human movement in video, Bregler and Malik ^[20]introduced the computer vision community to the representation of SE using the exponentials of twists, with the articulated human figure composed of a chain of such transformations.^[21]

This paper won the Longuet-Higgins test-of-time award. To analyze 3D human shape and pose from a single image, Kanazawa, Malik and their collaborators developed HMR (Human Mesh Recovery) over a series of papers.^[22]

Image Acquisition and Processing

One of Malik’s earliest and best-known contributions is the Perona-Malik anisotropic diffusion equation, a non-linear variant of the heat equation, which enables images to be smoothed while preserving edges.^[2] It created a new area of image processing, and is widely used for denoising medical and biological images. It was also fascinating from a mathematical point of view as it creates singularities and is ill-posed without some regularization. A fundamental contribution to computational photography was made by Debevec & Malik which provided an algorithm for capturing high dynamic range images using multiple exposures.^[21]

Datasets and Simulators

Malik played a pioneering role with the creation of BSDS, one of the earliest such datasets. More recently, together with Kristen Grauman, Malik co-led Ego4D, a consortium of 14 universities in 9 countries. The project collected egocentric video (more than 3600 hours) from nearly 1000 people going about their daily activities with headmounted cameras. The Ego4D project has had a major scientific impact in fields such as egocentric activity understanding, wearable agents and robotics. ^[23] Malik also helped create the widely used Habitat simulation environment in his role at Meta. ^[24]

Robotics

His work challenged the traditional robotics pipeline, which separates navigation into geometric map construction through simultaneous localization and mapping (SLAM) followed by path planning, by demonstrating that navigation can instead be learned end-to-end from visual input. This approach, first introduced in a 2018 study, showed that effective navigation does not require complete geometric maps alone but also benefits from semantic understanding of the environment, such as recognizing object categories.^[25] Building on this work, Malik and collaborators developed systems capable of autonomous exploration in previously unseen indoor environments, culminating in the "Airbnb test," in which a robot was placed in a randomly selected apartment and successfully navigated to a sequence of user-specified objects, including sinks, beds, televisions, and sofas, by constructing partial maps while exploiting learned prior knowledge.^[26]

Finally, Kumar et al represents a major advance in “in-the-wild” legged locomotion achieved by training the robot to walk, using reinforcements learning in simulation, with immediate transfer to the physical robot in the real world using a new technique called “Rapid Motor Adaptation”.^[27][28]