Pose2Room: Understanding 3D Scenes from Human Activities

(ECCV'2022)
Yinyu Nie1, Angela Dai1, Xiaoguang Han2, Matthias Nießner1
1Technical University of Munich, 2SRIBD, CUHKSZ
Paper arXiv Video Code Data

From an observed pose trajectory of a person performing daily activities in an indoor scene, we learn to estimate likely object configurations of the scene underlying these interactions, as set of object class labels and oriented 3D bounding boxes. By sampling from our probabilistic decoder, we synthesize multiple plausible object arrangements.

Abstract

With wearable IMU sensors, one can estimate human poses from wearable devices without requiring visual input. In this work, we pose the question: Can we reason about object structure in real-world environments solely from human trajectory information?

Crucially, we observe that human motion and interactions tend to give strong information about the objects in a scene -- for instance a person sitting indicates the likely presence of a chair or sofa. To this end, we propose P2R-Net to learn a probabilistic 3D model of the objects in a scene characterized by their class categories and oriented 3D bounding boxes, based on an input observed human trajectory in the environment.

P2R-Net models the probability distribution of object class as well as a deep Gaussian mixture model for object boxes, enabling sampling of multiple, diverse, likely modes of object configurations from an observed human trajectory. In our experiments we demonstrate that P2R-Net can effectively learn multi-modal distributions of likely objects for human motions, and produce a variety of plausible object structures of the environment, even without any visual information.

Video

Method

method

Overview of P2R-Net.

An illustration of our approach is shown in above Figure. Given a pose trajectory with N frames and J joints, a position encoder decouples each skeleton frame into a relative position encoding (from its root joint as the hip centroid) and a position-agnostic pose. After combining them, a pose encoder learns local pose features from both body joints per skeleton (spatial encoding) and their changes in consecutive frames (temporal encoding). Root joints as seeds are then used to vote for the center of a nearby object that each pose is potentially interacting with. A probabilistic mixture network learns likely object box distributions, from which object class labels and oriented 3D boxes can be sampled.

BibTeX


      @article{nie2021pose2room,
        title={Pose2Room: Understanding 3D Scenes from Human Activities},
        author={Yinyu Nie and Angela Dai and Xiaoguang Han and Matthias Nie{\ss}ner},
        journal={arXiv preprint arXiv:2112.03030},
        year={2021}
      }