Publications & Patents

Recent advances in large-scale pretrained visual transformers have achieved remarkable success in delivering high-quality 3D reconstruction results. However, for a stream of inputs (e.g., video), they require heavy global recomputation for each new frame, which also involves quadratic memory costs to the sequence length in attention modules. To address the challenge, techniques such as caching previous keys and values have been explored. However, such approaches still incur quadratic memory costs, which hinder their deployment with long sequence inputs on memory-constrained commodity hardware. To address the challenge, we first make an observation that not all input frames are critical for processing new inputs by a quantitative analysis and utilize that observation to develop a new compute- and memory-efficient vision transformer for 3D reconstruction, MBVGGT. It employs a memory bank that maintains information of important frames only, which significantly reduces memory requirements yet enables high-quality results. Our evaluations demonstrate that MBVGGT achieves comparable accuracy across depth estimation, pose estimation, and 3D reconstruction on long content, while running 6.3x faster than the SOTA model and getting rid of the OOM error.

Stereo depth estimation is a fundamental component in augmented reality (AR), which requires low latency for real-time processing. However, preprocessing such as rectification and non-ML computations such as cost volume require significant amount of latency exceeding that of an ML model itself, which hinders the real-time processing required by AR. Therefore, we develop alternative approaches to the rectification and cost volume that consider ML acceleration (GPU and NPUs) in recent hardware. For pre-processing, we eliminate it by introducing homography matrix prediction network with a rectification positional encoding (RPE), which delivers both low latency and robustness to unrectified images. For cost volume, we replace it with a group-pointwise convolution-based operator and approximation of cosine similarity based on layernorm and dot product. Based on our approaches, we develop MultiHeadDepth (replacing cost volume) and HomoDepth (MultiHeadDepth + removing pre-processing) models. MultiHeadDepth provides 11.8-30.3% improvements in accuracy and 22.9-25.2% reduction in latency compared to a state-of-the-art depth estimation model for AR glasses from industry. HomoDepth, which can directly process unrectified images, reduces the end-to-end latency by 44.5%. We also introduce a multi-task learning method to handle misaligned stereo inputs on HomoDepth, which reduces the AbsRel error by 10.0-24.3%. The overall results demonstrate the efficacy of our approaches, which not only reduce the inference latency but also improve the model performance.

A brain tissue segmentation method based on diagonal voxel local binary pattern texture operator, which performs supervoxel clustering segmentation, extracts diagonal voxel LBP texture features, and uses kNN for feature matching to improve 3D MRI data segmentation efficiency.

Accurate segmentation of brain tissues from magnetic resonance imaging (MRI) is of significant importance for clinical application and scientific research. Traditional strategies to handle the 2D images have the limitation of 3D data. In this paper, to overcome these issues, a tissue segmentation approach with supervoxel clustering and the novel 3D texture extraction method are proposed. At first, the simple linear iterative clustering in three-dimension is applied, to reduce the number of calculation objects. Then, a novel local binary pattern in three-dimension is proposed for better discriminate the supervoxels with different tissues. A clustering approach is also developed to classify supervoxels with features into different types of tissues. The labels of supervoxel are finally mapped back to original data to have the tissue type of voxels. The performance of the proposed method is evaluated on the commonly utilized Internet Brain Segmentation Repository 18 dataset. The experiment showed promising results with insufficient trainset.