The Laboratory for interpretable Visual Modeling, Computing and Learning (iVMCL) – iVMCL

Intelligent systems work by putting parts together. We study the three operations every such system performs: how it composes parts into wholes, selects among alternatives, and remembers what it has learned. Our tools are grammars of parts and alternatives (And-Or graphs) and learning methods that infer such structure from data. Our goal is coherent computing via grammar-typed modeling: models whose composition, selection and memory stay reliable as they scale and as their tasks grow longer.

Our long-term research goal is to pursue continually explainable and robust computing systems that can ALTER (Ask, Learn, Test, Explain and Refine) for AIGCGT (AI Generated Content and Ground-Truth).

The iVMCL laboratory in ECE at NC State is led by Dr. Tianfu (Matt) Wu, Associate Professor of Electrical and Computer Engineering, and is affiliated with the Visual Narrative Initiative.

Our research →

Overview slide of AOGNets. Left: an AOG building block with AND-nodes, OR-nodes and terminal nodes over the channels of an input feature map, and the node operations (splitting, concatenation, summation). Center: ImageNet top-1 accuracy against number of parameters for AOGNets, ResNets, ResNeXts, DenseNets and DPNs. Right: number of unique detectors per concept category from network dissection. Bottom left: activation heatmaps of ResNet and AOGNet models on example images.

Compositional and grammar-guided models: Can the structure of a model (which parts, which connections, which groupings) be learned from data instead of fixed by hand?

Overview slide of PaCa-ViT: patch-to-patch, patch-to-reduced-patch and patch-to-cluster attention compared, cluster heatmaps on example images, and three take-away points.

PaCa-ViT at a glance: image patches attend to a few learned clusters, whose heatmaps can be inspected directly.

Overview slide of HAWP. Top left: a line segment mapped into the local frame of a pixel by translation, rotation and scaling. Bottom left: an indoor image, its junction and line-segment proposals, and the parsed wireframe. Right: wireframes from AFM++, F-Clip, LETR and HAWP on building photos, and line detections from LSD, SOLD² and HAWPv3 on aerial images outside the training data.

Structured geometry: wireframes, planes and 3D: How can a network that predicts dense pixel maps reliably recover sparse geometric structure, such as line segments, wireframes, planes and 3D boxes, from images?

Overview slide of MvACon: detections on nuScenes by BEVFormer and by BEVFormer with MvACon, the multi-view 3D detection pipeline with MvACon between the 2D backbone and 2D-to-3D lifting, and three take-away points.

MvACon at a glance: cluster-based context added to 2D features before lifting improves query-based multi-view 3D detectors.

Overview slide of NEAT: 3D wireframes reconstructed on DTU and BlendedMVS scenes, the NEAT field and junction distillation framework, and three take-away points.

NEAT at a glance: the attraction field lifted into a 3D neural field reconstructs 3D wireframes without cross-view matching.

Overview slide of ScaleLSD: line segments detected by LSD, HAWPv3, DeepLSD and ScaleLSD on diverse images, the architecture and pseudo-label pipeline, and three take-away points.

ScaleLSD at a glance: self-supervised line detection trained on about 10 million unlabeled images.

LoGo-CAP (CVPR 2022): multi-person pose estimation in which the joints of each person refine one another’s locations with local and global context

Overview slide of Face3D: a ConvNet that estimates the 3D rotation and translation of a mean face model from facial key-points to propose face boxes, with key-point heatmaps and detections on crowded scenes.

Face3D (ECCV 2016): face detection that builds a 3D mean face model into the ConvNet, so face proposals come from estimated 3D pose and key-points

Overview slide of LostGAN. Top: a layout of labeled boxes (sky, snow, mountain, tree, person, skis) plus a style code is synthesized into an image. Bottom left: the layout-to-mask-to-image generator with ISLA-Norm. Right: snow scenes generated as object boxes are added one by one and as the person's box is moved, with three style codes per layout.

Controllable generation: How can a generator be told where each object goes and what each one looks like, so a user can move, relabel or restyle one object while the rest of the image stays fixed?

Overview slide of RisingAttacK. Left top: workflow of ordered top-K attacks, one constrained objective solved three ways (RisingAttacK in image space, QuadAttacK in feature space, AdvDistill with a surrogate loss). Left bottom: a redshank image attacked on ResNet-50 for top-K from 1 to 30, with the perturbation for each K. Right: three take-away points.

Adversarial attack and robustness: How much control can a tiny, carefully chosen change to an image give an attacker over a classifier, down to dictating its whole ranked list of top predictions, and what do such attacks reveal about what the model has learned?

Overview slide of ReCoG. Left top: the framework, a multimodal graph refined from text and image evidence while cross-modal message passing updates text and vision features, trained end to end. Left bottom: similarity distributions on RedditM showing that edges added by the refiner are far more similar than disconnected node pairs. Right: three take-away points.

Graph neural networks and self-supervised learning: How can a graph neural network learn useful node representations when the edges cannot be fully trusted, because connected nodes are often unlike each other or the observed edges are noisy and incomplete?

Overview slide of HarmonyGNNs: the teacher-student framework with joint structural node encoding, a bar chart of GCN, GAT and MLP accuracy on heterophilic and homophilic synthetic graphs, and three take-away points.

HarmonyGNNs at a glance: each node decides how much to trust its neighborhood, learned without labels.

Overview slide of CHEEM. Left: the method, a Vision Transformer whose blocks are reused, adapted, newly added or skipped for each task, with an external memory of task centroids that infers the task at test time. Right top: the models learned for four tasks on ViT-Base. Right bottom: three take-away points.

Continual and efficient adaptation: When a network must learn new tasks one after another, how should it decide which parts to reuse, which to adjust cheaply and which to add, without forgetting earlier tasks or wasting parameters?

Overview slide of WeGeFT: LoRA and WeGeFT compared as diagrams, accuracy against trainable parameters on commonsense, code and math tasks, and three take-away points.

WeGeFT at a glance: fine-tuning updates generated from the pretrained weights, with very few trainable parameters.

Research at a glance

Overview slide of AOGNets. Left: an AOG building block with AND-nodes, OR-nodes and terminal nodes over the channels of an input feature map, and the node operations (splitting, concatenation, summation). Center: ImageNet top-1 accuracy against number of parameters for AOGNets, ResNets, ResNeXts, DenseNets and DPNs. Right: number of unique detectors per concept category from network dissection. Bottom left: activation heatmaps of ResNet and AOGNet models on example images.

Compositional and grammar-guided models

Compose Can the structure of a model (which parts, which connections, which groupings) be learned from data instead of fixed by hand?

Overview slide of HAWP. Top left: a line segment mapped into the local frame of a pixel by translation, rotation and scaling. Bottom left: an indoor image, its junction and line-segment proposals, and the parsed wireframe. Right: wireframes from AFM++, F-Clip, LETR and HAWP on building photos, and line detections from LSD, SOLD² and HAWPv3 on aerial images outside the training data.

Structured geometry: wireframes, planes and 3D

Compose How can a network that predicts dense pixel maps reliably recover sparse geometric structure, such as line segments, wireframes, planes and 3D boxes, from images?

Overview slide of LostGAN. Top: a layout of labeled boxes (sky, snow, mountain, tree, person, skis) plus a style code is synthesized into an image. Bottom left: the layout-to-mask-to-image generator with ISLA-Norm. Right: snow scenes generated as object boxes are added one by one and as the person's box is moved, with three style codes per layout.

Controllable generation

Compose How can a generator be told where each object goes and what each one looks like, so a user can move, relabel or restyle one object while the rest of the image stays fixed?

Overview slide of RisingAttacK. Left top: workflow of ordered top-K attacks, one constrained objective solved three ways (RisingAttacK in image space, QuadAttacK in feature space, AdvDistill with a surrogate loss). Left bottom: a redshank image attacked on ResNet-50 for top-K from 1 to 30, with the perturbation for each K. Right: three take-away points.

Adversarial attack and robustness

Select How much control can a tiny, carefully chosen change to an image give an attacker over a classifier, down to dictating its whole ranked list of top predictions, and what do such attacks reveal about what the model has learned?

Overview slide of ReCoG. Left top: the framework, a multimodal graph refined from text and image evidence while cross-modal message passing updates text and vision features, trained end to end. Left bottom: similarity distributions on RedditM showing that edges added by the refiner are far more similar than disconnected node pairs. Right: three take-away points.

Graph neural networks and self-supervised learning

Select How can a graph neural network learn useful node representations when the edges cannot be fully trusted, because connected nodes are often unlike each other or the observed edges are noisy and incomplete?

Overview slide of CHEEM. Left: the method, a Vision Transformer whose blocks are reused, adapted, newly added or skipped for each task, with an external memory of task centroids that infers the task at test time. Right top: the models learned for four tasks on ViT-Base. Right bottom: three take-away points.

Continual and efficient adaptation

Remember When a network must learn new tasks one after another, how should it decide which parts to reuse, which to adjust cheaply and which to add, without forgetting earlier tasks or wasting parameters?

Meet the team

Portrait of Tianfu (Matt) Wu — Tianfu (Matt) Wu

Portrait of Chinmay Savadikar — Chinmay Savadikar

Portrait of Zhendong Liu — Zhendong Liu

Portrait of Gabriela Goldner — Gabriela Goldner

Portrait of Vidit Ghuliani — Vidit Ghuliani

Members and alumni →

Our research is supported by ARO, NSF, IARPA, the Laboratory for Analytic Sciences (LAS), Sony and Salesforce (details).