Publications de Shaokai Ye | EPFL Graph Search

Neuroscience-inspired computer vision and language modeling for behavior analysis

Measuring animal behaviors is a crucial task in a range of scientific applications. Modern approaches with deep neural networks calls for developing solutions that solve a wide range of vision and language tasks. This ubiquitous need for understanding beha ...

EPFL2025

Keypoint-MoSeq: parsing behavior by linking point tracking to pose dynamics

Alexander Mathis, Mackenzie Mathis, Shaokai Ye

Keypoint tracking algorithms can flexibly quantify animal movement from videos obtained in a wide variety of settings. However, it remains unclear how to parse continuous keypoint data into discrete actions. This challenge is particularly acute because key ...

2024

SuperAnimal pretrained pose estimation models for behavioral analysis

Alexander Mathis, Mackenzie Mathis, Steffen Schneider, Shaokai Ye, Maxime Michel André Vidal, Jessy Lauer, Tian Qiu

Quantification of behavior is critical in diverse applications from neuroscience, veterinary medicine to animal conservation. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, re ...

Nature Research2024

AmadeusGPT: a natural language interface for interactive animal behavioral analysis

Alexander Mathis, Mackenzie Mathis, Shaokai Ye, Jessy Lauer

The process of quantifying and analyzing animal behavior involves translating the naturally occurring descriptive language of their actions into machine-readable code. Yet, codifying behavior analysis is often challenging without deep understanding of anim ...

2023

Multi-animal pose estimation, identification and tracking with DeepLabCut

Alexander Mathis, Mackenzie Mathis, Steffen Schneider, Shaokai Ye, Tanmay Nath, Jessy Lauer

Estimating the pose of multiple animals is a challenging computer vision problem: frequent interactions cause occlusions and complicate the association of detected keypoints to the correct individuals, as well as having highly similar looking animals that ...

NATURE PORTFOLIO2022

Non-Structured DNN Weight Pruning--Is It Beneficial in Any Platform?

Shaokai Ye

Large deep neural network (DNN) models pose the key challenge to energy efficiency due to the significantly higher energy consumption of off-chip DRAM accesses than arithmetic or SRAM operations. It motivates the intensive research on model compression wit ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Towards Robust Vision Transformer

Shaokai Ye, Yuan He

Recent advances on Vision Transformer (ViT) and its improved variants have shown that self-attention-based networks surpass traditional Convolutional Neural Networks (CNNs) in most vision tasks. However, existing ViTs focus on the standard accuracy and com ...

IEEE COMPUTER SOC2022

QAIR: Practical Query-efficient Black-Box Attacks for Image Retrieval

Shaokai Ye, Yuan He, Hang Su, Jinfeng Li

We study the query-based attack against image retrieval to evaluate its robustness against adversarial examples under the black-box setting, where the adversary only has query access to the top-1 ranked unlabeled images from the database. Compared with que ...

IEEE COMPUTER SOC2021

Adversarial Laser Beam: Effective Physical-World Attack to DNNs in a Blink

Shaokai Ye, Yuan He

Though it is well known that the performance of deep neural networks (DNNs) degrades under certain light conditions, there exists no study on the threats of light beams emitted from some physical source as adversarial attacker on DNNs in a real-world scena ...

IEEE COMPUTER SOC2021