Granary is an important facility to ensure the safety of grain storage. The granary is a large closed space with dim lighting and poor air circulation. Operations such as fumigation and air conditioning increase personnel safety risks. The identification and analysis of abnormal behaviors of workers through security videos in the granary has become a key safe operations for workers as an important technical guarantee. This paper proposed a video recognition algorithm for abnormal behavior of workers in a granary based on a skeleton sequence multi-algorithm. First, the YOLOv3tiny model was used to quickly detect the human body, combined with Sort to track the motion trajectories of multiple targets, and the human skeleton coordinate sequence and weight information were extracted through the AlphaPose model. Then, based on the real spacial graph (RSG) composed of natural connection nodes of the human skeleton and virtual spacial graph (VSG) constructed by interconnecting the center of gravity of the virtual human body with the head, hands, and feet, the bin was extracted based on the balance of the interaction between the center of gravity of the human body dynamics and the hands and feet. Spatial characteristics of abnormal behavior of internal workers and spatiotemporal characteristics of concatenated temporal convolution (TC). Finally, a virtual-real combining spatial temporal graph convolution network (VR-STGCN) video recognition algorithm for abnormal behavior of granary workers was proposed. At the same time, a hybrid dataset was built, and comparative experiments and analysis were conducted between VR-STGCN and four algorithms such as SSD, PCANet, Two-StreamCNN, and STGCN. The results showed that all indicators of VR-STGCN were better than those of the other four algorithms. VR-STGCN can accurately identify abnormal behaviors such as falling, crawling, and lying down of people in the granary in complex environments such as insufficient light, multiple targets, and long distances. The recognition accuracy reached 97.7%, and the processing speed was 18.67fps, which can analyze the abnormal behavior of workers in real time. The research results could provide a new and efficient technology for the safety of granary workers in complex environments.