A Micro-Expression Apex Frame Spotting Method Based on Optical-Flow-Dual-Input Network
-
摘要: 微表情顶点帧蕴含着丰富的微表情信息，为了准确地检测出微表情顶点帧，本文提出了一种基于光流特征的神经网络分类，并利用先验知识规则进行取舍的检测方法. 该方法针对固定滑窗大小内的图像进行光流信息提取，利用双输入特征提取网络对x, y方向的光流信息进行时空特征提取，并进行分类，经根据微表情先验知识所设计的取舍规则后处理后，改善了检测准确度. 实验结果表明，在数据集CASMEⅡ上测试，顶点定位率（apex spotting rate，ASR）指标达到了0.945，F1-score指标达到了0.925.Abstract: Micro-expression apex frame contains abundant micro-expression information. In order to spot the apex frame accurately, a neural network classification was proposed based on optical flow characteristics. Taking prior knowledge as rules, a detection method was designed to realize micro-expression apex frame spotting. Firstly, optical flow information was extracted from the image in a fixed size sliding window. And then, the spatial and temporal features of optical flow information in x and y directions was extracted and classified based on dual input network. Finally, according to the trade-off rules based on prior knowledge of micro expression, a post-processing was carried out to improve the detection accuracy. The experimental results on data set CASMEⅡtesting show that the apex spotting rate (ASR) and F1-score can reach up to 0.945 and 0.925 respectively.
-
表 1 网络结构检测对比结果
Table 1. Evaluation of network structure
方法 检测结果数量 TP FP FN 单输入结构 100.32 51.97 48.32 0.12 双输入结构 80.82 41.35 40.47 0.05 
下载: 导出CSV
表 2 后处理方法检测对比结果
Table 2. Evaluation of post-processing
方法 检测结果数量 TP FP FN 后处理模块 80.82 41.350 40.470 0.05 有后处理模块 0.95 0.875 0.074 0.05
下载: 导出CSV
-
[1] 解仑, 卢亚楠, 姜波, 等. 基于人脸运动单元及表情关系模型的自动表情识别[J]. 北京理工大学学报, 2016, 36(2):163 − 169.XIE Lun, LU Yanan, JIANG Bo, et al. Expression automatic recognition based on facial action units and expression relationship model[J]. Transactions of Beijing Institute of Technology, 2016, 36(2):163 − 169. (in Chinese) [2] 岳雷, 沈庭芝. 基于自动提取特征点的三维人脸表情识别[J]. 北京理工大学学报, 2016, 36(5):508 − 513.YUE Lei, SHEN Tingzhi. 3D expression recognition based on automatically detected facial points[J]. Transactions of Beijing Institute of Technology, 2016, 36(5):508 − 513. (in Chinese) [3] LE NGO A C, PHAN R C W, SEE J. Spontaneous subtle expression recognition: imbalanced databases and solutions[C]//Proceedings of Asian Conference on Computer Vision. Cham: Springer, 2014: 33-48. [4] LI X, HONG X, MOILANEN A, et al. Towards reading hidden emotions: a comparative study of spontaneous micro-expression spotting and recognition methods[J]. IEEE Transactions on Affective Computing, 2017, 9(4):563 − 577. [5] DAVISON A, MERGHANI W, LANSLEY C, et al. Objective micro-facial movement detection using facs-based regions and baseline evaluation[C]//Proceedings of 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018). [S. l. ]: IEEE, 2018: 642 − 649. [6] DING J, TIAN Z, LYU X, et al. Real-time micro-expression detection in unlabeled long videos using optical flow and lstm neural network[C]//Proceedings of International Conference on Computer Analysis of Images and Patterns. [S. l. ]: Springer, 2019: 622 − 634. [7] TAKALKAR M A, THUSEETHAN S, RAJASEGARAR S, et al. LGAttNet: automatic micro-expression detection using dual-stream local and global attentions[J]. Knowledge-Based Systems, 2021, 212:106566. doi: 10.1016/j.knosys.2020.106566 [8] ESMAEILI V, FEGHHI M M, SHAHDI S O. Automatic micro-expression apex frame spotting using local binary pattern from six intersection planes[J/OL]. [2021-04-05]. https://arxiv.org/abs/2104.02149. [9] TRAN T K, VO Q N, HONG X, et al. Dense prediction for micro-expression spotting based on deep sequence model[J]. Electronic Imaging, 2019, 2019(8):401−1 − 401−6. doi: 10.2352/ISSN.2470-1173.2019.8.IMAWM-401 [10] ZHANG Z, CHEN T, MENG H, et al. SMEConvNet: a convolutional neural network for spotting spontaneous facial micro-expression from long videos[J]. IEEE Access, 2018, 6:71143 − 71151. doi: 10.1109/ACCESS.2018.2879485 [11] DONAHUE J, HENDRICKS A L, GUADARRAMA S, et al. Long-term recurrent convolutional networks for visual recognition and description[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. [S. l. ]: IEEE, 2015: 2625 − 2634. [12] SUN B, CAO S, HE J, et al. Two-stream attention-aware network for spontaneous micro-expression movement spotting[C]//Proceedings of 2019 IEEE 10th International Conference on Software Engineering and Service Science (ICSESS). [S. l. ]: IEEE, 2019: 702 − 705. [13] ZACH C, POCK T, BISCHOF H. A duality based approach for realtime tv-l 1 optical flow[C]//Proceedings of Joint Pattern Recognition Symposium. Berlin, Heidelberg: Springer, 2007: 214 − 223. [14] XIA B, WANG W, WANG S, et al. Learning from macro-expression: a micro-expression recognition framework[C]//Proceedings of the 28th ACM International Conference on Multimedia. [S. l. ]: ACM, 2020: 2936 − 2944. [15] YAN W J, LI X, WANG S J, et al. CASMEⅡ: an improved spontaneous micro-expression database and the baseline evaluation[J]. PloS one, 2014, 9(1):e86041. doi: 10.1371/journal.pone.0086041 [16] PAN H, XIE L, WANG Z, et al. Review of micro-expression spotting and recognition in video sequences[J]. Virtual Reality & Intelligent Hardware, 2021, 3(1):1 − 17. [17] OH Y H, SEE J, LE NGO A C, et al. A survey of automatic facial micro-expression analysis: databases, methods, and challenges[J]. Frontiers in Psychology, 2018, 9:1128. doi: 10.3389/fpsyg.2018.01128 [18] TRAN T K, VO Q N, HONG X, et al. Dense prediction for micro-expression spotting based on deep sequence model[J]. Electronic Imaging, 2019(8):401 − 406. [19] FU X, NIU L, HU Z, et al. Deep micro-expression spotting network training based on concept of transition frame[J]. Journal of ZheJiang University (Engineering Science), 2020, 54(11):2128 − 2137. -
下载:
点击查看大图
图(5) / 表(4)
计量
- 文章访问数: 91
- HTML全文浏览量: 93
- PDF下载量: 0
- 被引次数: 0
京公网安备 11010502036328号 京ICP备17033152号