We present PastVision, a proof-of-concept approach that explores combining thermal touch sensing and object detection to infer recent actions by a person which have not been directly observed by a system. Inferring such past actions has received little attention yet in the literature, but would be highly useful in scenarios in which sensing can fail (e.g., due to occlusions) and the cost of not recognizing an action is high. In particular, we focus on one such application, involving a robot which should monitor if an elderly person with dementia has taken medicine. For this application, we explore how to combine detection of touches and objects, as well as how heat traces vary based on materials and a person’s grip, and how robot motions and activity models can be leveraged. The observed results indicate promise for the proposed approach.
[1] Abdelrahman Y, Shirazi AS, Henze N, Schmidt A. 2015. Investigation of Material Properties for Thermal Imaging-Based Interaction. CHI 2015. http://dx.doi.org/10.1145/2702123.2702290
[2] Amft O, Troster G. 2006. Methods for Detection and Classification of Normal Swallowing from Muscle Activation and Sound. Pervasive Health Conference and Workshops.
[3] Bell AF, Walton K, Chevis JS, Davies K, Manson C, Wypych A, Yoxall A, Kirkby J, Alexander N. 2013. Accessing packaged food and beverages in hospital. Exploring experiences of patients and staff. Appetite 60, 231-238.
[4] Benko H, Wilson AD, Balakrishnan R. 2008. Sphere: Multi-Touch Interactions on a Spherical Display. Proceedings of the 21st annual ACM symposium on User interface software and technology (UIST 2008), 77-86.
[5] Bhowmik A, Repaka R, Mulaveesala R, Mishra S. 2015. Suitability of frequency modulated thermal wave imaging for skin cancer detection: A theoretical prediction. Journal of Thermal Biology 51(2015) 6582.
[6] Chelvama YK, Zamina N, Steeleb GS. 2014. A Preliminary Investigation of M3DITRACK3R: A Medicine Dispensing Mobile Robot for Senior Citizens. Procedia Computer Science 42, 240 246.
[7] Chen L, Hoey J, Nugent CD, Cook DJ, Yu Z. 2012. Sensor-Based Activity Recognition. Transactions on Systems, Man, and Cybernetics. Part C: Applications and Reviews, Vol. 42, No. 6.
[8] Cohen PR, Sutton C, Burns B. 2002. Learning Effects of Robot Actions using Temporal Associations. Proceedings of the 2nd International Conference on development and Learning. DOI: 10.1109/DEVLRN.2002.1011807
[9] Correa M, Hermosilla G, Verschae R, Ruiz-del-Solar J. 2012. Human Detection and Identification by Robots Using Thermal and Visual Information in Domestic Environments. J Intell Robot Syst, 66:223243. DOI 10.1007/s10846-011-9612-2
[10] Crick C, Scassellati B. 2008. Inferring Narrative and Intention from Playground Games. Proceedings of the 12th IEEE Conference on Development and Learning.
[11] Dragone M, Saunders J, Dautenhahn K. 2015. On the integration of adaptive and interactive robotic smart spaces. Paladyn, Journal of Behavioral Robotics 6(1):165179
[12] Fitzpatrick P. 2003. First Contact: an Active Vision Approach to Segmentation. Proceedings of 2003 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2003). DOI: 10.1109/IROS.2003.1249191
[13] Gold K, Scassellati B. 2007. A Bayesian Robot That Distinguishes Self from Other. 29th Annual Meeting of the Cognitive Science Society.
[14] Gray J, Breazeal C, Berlin M, Brooks A, Lieberman J. 2005. Action parsing and goal inference using self as simulator. RO-MAN 2005: 202-209
[15] Iwai D, Sato K. 2005. Heat sensation in image creation with thermal vision. In Proceedings of the 2005 ACM SIGCHI International Conference on Advances in computer entertainment technology (ACE ’05 ), 213216.
[16] Jung MM, Poel M, Poppe R, Heylen DKJ. 2017. Automatic recognition of touch gestures in the corpus of social touch. J Multimodal User Interfaces, 11:8196. DOI 10.1007/s12193-016-0232-9
[17] Koppula HS, Saxena A. 2013. Learning Spatio-Temporal Structure from RGB-D Videos for Human Activity Detection and Anticipation. Proceedings of the 30th International Conference on Machine Learning.
[18] Larson E, Cohn G, Gupta S, Ren X, Harrison B, Fox D, Patel SN. 2011. HeatWave: Thermal Imaging for Surface User Interaction. CHI 2011, Session: Touch 3: Sensing, 2011.
[19] Lee K, Ikeda T, Miyashita T, Ishiguro H, Hagita N. 2011. Separation of Tactile Information from Multiple Sources Based on Spatial ICA and Time Series Clustering. IEEE/SICE Int. Symposium on System Integration (SII), pp. 791-796
[20] Li K, Fu Y. 2014. Prediction of Human Activity by Discovering Temporal Sequence Patterns. IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 36, No. 8.
[21] Murthy JN, van Jaarsveld J, Fei J, Pavlidis I, Harrykissoon RI, Lucke JF, Faiz S, Castriotta RJ. 2009. Thermal Infrared Imaging: A Novel Method to Monitor Airflow During Polysomnography. SLEEP, Vol. 32, No. 11.
[22] Osterberg L, Blaschke T. 2005. Adherence to Medication. N Engl J Med, 353:487-97.
[23] Pollack M, Engberg S, Thrun S, Brown L, Colbry D, Orosz C, Peintner B, Ramakrishnan S, Dunbar-Jacob J, McCarthy C. 2002. Pearl: A Mobile Robotic Assistant for the Elderly, in AAAI Workshop on Automatino as Caregiver.
[24] Redmon J, Divvala S, Girshick R, Farhadi A. 2016. You only look once: Unified, real-time object detection. n:CVPR.
[25] Ring EFJ, Ammer K. 2012. Infrared thermal imaging in medicine. Physiol. Meas. 33(3), R33.
[26] Rudol P., Doherty P. 2008. Human Body Detection and Geolocalization for UAV Search and Rescue Missions Using Color and Thermal Imagery. IEEEAC Paper 1274.
[27] Shiga Y, Dengel A, Toyama T, Kise K, Utsumi Y. 2014. Daily activity recognition combining gaze motion and visual features. UbiComp Adjunct: 1103-1111
[28] Shiomi M, Zanlungo F, Hayashi K. et al. 2014. Towards a Socially Acceptable Collision Avoidance for a Mobile Robot Navigating Among Pedestrians Using a Pedestrian Model. Int J of Soc Robotics 6(3): 443-455. doi:10.1007/s12369-014-0238-y
[29] Shirazi AS, Abdelrahman Y, Henze N, Schneegass S, Khalilbeigiy M, Schmidt A. 2014. Exploiting Thermal Reflection for Interactive Systems. Session: Novel Mobile Displays and Devices, CHI 2014.
[30] Silvera-Tawil D, Rye D, Velonaki M. 2015. Artificial skin and tactile sensing for socially interactive robots: A review. Robotics and Autonomous Systems 63 230-43.
[31] Stork JA, Spinello L, Silva J, Arras KO. 2012. Audio-Based Human Activity Recognition Using Non-Markovian Ensemble Voting. IEEE RO-MAN: The 21st IEEE International Symposium on Robot and Human Interactive Communication.
[32] Stoytchev A. 2005. Behavior-Grounded Representation of Tool Affordances, ICRA.
[33] Takacs B, Hanak D. 2008. A prototype home robot with an ambient facial interface to improve drug compliance. Journal of Telemedicine and Telecare 2008; 14: 393395.
[34] Tiwari P, Warren J, Day K, MacDonald B, Jayawardena C, Kuo IH, Igic A, Datta C. 2011. Feasibility study of a robotic medication assistant for the elderly (2011). Conference: Proceedings of the Twelfth Australasian User Interface Conference - Volume 117.
[35] Wang L, Zhao X, Si Y, Cao L, Liu Y. 2017. Context-Associative Hierarchical Memory Model for Human Activity Recognition and Prediction. IEEE Transactions on Multimedia, Vol. 19, No. 3.
[36] Wang Z. 2006. A field study of the thermal comfort in residential buildings in Harbin. 2005. Building and Environment 41, 10341039. doi:10.1016/j.buildenv.2005.04.020
[37] Wong WK, Chew ZY, Lim HL, Loo CK, Lim WS. 2011. Omnidirectional Thermal Imaging Surveillance System Featuring Trespasser and Faint Detection. International Journal of Image Processing (IJIP), Volume 4, Issue 6. version 32.
