Cover not available
Article published In:
Socially Acceptable Robot Behavior: Approaches for Learning, Adaptation and Evaluation
Edited by Oliver Roesler, Elahe Bagheri, Amir Aly, Silvia Rossi and Rachid Alami
[Interaction Studies 23:3] 2022
► pp. 427–468

Learning social navigation from demonstrations with conditional neural processes

| Bogazici University
| Bogazici University

Sociability is essential for modern robots to increase their acceptability in human environments. Traditional techniques use manually engineered utility functions inspired by observing pedestrian behaviors to achieve social navigation. However, social aspects of navigation are diverse, changing across different types of environments, societies, and population densities, making it unrealistic to use hand-crafted techniques in each domain. This paper presents a data-driven navigation architecture that uses state-of-the-art neural architectures, namely Conditional Neural Processes, to learn global and local controllers of the mobile robot from observations. Additionally, we leverage a state-of-the-art, deep prediction mechanism to detect situations not similar to the trained ones, where reactive controllers step in to ensure safe navigation. Our results demonstrate that the proposed framework can successfully carry out navigation tasks regarding social norms in the data. Further, we showed that our system produces fewer personal-zone violations, causing less discomfort.

Keywords: social navigation, path planning, conditional neural process, data-driven control, random network distillation, generative adversarial networks, hybrid navigation architecture
Article outline
  • Introduction
  • Related work
    • Hybrid path planning
      • Global path planning
      • Local path planning
    • Social navigation
  • Our method
    • I – data-driven global controller
      • Training the data-driven global controller
      • Querying the trained data-driven global controller
    • II – data-driven local controller
      • Training the data-driven local controller
      • Querying the trained data-driven local controller
    • III – failure prediction module
    • IV – hand-crafted reactive controller
  • Experiments and results
    • Analysis of the generated global trajectories
    • Analysis of the local controller: Evasive maneuvers
    • Comparison of local controllers
    • Performance of the complete system
    • Contribution of the Failure Prediction Module
    • Scalability of CNP
  • Conclusion
  • Acknowledgements
  • References
Published online: 21 April 2023
DOI logo
https://doi.org/10.1075/is.22018.yil
References
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., & Savarese, S.
(2016) Social lstm: Human trajectory prediction in crowded spaces. In 2016 ieee conference on computer vision and pattern recognition (cvpr) (p. 961–971). DOI logoGoogle Scholar
Arjovsky, M., Chintala, S., & Bottou, L.
(2017) Wasserstein gan.Google Scholar
Asghari Oskoei, M., Walters, M., & Dautenhahn, K.
(2010) An autonomous proxemic system for a mobile companion robot. In Proceedings of the aisb 2010 symposium on new frontiers for human robot interaction. Leicester, UK.Google Scholar
Biswas, A., Wang, A., Silvera, G., Steinfeld, A., & Admoni, H.
(2022) Socnavbench: A grounded simulation testing framework for evaluating social navigation. ACM Transactions on Human-Robot Interaction (THRI), 11(3), 1–24. DOI logoGoogle Scholar
Borenstein, J., Koren, Y., et al.
(1991) The vector field histogram-fast obstacle avoidance for mobile robots. IEEE transactions on robotics and automation, 7(3), 278–288. DOI logoGoogle Scholar
Brooks, R.
(1986) A robust layered control system for a mobile robot. IEEE journal on robotics and automation, 2 (1), 1423. DOI logoGoogle Scholar
Burda, Y., Edwards, H., Storkey, A., & Klimov, O.
(2018) Exploration by random network distillation. ar Xiv preprint arXiv:1810.12894.Google Scholar
Burgard, W., Cremers, A., Fox, D., Hähnel, D., Lakemeyer, G., Schulz, D., … Thrun, S.
(1999) Experiences with an interactive museum tour-guide robot. Artif. Intell., 114 1, 3–55. DOI logoGoogle Scholar
Cai, K., Wang, C., Cheng, J., De Silva, C. W., & Meng, M. Q.-H.
(2020) Mobile Robot Path Planning in Dynamic Environments: A Survey. arXiv preprint arXiv:2006.14195.Google Scholar
Che, Y., Okamura, A. M., & Sadigh, D.
(2020) Efficient and trustworthy social navigation via explicit and implicit robot-human communication. IEEE Transactions on Robotics, 36(3), 692–707. DOI logoGoogle Scholar
Chen, Y. F., Everett, M., Liu, M., & How, J. P.
(2017) Socially aware motion planning with deep reinforcement learning. CoRR, abs/1703.08862. Retrieved from [URL]. DOI logo
Dudek, G., & Jenkin, M.
(2010) Computational principles of mo-bile robotics. Cambridge university press. DOI logoGoogle Scholar
Farina, F., Fontanelli, D., Garulli, A., Giannitrapani, A., & Prat-tichizzo, D.
(2017) Walking ahead: The headed social force model. PloS one, 12(1), e0169734. DOI logoGoogle Scholar
Ferrer, G., Garrell, A., & Sanfeliu, A.
(2013) Robot companion: A social-force based approach with human awareness-navigation in crowded environments. In 2013 ieee/rsj international conference on intelligent robots and systems (pp. 1688–1694). DOI logoGoogle Scholar
Festo Robotics, R.
(2020) Robotino 4: For research and education. Retrieved from [URL]
Fong, T., Nourbakhsh, I., & Dautenhahn, K.
(2003) A survey of socially interactive robots. Robotics and autonomous systems, 42(3–4), 143–166. DOI logoGoogle Scholar
Fox, D., Burgard, W., & Thrun, S.
(1997) The dynamic window approach to collision avoidance. IEEE Robotics Automation Magazine, 4(1), 23–33. DOI logoGoogle Scholar
Garnelo, M., Rosenbaum, D., Maddison, C., Ramalho, T., Sax-ton, D., Shanahan, M., … Eslami, S. M. A.
(2018 10–15 Jul). Conditional Neural Processes. In J. Dy & A. Krause (Eds.), Proceedings of the 35th international conference on machine learning (Vol. 801, pp. 1704–1713). PMLR. Retrieved from [URL]
Giesbrecht, J.
(2004) Global path planning for unmanned ground vehicles (Tech. Rep.). Defence Research and Development Suffield (Alberta).Google Scholar
Glasius, R., Komoda, A., & Gielen, S. C.
(1995) Neural network dynamics for path planning and obstacle avoidance. Neural Networks, 8 (1), 125–133. DOI logoGoogle Scholar
Good fellow, I.
(2016) Nips 2016 tutorial: Generative adversarial networks. arXiv preprint arXiv:1701.00160.Google Scholar
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., … Bengio, Y.
(2014) Generative adversarial networks. arXiv preprint arXiv:1406.2661.Google Scholar
Gordon, J., Bruinsma, W. P., Foong, A. Y., Requeima, J., Dubois, Y., & Turner, R. E.
(2019) Convolutional conditional neural processes. arXiv preprint arXiv:1910.13556.Google Scholar
Gupta, A., Johnson, J., Fei-Fei, L., Savarese, S., & Alahi, A.
(2018) Social gan: Socially acceptable trajectories with generative adversarial networks. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 2255–2264). DOI logoGoogle Scholar
Hall, E.
(1966) The hidden dimension. New York, NY, US: Anchor Books.Google Scholar
Helbing, D., & Molnar, P.
(1995) Social force model for pedestrian dynamics. Physical review E, 51 (5), 4282. DOI logoGoogle Scholar
Holtz, J., & Biswas, J.
(2021) Socialgym: A framework for benchmarking social robot navigation. arXiv preprint arXiv:2109.11011.Google Scholar
Huang, K.-C., Li, J.-Y., & Fu, L.-C.
(2010) Human-oriented navigation for service providing in home environment. In Sice annual conference 2010, proceedings of (pp. 1892–1897). Taipei, Taiwan.Google Scholar
Kambhampati, S., & Davis, L.
(1986) Multiresolution path planning for mobile robots. IEEE Journal on Robotics and Automation, 2(3), 135–145. DOI logoGoogle Scholar
Karnan, H., Nair, A., Xiao, X., Warnell, G., Pirk, S., Toshev, A., … Stone, P.
(2022) Socially compliant navigation dataset (scand): A large-scale dataset of demonstrations for social navigation. arXiv preprint arXiv:2203.15041. DOI logoGoogle Scholar
Khatib, O.
(1985) Real-time obstacle avoidance for manipulators and mobile robots. In Proceedings. 1985 ieee international conference on robotics and automation (Vol. 21, p. 500–505). DOI logoGoogle Scholar
Kim, B., & Pineau, J.
(2016) Socially adaptive path planning in human environments using inverse reinforcement learning. International Journal of Social Robotics, 8 (1), 51–66. DOI logoGoogle Scholar
Kitani, K., Ziebart, B., Bagnell, J., & Hebert, M.
(2012) Activity forecasting. Computer Vision-ECCV 2012, 201–214. DOI logoGoogle Scholar
Koren, Y., & Borenstein, J.
(1991) Potential field methods and their inherent limitations for mobile robot navigation. In Proceedings. 1991 ieee international conference on robotics and automation (p. 1398–1404 vol.21). DOI logoGoogle Scholar
Kothari, P., Kreiss, S., & Alahi, A.
(2021) Human trajectory forecasting in crowds: A deep learning perspective. IEEE Trans-actions on Intelligent Transportation Systems.Google Scholar
Kretzschmar, H., Spies, M., Sprunk, C., & Burgard, W.
(2016) Socially compliant mobile robot navigation via inverse reinforcement learning. The International Journal of Robotics Research, 35(11), 1289–1307. DOI logoGoogle Scholar
Kruse, T., Pandey, A. K., Alami, R., & Kirsch, A.
(2013) Human-aware robot navigation: A survey. Robotics and Autonomous Systems, 61(12), 1726–1743. DOI logoGoogle Scholar
Kuderer, M., Kretzschmar, H., Sprunk, C., & Burgard, W.
(2012) Feature-based prediction of trajectories for socially compliant navigation. In Robotics: science and systems.Google Scholar
Lam, C.-P., Chou, C.-T., Chang, C.-F., & Fu, L.-C.
(2010) Human-centered robot navigation – toward a harmoniously coexisting multi-human and multi-robot environment. In Intelligent robots and systems (iros), 2010 ieee/rsj international conference on (pp. 1813–1818). Taipei, Taiwan.Google Scholar
Latombe, J.
(1991) Robot motion planning: Edition en anglais. Springer. Retrieved from [URL]. DOI logo
Lerner, A., Chrysanthou, Y., & Lischinski, D.
(2007) Crowds by example. In Computer graphics forum (Vol. 261, pp. 655–664). DOI logoGoogle Scholar
Levine, S., Popovic, Z., & Koltun, V.
(2011) Nonlinear inverse reinforcement learning with gaussian processes. Advances in neural information processing systems, 24 1, 19–27.Google Scholar
Manso, L. J., Nunez, P., Calderita, L. V., Faria, D. R., & Bachiller, P.
(2020) Socnav1: A dataset to benchmark and learn social navigation conventions. Data, 5 (1), 7. DOI logoGoogle Scholar
Martin-Martin, R., Patel, M., Rezatofighi, H., Shenoi, A., Gwak, J., Frankel, E., … Savarese, S.
(2021) Jrdb: A dataset and benchmark of egocentric robot visual perception of humans in built environments. IEEE transactions on pattern analysis and machine intelligence. DOI logoGoogle Scholar
Mavrogiannis, C., Alves-Oliveira, P., Thomason, W., & Knepper, R. A.
(2022) Social momentum: Design and evaluation of a framework for socially competent robot navigation. ACM Transactions on Human-Robot Interaction (THRI), 11 (2), 137. DOI logoGoogle Scholar
Mavrogiannis, C., Baldini, F., Wang, A., Zhao, D., Trautman, P., Steinfeld, A., & Oh, J.
(2021) Core challenges of social robot navigation: A survey. arXiv preprint arXiv:2103.05668.Google Scholar
Mead, R., Atrash, A., & Matarié, M. J.
(2011) Proxemic feature recognition for interactive robots: Automating metrics from the social sciences. In Social robotics (pp. 52–61). Springer. DOI logoGoogle Scholar
Murphy, R. R.
(2019) Introduction to ai robotics. MIT press.Google Scholar
Nonaka, S., Inoue, K., Arai, T., & Mae, Y.
(2004) Evaluation of human sense of security for coexisting robots using virtual reality. 1st report: evaluation of pick and place motion of hu-manoid robots. In Ieee international conference on robotics and automation, 2004. proceedings. icra’04. 2(004 (Vol. 31, pp. 2770–2775). DOI logoGoogle Scholar
Nourbakhsh, I., Kunz, C., & Willeke, T.
(2003) The mobot museum robot installations: a five year experiment. In Proceedings 2003 ieee/rsj international conference on intelligent robots and systems (iros 2003) (cat. no.03ch37453) (Vol. 41, p. 3636–3641 vol.31). DOI logoGoogle Scholar
Okal, B., & Arras, K. O.
(2016) Formalizing normative robot behavior. In International conference on social robotics (pp. 62–71). DOI logoGoogle Scholar
Orebäck, A., & Christensen, H. I.
(2003) Evaluation of architectures for mobile robotics. Autonomous robots, 14(1), 33–49. DOI logoGoogle Scholar
Pacchierotti, E., Christensen, H. I., & Jensfelt, P.
(2006) Evaluation of passing distance for social robots. In Roman 2006-the 15th ieee international symposium on robot and human interactive communication (pp. 315–320). DOI logoGoogle Scholar
Pellegrini, S., Ess, A., Schindler, K., & Van Gool, L.
(2009) You’ll never walk alone: Modeling social behavior for multi-target tracking. In 2009 ieee 12th international conference on computer vision (pp. 261–268). DOI logoGoogle Scholar
Pérez-Higueras, N., Caballero, F., & Merino, L.
(2018) Learning human-aware path planning with fully convolutional networks. In 2018 ieee international conference on robotics and automation (iera) (pp. 5897–5902). DOI logoGoogle Scholar
Quinlan, S., & Khatib, O.
(1993) Elastic bands: Connecting path planning and control. In [1993] proceedings ieee international conference on robotics and automation (pp. 802–807). DOI logoGoogle Scholar
Rohmer, E., Singh, S. P. N., & Freese, M.
(2013) Cop-peliaSim (formerly V-REP): a Versatile and Scalable Robot Simulation Framework. In Proc. of the international conference on intelligent robots and systems (iros). ([URL])
Rosmann, C., Hoffmann, F., & Bertram, T.
(2015) Timed-Elastic-Bands for time-optimal point-to-point nonlinear model predictive control. In 2015 european control conference (ecc) (p. 3352–3357). DOI logoGoogle Scholar
Shorten, C., & Khoshgoftaar, T. M.
(2019) A survey on image data augmentation for deep learning. Journal of big data, 6(1), 1–48. DOI logoGoogle Scholar
Stein, P., Spalanzani, A., Santos, V., & Laugier, C.
(2016) Leader following: A study on classification and selection. Robotics and Autonomous Systems, 75 1, 79–95. DOI logoGoogle Scholar
Syrdal, D. S., Koay, K. L., Walters, M. L., & Dautenhahn, K.
(2007) A personalized robot companion-the role of individual differences on spatial preferences in hri scenarios. In Robot and human interactive communication, 2007. ro-man 2007. the 16th ieee international symposium on (pp. 1143–1148). Jeju Island, Korea. DOI logoGoogle Scholar
Tai, L., Zhang, J., Liu, M., & Burgard, W.
(2018) Socially compliant navigation through raw depth inputs with generative adversarial imitation learning. In 2018 ieee international conference on robotics and automation (icra) (pp. 1111–1117). DOI logoGoogle Scholar
Thrun, S., Beetz, M., Bennewitz, M., Burgard, W., Cremers, A. B., Dellaert, F.
, … others (2000) Probabilistic algorithms and the interactive museum tour-guide robot minerva. The International Journal of Robotics Research, 19(11), 972–999. DOI logoGoogle Scholar
Tipaldi, G. D., & Arras, K. O.
(2011) Please do not disturb! minimum interference coverage for social robots. In 2011 ieee/rsj international conference on intelligent robots and systems (pp. 1968–1973).Google Scholar
Trautman, P., & Krause, A.
(2010) Unfreezing the robot: Navigation in dense, interacting crowds. In 2010 ieee/rsj international conference on intelligent robots and systems (pp. 797–803). DOI logoGoogle Scholar
Tsoi, N., Hussein, M., Espinoza, J., Ruiz, X., & Vázquez, M.
(2020) Sean: Social environment for autonomous navigation. In Proceedings of the 8th international conference on human-agent interaction (pp. 281–283). DOI logoGoogle Scholar
Vadakkepat, P., Tan, K. C., & Ming-Liang, W.
(2000) Evolutionary artificial potential fields and their application in real time robot path planning. In Proceedings of the 2000 congress on evolutionary computation. cec 00 (cat. no. 00th8512) (Vol. 11, pp. 256–263). DOI logoGoogle Scholar
Van den Berg, J., Lin, M., & Manocha, D.
(2008) Reciprocal velocity obstacles for real-time multi-agent navigation. In 2008 ieee international conference on robotics and automation (pp. 1928–1935). DOI logoGoogle Scholar
Vasquez, D., Okal, B., & Arras, K. O.
(2014) Inverse reinforcement learning algorithms and features for robot navigation in crowds: an experimental comparison. In 2014 ieee/rsj international conference on intelligent robots and systems (pp. 1341–1346). DOI logoGoogle Scholar
Vemula, A., Muelling, K., & Oh, J.
(2018) Social attention: Modeling attention in human crowds. In 2018 ieee international conference on robotics and automation (icra) (pp. 4601–4607). DOI logoGoogle Scholar
Wulfmeier, M., Ondruska, P., & Posner, I.
(2015) Maximum entropy deep inverse reinforcement learning. arXiv preprint arXiv:1507.04888.Google Scholar
Yan, Z., Duckett, T., & Bellotto, N.
(2017, September). Online learning for human classification in 3d lidar-based tracking. In In proceedings of the 2017 ieee/rsj international conference on intelligent robots and systems (iros). Vancouver, Canada. DOI logoGoogle Scholar
Yi, S., Li, H., & Wang, X.
(2015) Understanding pedestrian behaviors from stationary crowd groups. In Proceedings of the ieee conference on computer vision and pattern recognition (pp. 3488–3496). DOI logoGoogle Scholar
Zanlungo, F., Ikeda, T., & Kanda, T.
(2011) Social force model with explicit collision prediction. EPL(Europhysics Letters), 93 (6), 68005. DOI logoGoogle Scholar
Zhu, Q., Yan, Y., & Xing, Z.
(2006) Robot path planning based on artificial potential field approach with simulated annealing. In Sixth international conference on intelligent systems design and applications (Vol. 21, pp. 622–627). DOI logoGoogle Scholar