Cover not available
Article published In:
Socially Acceptable Robot Behavior: Approaches for Learning, Adaptation and Evaluation
Edited by Oliver Roesler, Elahe Bagheri, Amir Aly, Silvia Rossi and Rachid Alami
[Interaction Studies 23:3] 2022
► pp. 391–426

An epistemic logic for formalizing group dynamics of agents

| University of L’Aquila
| University of Udine
| University of L’Aquila

In the multi-agent setting, it is relevant to model group dynamics of agents, and logic has proved a good tool to do so. We propose an epistemic logic, L-DINF-E, that allows one to formalize what are the beliefs formed by a group of agents, where several groups exist and agents can pass from a group to another one. We introduce a new modality which allows an agent to reason about the beliefs of other agents. This allows us to model aspects of the “Theory of Mind”, understood as the set of social-cognitive skills involving the ability to attribute and reason about mental states, desires, beliefs, and knowledge of agents. In this paper, we present the logic L-DINF-E and illustrate how it can be used to solve “false-belief tasks”, i.e., tests in which an agent should understand that some other agent may develop, under some circumstances, false beliefs.

Keywords: epistemic logic, agents and multi-agent systems, theory of mind
Article outline
  • Introduction
  • Background on agents and verification, and on the DALI Language
  • Logical framework
  • False-beliefs: The Sally-Anne task
  • Future directions and conclusion
  • Acknowledgements
  • Notes
  • References
Published online: 21 April 2023
DOI logo
https://doi.org/10.1075/is.22019.cos
References
Adam, C., Herzig, A., & Longin, D.
(2009) A logical formalization of the OCC theory of emotions. Synth., 168 (2), 201–248. DOI logoGoogle Scholar
Aielli, F., Ancona, D., Caianiello, P., Costantini, S., De Gasperis, G., Di Marco, A., … Mascardi, V.
(2016) FRIENDLY & KIND with your health: Human-friendly knowledge-intensive dynamic systems for the e-health domain. In J. Bajo et al. (Eds.), Highlights of practical applications of scalable multi-agent systems. The PAAMS collection – international workshops of PAAMS 2016, proceedings (Vol. 616 of Communications in Computer and Information Science, pp. 15–26). Springer. DOI logoGoogle Scholar
Alberti, M., Chesani, F., Gavanelli, M., Lamma, E., Mello, P., & Torroni, P.
(2008) Verifiable agent interaction in abductive logic programming: The SCIFF framework. ACM Trans. Comput. Log., 9 (4), 291:1–29:431. DOI logoGoogle Scholar
Amir, E., Andreson, M. L., & Chaudri, V. K.
(2007) Report on DARPA workshop on self aware computer systems. Tech. Rep., SRI International Menlo Park, USA. (Full text: [URL])
Balaji, P. G., & Srinivasan, D.
(2010) An introduction to multi-agent systems. In D. Srinivasan & L. C. Jain (Eds.), Innovations in MASs and applications (Vol. 310 of Studies in Computational Intelligence). Springer. DOI logoGoogle Scholar
Balbiani, P., Fernández-Duque, D., & Lorini, E.
(2016) A logical theory of belief dynamics for resource-bounded agents. In Proceedings of AAMAS 2016 (pp. 644–652). ACM.Google Scholar
Balbiani, P., Fernandez-Duque, D., & Lorini, E.
(2019) The dynamics of epistemic attitudes in resource-bounded agents. Studia Logica, 107 (3), 457–488. DOI logoGoogle Scholar
Baron-Cohen, S., Leslie, A. M., & Frith, U.
(1985) Does the autistic child have a “Theory of Mind” ? Cognition, 21 (1), 37–46. DOI logoGoogle Scholar
Bolander, T., Dissing, L., & Herrmann, N.
(2021) DEL-based epistemic planning for human-robot collaboration: Theory and implementation. In M. Bienvenu, G. Lakemeyer, & E. Erdem (Eds.), Proceedings of the 18th international conference on principles of knowledge representation and reasoning, KR 2021, online event, november 3–12, 2021 (pp. 120–129). DOI logoGoogle Scholar
Bordini, R. H., Braubach, L., Dastani, M., El Fallah-Seghrouchni, A., Gómez-Sanz, J. J., Leite, J., … Ricci, A.
(2006) A survey of programming languages and platforms for multi-agent systems. Informatica (Slovenia), 30 (1), 33–44.Google Scholar
Bordini, R. H., Fisher, M., Visser, W., & Wooldridge, M.
(2006) Verifying multi-agent programs by model checking. Autonomous Agents and Multi-Agent Systems, 12 (2), 239–256. DOI logoGoogle Scholar
Bordini, R. H., & Hübner, J. F.
(2005) BDI agent programming in AgentSpeak using Jason (tutorial paper). In F. Toni & P. Torroni (Eds.), Computational logic in multi-agent systems, 6th international workshop, CLIMA VI, revised selected and invited papers (Vol. 3900 of Lecture Notes in Computer Science, pp. 143–164). Springer. DOI logoGoogle Scholar
Bosse, T., Memon, Z. A., & Treur, J.
(2011) A recursive BDI agent model for theory of mind and its applications. Applied Artificial Intelligence, 25 (1), 1–44. DOI logoGoogle Scholar
Bösser, T.
(2001) Autonomous agents. In N. J. Smelser & P. B. Baltes (Eds.), International encyclopedia of the social & behavioral sciences. Science Direct, Elsevier. DOI logoGoogle Scholar
Calegari, R., Ciatto, G., Mascardi, V., & Omicini, A.
(2021) Logic-based technologies for multi-agent systems: a systematic literature review. Auton. Agents Multi Agent Syst., 35 (1), 1. DOI logoGoogle Scholar
Carlsson, M., & Mildner, P.
(2012) SICStus Prolog – the first 25 years. Theory Pract. Log. Program., 12 (1–2), 35–66. DOI logoGoogle Scholar
Clark, K., & Robinson, P.
(2021) Qulog/TeleoR home page. ([URL])
Costantini, S.
(2011) Defining and maintaining agent’s experience in logical agents. In Proceedings of LANMR 2011, seventh Latin-American workshop on non-monotonic reasoning (Vol. 804 of CEUR Workshop Proceedings, p. 151–165). [URL]
(2015) ACE: a flexible environment for complex event processing in logical agents. In M. Baldoni, L. Baresi, & M. Dastani (Eds.), Engineering multi-agent systems, third intl. works., EMAS 2015, revised selected papers (Vol. 9318 of Lecture Notes in Computer Science). Springer. DOI logoGoogle Scholar
(2022) Ensuring trustworthy and ethical behaviour in intelligent logical agents. J. Log. Comput., 32 (2), 443–478. DOI logoGoogle Scholar
Costantini, S., & De Gasperis, G.
(2018) Flexible goal-directed agents’ behavior via DALI MASs and ASP modules. In 2018 AAAI spring symposia. AAAI Press.Google Scholar
Costantini, S., De Gasperis, G., Dyoub, A., & Pitoni, V.
(2018) Trustworthiness and safety for intelligent ethical logical agents via interval temporal logic and runtime self-checking. In AAAI spring symposia, Stanford university. AAAI Press.Google Scholar
Costantini, S., De Gasperis, G., & Migliarini, P.
(2019) Multi-agent system engineering for emphatic human-robot interaction. In 2019 IEEE second international conference on artificial intelligence and knowledge engineering (AIKE) (pp. 36–42). DOI logoGoogle Scholar
Costantini, S., De Gasperis, G., & Nazzicone, G.
(2017) DALI for cognitive robotics: Principles and prototype implementation. In Y. Lierler & W. Taha (Eds.), Practical aspects of declarative languages – 19th international symposium, proceedings (Vol. 10137 of Lecture Notes in Computer Science, pp. 152–162). Springer. DOI logoGoogle Scholar
Costantini, S., De Gasperis, G., Pitoni, V., & Salutari, A.
(2017) DALI: A multi agent system framework for the web, cognitive robotic and complex event processing. In D. Monica, A. Murano, S. Rubin, & L. Sauro (Eds.), Proceedings of ICTCS’17 and CILC’17 (Vol. 1949 of CEUR Workshop Proceedings, pp. 286–300). [URL]
Costantini, S., De Lauretis, L., Ferri, C., Giancola, J., & Persia, F.
(2021) A smart health assistant via DALI logical agents. In S. Monica & F. Bergenti (Eds.), Proceedings of the 36th Italian conference on computational logic, 2021 (Vol. 3002 of CEUR Workshop Proceedings, pp. 173–187). [URL]
Costantini, S., De Lauretis, L., & Persia, F.
(2022a) Intelligent agents and complex event processing to enhance patient monitoring. In M. Alviano & A. Pieris (Eds.), Proceedings Datalog 2.0 2022, co-located with LPNMR 2022. [URL]
(2022b) An intelligent ecosystem to improve patient monitoring using wearables and artificial intelligence. In R. Calegari & G. Ciatto (Eds.), Proceedings of CILC 2022, 37th Italian conference on computational logic. [URL]
Costantini, S., Dell’Acqua, P., & Pereira, L. M.
(2011) Conditional learning of rules and plans by knowledge exchange in logical agents. In N. Bassiliades, G. Governatori, & A. Paschke (Eds.), Rule-based reasoning, programming, and applications – proceedings of 5th international symposium, RuleML 2011 (Vol. 6826 of Lecture Notes in Computer Science, pp. 250–265). Springer. DOI logoGoogle Scholar
Costantini, S., Dyoub, A., & Pitoni, V.
(2018) Towards humanized ethical intelligent agents: the role of reflection and introspection. In P. Felli & M. Montali (Eds.), Proceedings of the 33rd Italian conference on computational logic (Vol. 2214 of CEUR Workshop Proceedings, pp. 82–96). [URL]. Retrieved from [URL]
Costantini, S., Formisano, A., & Pitoni, V.
(2021a) An epistemic logic for modular development of multi-agent systems. In N. Alechina, M. Baldoni, & B. Logan (Eds.), Engineering multi-agent systems – 9th international workshop, EMAS 2021, revised selected papers (Vol. 13190 of Lecture Notes in Computer Science, pp. 72–91). Springer. DOI logoGoogle Scholar
(2021b) An epistemic logic for multi-agent systems with budget and costs. In W. Faber, G. Friedrich, M. Gebser, & M. Morak (Eds.), Logics in artificial intelligence – 17th European conference, JELIA 2021, proceedings (Vol. 12678 of Lecture Notes in Computer Science, pp. 101–115). Springer. DOI logoGoogle Scholar
(2021c) A logic of inferable in multi-agent systems with budget and costs. In F. Dignum, A. Lomuscio, U. Endriss, & A. Nowé (Eds.), AAMAS ’21: 20th international conference on autonomous agents and multiagent systems, virtual event, UK, 2021 (pp. 1483–1485). ACM. DOI logoGoogle Scholar
(2022a) Cooperation among groups of agents in the epistemic logic L-DINF. In G. Governatori & A.-Y. Turhan (Eds.), Rules and reasoning – 6th international joint conference on rules and reasoning, Rule ML+RR 2022, Berlin, Germany, 2022, proceedings (Vol. 13752 of Lecture Notes in Computer Science). Springer. DOI logoGoogle Scholar
(2022b) Modelling agents roles in the epistemic logic L-DINF. In O. Arieli, G. Casini, & L. Giordano (Eds.), Proceedings of the 20th international workshop on non-monotonic reasoning, NMR 2022, part of the federated logic conference FLoC 2022, Haifa, Israel, 2022 (Vol. 3197 of CEUR Workshop Proceedings, pp. 70–79). [URL]
(2022c) Temporalizing epistemic logic L-DINF. In R. Calegari, G. Ciatto, & A. Omicini (Eds.), Proceedings of the 37th Italian conference on computational logic, Bologna, Italy, 2022 (Vol. 3204 of CEUR Workshop Proceedings, pp. 119–133). [URL]
Costantini, S., & Pitoni, V.
(2020) Towards a logic of “inferable” for self-aware transparent logical agents. In C. Musto, D. Magazzeni, S. Ruggieri, & G. Semeraro (Eds.), Proceedings of XAI.it@AIxIA 2020 (Vol. 2742 of CEUR Workshop Proceedings, pp. 68–79). [URL]. Retrieved from [URL]
Costantini, S., & Tocchio, A.
(2002) A logic programming language for multi-agent systems. In S. Flesca, S. Greco, N. Leone, & G. Ianni (Eds.), Logics in artificial intelligence, European conference, JELIA 2002, proceedings (Vol. 2424 of Lecture Notes in Computer Science, pp. 1–13). Springer. DOI logoGoogle Scholar
(2004) The DALI logic programming agent-oriented language. In J. J. Alferes & J. A. Leite (Eds.), Logics in artificial intelligence, 9th European conference, JELIA 2004, proceedings (Vol. 3229 of Lecture Notes in Computer Science, pp. 685–688). Springer. DOI logoGoogle Scholar
(2005) About declarative semantics of logic-based agent languages. In M. Baldoni, U. Endriss, A. Omicini, & P. Torroni (Eds.), Declarative agent languages and technologies III, third international workshop, DALT 2005, selected and revised papers (Vol. 3904 of Lecture Notes in Computer Science, pp. 106–123). Springer. DOI logoGoogle Scholar
Costantini, S., Tocchio, A., & Verticchio, A.
(2005) Communication and trust in the DALI logic programming agent-oriented language. Intelligenza Artificiale, 2 (1), 39–46. (Journal of the Italian Association AI*IA)Google Scholar
Dastani, M., van Riemsdijk, M. B., & Meyer, J. C.
(2005) Programming multi-agent systems in 3APL. In Multi-agent programming (Vol. 15 of Multiagent Systems Artificial Societies and Simulated Organizations, pp. 39–67). Springer. DOI logoGoogle Scholar
De Gasperis, G., Costantini, S., & Nazzicone, G.
(2014, July). DALI multi agent systems framework. DALI GitHub Software Repository. (DALI: [URL]) DOI logo
de Weerd, H., Verbrugge, R., & Verheij, B.
(2022) Higher-order theory of mind is especially useful in unpredictable negotiations. Auton. Agents Multi Agent Syst., 36 (1), 30. DOI logoGoogle Scholar
Dennis, L. A.
(2018) The MCAPL framework including the agent infrastructure layer an agent Java pathfinder. J. of Open Source Software, 3 (24), 617. DOI logoGoogle Scholar
Dennis, L. A., Bentzen, M. M., Lindner, F., & Fisher, M.
(2021) Verifiable machine ethics in changing contexts. In Thirty-fifth AAAI conference on artificial intelligence, AAAI 2021, thirty-third conference on innovative applications of artificial intelligence, IAAI 2021, the eleventh symposium on educational advances in artificial intelligence, EAAI 2021 (pp. 11470–11478). AAAI Press. DOI logoGoogle Scholar
Dennis, L. A., Fisher, M., Lincoln, N., Lisitsa, A., & Veres, S. M.
(2016) Practical verification of decision-making in agent-based autonomous systems. Autom. Softw. Eng., 23 (3), 305–359. DOI logoGoogle Scholar
Dissing, L., & Bolander, T.
(2020) Implementing theory of mind on a robot using dynamic epistemic logic. In C. Bessiere (Ed.), Proceedings of IJCAI 2020 (pp. 1615–1621). [URL]. DOI logo
Dorri, A., Kanhere, S. S., & Jurdak, R.
(2018) Multi-agent systems: A survey. IEEE Access, 6 1, 28573–28593. DOI logoGoogle Scholar
Duží, M., & Menšík, M.
(2017) Logic of inferable knowledge. In Frontiers in Artificial Intelligence and Applications, Information Modelling and Knowledge Bases XXVIII, Volume 292. IOS Press.Google Scholar
Dyoub, A., Costantini, S., Letteri, I., & Lisi, F. A.
(2021) A logic-based multi-agent system for ethical monitoring and evaluation of dialogues. In A. Formisano et al. (Eds.), Proceedings of the 37th international conference on logic programming (technical communications) (Vol. 345 of Electronic Proceedings in Theoretical Computer Science, pp. 182–188). DOI logoGoogle Scholar
Dyoub, A., Costantini, S., & Lisi, F. A.
(2019) Towards ethical machines via logic programming. In B. Bogaerts et al. (Eds.), Proceedings of the 35th international conference on logic programming (technical communications) (Vol. 306 of Electronic Proceedings in Theoretical Computer Science, pp. 333–339). DOI logoGoogle Scholar
Ferrando, A., Dennis, L. A., Ancona, D., Fisher, M., & Mascardi, V.
(2018) Verifying and validating autonomous systems: Towards an integrated approach. In C. Colombo & M. Leucker (Eds.), Runtime verification, 18th international conference, RV 2018, proceedings (Vol. 11237 of Lecture Notes in Computer Science, pp. 263–281). Springer. DOI logoGoogle Scholar
Ferrando, A., Winikoff, M., Cranefield, S., Dignum, F., & Mascardi, V.
(2019) On enactability of agent interaction protocols: Towards a unified approach. In L. A. Dennis, R. H. Bordini, & Y. Lespérance (Eds.), Engineering multi-agent systems – 7th international workshop, EMAS 2019, revised selected papers (Vol. 12058 of Lecture Notes in Computer Science, pp. 43–64). Springer. DOI logoGoogle Scholar
Ferrario, A., & Loi, M.
(2022) How explainability contributes to trust in AI. In Facct ’22: 2022 ACM conference on fairness, accountability, and transparency (pp. 1457–1466). ACM. DOI logoGoogle Scholar
Fisher, M., Bordini, R. H., Hirsch, B., & Torroni, P.
(2007) Computational logics and agents: a road map of current technologies and future trends. Computational Intelligence Journal, 23 (1), 61–91. DOI logoGoogle Scholar
Garro, A., Mühlhäuser, M., Tundis, A., Baldoni, M., Baroglio, C., Bergenti, F., & Torroni, P.
(2019) Intelligent agents: Multi-agent systems. In S. Ranganathan, M. Gribskov, K. Nakai, & C. Schonbach (Eds.), Encyclopedia of bioin formatics and computational biology – Vol. 1 (pp. 315–320). Elsevier. DOI logoGoogle Scholar
Goldman, A. I.
(2012) Theory of mind. In E. Margolis, R. Samuels, & S. P. Stich (Eds.), The Oxford handbook of philosophy of cognitive science (Vol. 11). Oxford University Press. DOI logoGoogle Scholar
Hindriks, K. V., van der Hoek, W., & Meyer, J. C.
(2012) GOAL agents instantiate intention logic. In Logic programs, norms and action (Vol. 7360 of Lecture Notes in Computer Science, pp. 196–219). Springer. DOI logoGoogle Scholar
Holzmann, G. J.
(1991) Design and validation of computer protocols. Prentice Hall Intl.: Hemel Hempstead, England.Google Scholar
Jones, A. V., & Lomuscio, A.
(2010) Distributed BDD-based BMC for the verification of multi-agent systems. In W. van der Hoek, G. A. Kaminka, Y. Lespérance, M. Luck, & S. Sen (Eds.), 9th international conference on autonomous agents and multiagent systems (AAMAS 2010) (pp. 675–682). IFAAMAS.Google Scholar
Kacprzak, M., Lomuscio, A., & Penczek, W.
(2004) Verification of multiagent systems via unbounded model checking. In Proceedings of the third int. joint conf. on autonomous agents and multiagent systems, AAMAS 04 (p. 638–645). ACM Press.Google Scholar
Kong, J., & Lomuscio, A.
(2017) Symbolic model checking multi-agent systems against CTL*K specifications. In K. Larson, M. Winikoff, S. Das, & E. H. Durfee (Eds.), Proceedings of the 16th conference on autonomous agents and multiagent systems, AAMAS 2017 (pp. 114–122). ACM.Google Scholar
Lloyd, J. W.
(1987) Foundations of logic programming, second edition. Berlin: Springer. DOI logoGoogle Scholar
Lomuscio, A., Lasica, T., & Penczek, W.
(2002) Bounded model checking for interpreted systems: Preliminary experimental results. In M. G. Hinchey, J. L. Rash, W. Truszkowski, C. A. Rouff, & D. F. Gordon-Spears (Eds.), Formal approaches to agent-based systems, 2nd international workshop, FAABS 2002, revised papers (Vol. 2699 of Lecture Notes in Computer Science, pp. 115–125). Springer. DOI logoGoogle Scholar
Lomuscio, A., Qu, H., & Raimondi, F.
(2017) MCMAS: an open-source model checker for the verification of multi-agent systems. Int. J. Softw. Tools Technol. Transf., 19 (1), 9–30. DOI logoGoogle Scholar
Martinich, A. P.
(2009) The philosophy of language. Oxford University Press. (International Fifth Edition)Google Scholar
McMillan, K. L.
(1993) Symbolic model checking. Kluwer Academic Publishers. DOI logoGoogle Scholar
Panisson, A. R., Sarkadi, S., McBurney, P., Parsons, S., & Bordini, R. H.
(2018) On the formal semantics of theory of mind in agent communication. In M. Lujak (Ed.), Agreement technologies – 6th international conference, AT 2018, revised selected papers (Vol. 11327 of Lecture Notes in Computer Science, pp. 18–32). Springer.Google Scholar
Rao, A. S., & Georgeff, M. P.
(1991) Modeling rational agents within a BDI-architecture. In J. F. Allen, R. Fikes, & E. Sandewall (Eds.), Proceedings of the 2nd international conference on principles of knowledge representation and reasoning (KR 91) (pp. 473–484). Morgan Kaufmann.Google Scholar
Reisenzein, R., Hudlicka, E., Dastani, M., Gratch, J., Hindriks, K. V., Lorini, E., & Meyer, J. C.
(2013) Computational modeling of emotion: Toward improving the inter- and intradisciplinary exchange. IEEE Trans. Affect. Comput., 4 (3), 246–266. DOI logoGoogle Scholar
Rozier, K. Y.
(2011) Linear temporal logic symbolic model checking. Comput. Sci. Rev., 5 (2), 163–203. DOI logoGoogle Scholar
(2016) Specification: The biggest bottleneck in formal methods and autonomy. In S. Blazy & M. Chechik (Eds.), Verified software. theories, tools, and experiments – 8th international conference, VSTTE 2016, revised selected papers (Vol. 9971 of Lecture Notes in Computer Science, pp. 8–26). DOI logoGoogle Scholar
Rozier, K. Y., & Vardi, M. Y.
(2010) LTL satisfiability checking. Int. J. Softw. Tools Technol. Transf., 12 (2), 123–137. DOI logoGoogle Scholar
Sarkadi, S., Panisson, A. R., Bordini, R. H., McBurney, P., & Parsons, S.
(2018) Towards an approach for modelling uncertain theory of mind in multi-agent systems. In M. Lujak (Ed.), Agreement technologies – 6th international conference, AT 2018, revised selected papers (Vol. 11327 of Lecture Notes in Computer Science, pp. 3–17). Springer. DOI logoGoogle Scholar
Stone, P., & Veloso, M. M.
(2000) Multiagent systems: A survey from a machine learning perspective. Auton. Robots, 8 (3), 345–383. DOI logoGoogle Scholar
Tørresen, J., Plessl, C., & Yao, X.
(2015) Self-aware and self-expressive systems. IEEE Computer, 48 (7), 18–20. DOI logoGoogle Scholar
van Ditmarsch, H., Halpern, J. Y., van der Hoek, W., & Kooi, B.
(2015) Handbook of epistemic logic. College Publications.Google Scholar
Vardi, M. Y.
(2001) Branching vs. linear time: Final showdown. In Proceedings of the 2001 conf. on tools and algorithms for the construction and analysis of systems, TACAS 2001 (p. 1–22). Springer-Verlag. DOI logoGoogle Scholar
Veltman, K., de Weerd, H., & Verbrugge, R.
(2019) Training the use of theory of mind using artificial agents. J. Multimodal User Interfaces, 13 (1), 3–18. DOI logoGoogle Scholar
Weyhrauch, R. W.
(1980) Prolegomena to a theory of mechanized formal reasoning. Artif. Intell., 13 (1–2), 133–170. DOI logoGoogle Scholar
Yakouda, M., Abbel, W., Corneille, K. V., & Sinclair, N. D.
(2020) Multi-agent system: A two-level BDI model integrating theory of mind. International Journal of Engineering Research & Technology (IJERT), 9 (7). Retrieved from [URL]
Zhao, S., Wang, S., Soleymani, M., Joshi, D., & Ji, Q.
(2019) Affective computing for large-scale heterogeneous multimedia data: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications, 15 ( 3s ), 1–32. DOI logoGoogle Scholar