What Dr. Vardeman is worried about: Trusted cognitive Agents

Charles F Vardeman II

2023-08-21

Motivation: Tatical ammunition management Micro Services (TAMMS)

Motivation: TAMMS cognitive Agents

Motivation: TAMMS KG

Cognitive Agents based on LLMs

Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv, August 5, 2023. http://arxiv.org/abs/2304.03442.

Cognitive Agents based on LLMs Architecture

Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. “Generative Agents: Interactive Simulacra of Human Behavior.” arXiv, August 5, 2023. http://arxiv.org/abs/2304.03442.

Cognitive Agents based on LLMs Architecture Framework

Park, Joon Sung. “Generative Agents: Interactive Simulacra of Human Behavior,” August 22, 2023. https://github.com/joonspk-research/generative_agents.

Activity Specific Agents: Web Agents

Deng, Xiang, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. “Mind2Web: Towards a Generalist Agent for the Web.” arXiv, June 14, 2023. http://arxiv.org/abs/2306.06070.

Activity Specific Agents: Web Agents Architecture

Deng, Xiang, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. “Mind2Web: Towards a Generalist Agent for the Web.” arXiv, June 14, 2023. http://arxiv.org/abs/2306.06070.

Activity Specific Agents: Visual Agents

“Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.

Visual Agents Architecture: Different LLMs based on Role

“Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.

Activity Specific Agents: Visual Agents Transition Graph

“Autonomous Visual Information Seeking with Large Language Models,” Google AI Blog,August 18, 2023. https://ai.googleblog.com/2023/08/autonomous-visual-information-seeking.html.

Aside: Transformer Vision and segmentation models

“Alaamaalouf/FollowAnything.” Accessed August 22, 2023. https://github.com/alaamaalouf/FollowAnything.

Maalouf, Alaa, Ninad Jadhav, Krishna Murthy Jatavallabhula, Makram Chahine, Daniel M. Vogt, Robert J. Wood, Antonio Torralba, and Daniela Rus. “Follow Anything: Open-Set Detection, Tracking, and Following in Real-Time.” arXiv, August 10, 2023. https://doi.org/10.48550/arXiv.2308.05737.

Graphs as a Prompt Structure

Besta, Maciej, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, et al. “Graph of Thoughts: Solving Elaborate Problems with Large Language Models.” arXiv, August 21, 2023. http://arxiv.org/abs/2308.09687.

Graphs as a Prompt Structure

Besta, Maciej, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, et al. “Graph of Thoughts: Solving Elaborate Problems with Large Language Models.” arXiv, August 21, 2023. http://arxiv.org/abs/2308.09687.

Graphs as a Prompt Structure

Besta, Maciej, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, et al. “Graph of Thoughts: Solving Elaborate Problems with Large Language Models.” arXiv, August 21, 2023. http://arxiv.org/abs/2308.09687.

Aside: Prompt Engineering Guide

“Prompt Engineering Guide.” Accessed August 22, 2023. https://www.promptingguide.ai/.

APIs as Tools: Gorilla

Patil, Shishir G., Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. “Gorilla: Large Language Model Connected with Massive APIs.” arXiv, May 24, 2023. http://arxiv.org/abs/2305.15334.

APIs as Tools: Architecture

Patil, Shishir G., Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. “Gorilla: Large Language Model Connected with Massive APIs.” arXiv, May 24, 2023. http://arxiv.org/abs/2305.15334.

APIs as Tools

Patil, Shishir. “Gorilla: Large Language Model Connected with Massive APIs [Project Website].” Python, August 22, 2023. https://github.com/ShishirPatil/gorilla.

LLM Personality Traits

Safdari, Mustafa, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. “Personality Traits in Large Language Models.” arXiv, June 30, 2023. http://arxiv.org/abs/2307.00184.

Using Personality Tests

Safdari, Mustafa, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. “Personality Traits in Large Language Models.” arXiv, June 30, 2023. http://arxiv.org/abs/2307.00184.

Behavior Persona Through Prompting

Safdari, Mustafa, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. “Personality Traits in Large Language Models.” arXiv, June 30, 2023. http://arxiv.org/abs/2307.00184.

LLM Personality Traits Malleability

Safdari, Mustafa, Greg Serapio-García, Clément Crepy, Stephen Fitz, Peter Romero, Luning Sun, Marwa Abdulhai, Aleksandra Faust, and Maja Matarić. “Personality Traits in Large Language Models.” arXiv, June 30, 2023. http://arxiv.org/abs/2307.00184.

Grounding Through Hybrid KGs

Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. “Unifying Large Language Models and Knowledge Graphs: A Roadmap.” arXiv, June 14, 2023. http://arxiv.org/abs/2306.08302.

LlamaIndex to Build Hybrid KGs

Liu, Jerry. “LlamaIndex.” Python, November 2022. https://doi.org/10.5281/zenodo.1234.

LlamaIndex to Build Hybrid KGs

“Custom Retriever Combining KG Index and VectorStore Index - LlamaIndex 🦙 0.8.5.Post2.” Accessed August 22, 2023. https://gpt-index.readthedocs.io/en/stable/examples/index_structs/knowledge_graph/KnowledgeGraphIndex_vs_VectorStoreIndex_vs_CustomIndex_combined.html.

Data Agents in LlamaIndex

From: Liu, Jerry. “Data Agents.” LlamaIndex Blog (blog), July 13, 2023. https://medium.com/llamaindex-blog/data-agents-eed797d7972f.