Joint Human-LLM Curation: Crafting Understandable and Relevant AI-Powered Information

Charles F. Vardeman II

Center for Research Computing, University of Notre Dame

2024-02-02

When last we spoke…

Motivating Use Case

AI Based Curation

KG Driven Architecture

Inspiration from TAMMS Agent Cognitive Architecture

TAMMS Agent Architecture

“Society of SME Agents”

SME Agent “Ecosystem”

The Bad News: The world has cought up to us…

Visual Agents Architecture: Multi-Agent Architecture based on Role

Activity Specific Agents: Visual Agents Transition Graph

LangChain Multi-Agent Workflows

LangGraph: Multi-Agent Workflows

Multi-Agent Example

Multi-agent collaboration example

Curator Agents

Retrieval-Augmented Generation for Large Language Models: A Survey

Gao, Yunfan, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, et al. 2024. “Retrieval-Augmented Generation for Large Language Models: A Survey.” arXiv. https://doi.org/10.48550/arXiv.2312.10997.

Retrieval Augmented Generation – The Idea

Gao, Yunfan, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, et al. 2024. “Retrieval-Augmented Generation for Large Language Models: A Survey.” arXiv. https://doi.org/10.48550/arXiv.2312.10997.

Knowledge Engineering Dynamic QA Systems

Jerry Liu, “Beyond Naive RAG: Adding Agentic Layers, AI User Conference, 2024

Naive RAG

Jerry Liu, “Beyond Naive RAG: Adding Agentic Layers, AI User Conference, 2024

What does this concept “look like” in LLM driven architecture for QA systems

  • Agents based on LLM Cognitive Architecture

    • Curator Agents – Curation Workflow

    • Subject Matter Expert Agents

  • Knowledge Graphs

  • Tools to Query KG

  • Some form of Knowledge Engineering and Ontology

How does this “fit” into the Enterprise Data Architecture?

Gartner Data Architecture

The Curator Agent Should Interact with the Data Catalog KG

Juan Sequeda, “What Does It Mean for a Data Catalog to Be Powered by a Knowledge Graph?”,https://www.datanami.com/2022/09/30/what-does-it-mean-for-a-data-catalog-to-be-powered-by-a-knowledge-graph/

Data Catalogs Should have an Ontology

Juan Sequeda, “What Does It Mean for a Data Catalog to Be Powered by a Knowledge Graph?”,https://www.datanami.com/2022/09/30/what-does-it-mean-for-a-data-catalog-to-be-powered-by-a-knowledge-graph/

Example Ontology

Juan Sequeda, “What Does It Mean for a Data Catalog to Be Powered by a Knowledge Graph?”,https://www.datanami.com/2022/09/30/what-does-it-mean-for-a-data-catalog-to-be-powered-by-a-knowledge-graph/

The Curator Agent should follow best practices for constructing ontologies

Shimizu, Cogan, Karl Hammar, and Pascal Hitzler. 2023. “Modular Ontology Modeling.” Edited by Sabrina Kirrane, Axel-Cyrille Ngonga Ngomo, Sabrina Kirrane, and Axel-Cyrille Ngonga Ngomo. Semantic Web 14 (3): 459–89. https://www.semantic-web-journal.net/system/files/swj2886.pdf.

Extreme Design Methodology

De Berardinis, Jacopo, Valentina Anita Carriero, Nitisha Jain, Nicolas Lazzari, Albert Meroño-Peñuela, Andrea Poltronieri, and Valentina Presutti. 2023. “The Polifonia Ontology Network: Building a Semantic Backbone for Musical Heritage.” In The Semantic Web – ISWC 2023, edited by Terry R. Payne, Valentina Presutti, Guilin Qi, María Poveda-Villalón, Giorgos Stoilos, Laura Hollink, Zoi Kaoudi, Gong Cheng, and Juanzi Li, 14266:302–22. Lecture Notes in Computer Science. Cham: Springer Nature Switzerland. https://doi.org/10.1007/978-3-031-47243-5_17.

LLM Assisted Competency Question Design

IDEA: Infer, DEsign, creAte:https://github.com/polifonia-project/idea

A Curator Agent is a Knowledge Engineer…

Knowledge Engineering Using Large Language Models

Allen, Bradley P, Lise Stork, and Paul Groth. 2023. “Knowledge Engineering Using Large Language Models.” arXiv.Org. October 1, 2023. https://arxiv.org/abs/2310.00637

Prompt Engineering as Knowledge Engineering

Allen, Bradley P, Lise Stork, and Paul Groth. 2023. “Knowledge Engineering Using Large Language Models.” arXiv.Org. October 1, 2023. https://arxiv.org/abs/2310.00637

Systematic Approaches to Prompt Engineering

DSPy: Programming—not prompting—Foundation Models, https://github.com/stanfordnlp/dspy LangChain Integration: https://python.langchain.com/docs/integrations/providers/dspy

Knowledge Engineering Practice

Allen, Bradley P, Lise Stork, and Paul Groth. 2023. “Knowledge Engineering Using Large Language Models.” arXiv.Org. October 1, 2023. https://arxiv.org/abs/2310.00637

Trusted AI, LLMs and KE

Allen, Bradley P, Lise Stork, and Paul Groth. 2023. “Knowledge Engineering Using Large Language Models.” arXiv.Org. October 1, 2023. https://arxiv.org/abs/2310.00637

Building Detailed KGs as part of the Curator Agent Workflow

Paco Nathan, “Language, Graphs, and AI in Industry”, https://youtu.be/ScIHpAhPccM?si=fzVTe4KKDCgybmAQ

Knowledge Graph Construction and the Curator Agent

Paco Nathan, “Language, Graphs, and AI in Industry”, https://derwen.ai/s/mqqm