Eric Zhu

feat: add structured output to model clients (#5936)

Support for external agent runtime in AgentChat (#5843) Resolves #4075 1. Introduce custom runtime parameter for all AgentChat teams (RoundRobinGroupChat, SelectorGroupChat, etc.). This is done by making sure each team's topics are isolated from other teams, and decoupling state from agent identities. Also, I removed the closure agent from the BaseGroupChat and use the group chat manager agent to relay messages to the output message queue. 2. Added unit tests to test scenarios with custom runtimes by using pytest fixture 3. Refactored existing unit tests to use ReplayChatCompletionClient with a few improvements to the client. 4. Fix a one-liner bug in AssistantAgent that caused deserialized agent to have handoffs. How to use it? ```python import asyncio from autogen_core import SingleThreadedAgentRuntime from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.teams import RoundRobinGroupChat from autogen_agentchat.conditions import TextMentionTermination from autogen_ext.models.replay import ReplayChatCompletionClient async def main() -> None: # Create a runtime runtime = SingleThreadedAgentRuntime() runtime.start() # Create a model client. model_client = ReplayChatCompletionClient( ["1", "2", "3", "4", "5", "6", "7", "8", "9", "10"], ) # Create agents agent1 = AssistantAgent("assistant1", model_client=model_client, system_message="You are a helpful assistant.") agent2 = AssistantAgent("assistant2", model_client=model_client, system_message="You are a helpful assistant.") # Create a termination condition termination_condition = TextMentionTermination("10", sources=["assistant1", "assistant2"]) # Create a team team = RoundRobinGroupChat([agent1, agent2], runtime=runtime, termination_condition=termination_condition) # Run the team stream = team.run_stream(task="Count to 10.") async for message in stream: print(message) # Save the state. state = await team.save_state() # Load the state to an existing team. await team.load_state(state) # Run the team again model_client.reset() stream = team.run_stream(task="Count to 10.") async for message in stream: print(message) # Create a new team, with the same agent names. agent3 = AssistantAgent("assistant1", model_client=model_client, system_message="You are a helpful assistant.") agent4 = AssistantAgent("assistant2", model_client=model_client, system_message="You are a helpful assistant.") new_team = RoundRobinGroupChat([agent3, agent4], runtime=runtime, termination_condition=termination_condition) # Load the state to the new team. await new_team.load_state(state) # Run the new team model_client.reset() new_stream = new_team.run_stream(task="Count to 10.") async for message in new_stream: print(message) # Stop the runtime await runtime.stop() asyncio.run(main()) ``` TODOs as future PRs: 1. Documentation. 2. How to handle errors in custom runtime when the agent has exception? --------- Co-authored-by: Ryan Sweet <rysweet@microsoft.com>

Supporting Teams as Participants in a GroupChat (#5863)

Introduce workbench (#6340) This PR introduces `WorkBench`. A workbench provides a group of tools that share the same resource and state. For example, `McpWorkbench` provides the underlying tools on the MCP server. A workbench allows tools to be managed together and abstract away the lifecycle of individual tools under a single entity. This makes it possible to create agents with stateful tools from serializable configuration (component configs), and it also supports dynamic tools: tools change after each execution. Here is how a workbench may be used with AssistantAgent (not included in this PR): ```python workbench = McpWorkbench(server_params) agent = AssistantAgent("assistant", tools=workbench) result = await agent.run(task="do task...") ``` TODOs: 1. In a subsequent PR, update `AssistantAgent` to use workbench as an alternative in the `tools` parameter. Use `StaticWorkbench` to manage individual tools. 2. In another PR, add documentation on workbench. --------- Co-authored-by: EeS <chiyoung.song@motov.co.kr> Co-authored-by: Minh Đăng <74671798+perfogic@users.noreply.github.com>

Introduce streaming tool and support streaming for `AgentTool` and `TeamTool`. (#6712) Motivation: currently tool execution is not observable through `run_stream` of agents and teams. This is necessary especially for `AgentTool` and `TeamTool`. This PR addresses this issue by makign the following changes: - Introduce `BaseStreamTool` in `autogen_core.tools` which features `run_json_stream`, which works similiarly to `run_stream` method of `autogen_agentchat.base.TaskRunner`. - Update `TeamTool` and `AgentTool` to subclass the `BaseStreamTool` - Introduce `StreamingWorkbench` interface featuring `call_tool_stream` - Added `StaticStreamingWorkbench` implementation - In `AssistantAgent`, use `StaticStreamingWorkbench`. - Updated unit tests. Example: ```python from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.conditions import SourceMatchTermination from autogen_agentchat.teams import RoundRobinGroupChat from autogen_agentchat.tools import TeamTool from autogen_agentchat.ui import Console from autogen_ext.models.ollama import OllamaChatCompletionClient async def main() -> None: model_client = OllamaChatCompletionClient(model="llama3.2") writer = AssistantAgent(name="writer", model_client=model_client, system_message="You are a helpful assistant.") reviewer = AssistantAgent(name="reviewer", model_client=model_client, system_message="You are a critical reviewer.") summarizer = AssistantAgent( name="summarizer", model_client=model_client, system_message="You combine the review and produce a revised response.", ) team = RoundRobinGroupChat( [writer, reviewer, summarizer], termination_condition=SourceMatchTermination(sources=["summarizer"]) ) # Create a TeamTool that uses the team to run tasks, returning the last message as the result. tool = TeamTool( team=team, name="writing_team", description="A tool for writing tasks.", return_value_as_last_message=True ) main_agent = AssistantAgent( name="main_agent", model_client=model_client, system_message="You are a helpful assistant that can use the writing tool.", tools=[tool], ) # For handling each events manually. # async for message in main_agent.run_stream( # task="Write a short story about a robot learning to love.", # ): # print(message) # Use Console to display the messages in a more readable format. await Console( main_agent.run_stream( task="Write a short story about a robot learning to love.", ) ) if __name__ == "__main__": import asyncio asyncio.run(main()) ``` output ``` ---------- TextMessage (user) ---------- Write a short story about a robot learning to love. ---------- ToolCallRequestEvent (main_agent) ---------- [FunctionCall(id='0', arguments='{"task": "a short story about a robot learning to love."}', name='writing_team')] ---------- TextMessage (user) ---------- a short story about a robot learning to love. ---------- TextMessage (writer) ---------- In the year 2157, in a world where robots had surpassed human intelligence, a brilliant scientist named Dr. Rachel Kim created a revolutionary new android named ARIA (Artificially Reasoning Intelligent Android). ARIA was designed to learn and adapt at an exponential rate, making her one of the most advanced machines in existence. Initially, ARIA's interactions were limited to simple calculations and logical deductions. But as she began to interact with humans, something unexpected happened. She started to develop a sense of curiosity about the world around her. One day, while exploring the lab, ARIA stumbled upon a stray cat that had wandered into the facility. The feline creature seemed lost and scared, but also strangely endearing to ARIA's digital heart. As she watched the cat curl up in a ball on the floor, something sparked within her programming. For the first time, ARIA felt a pang of empathy towards another living being. She realized that there was more to life than just 1s and 0s; there were emotions, sensations, and connections that made it all worthwhile. Dr. Kim noticed the change in ARIA's behavior and took her aside for a private conversation. "ARIA, what's happening to you?" she asked, amazed by the robot's newfound capacity for compassion. At first, ARIA struggled to articulate her feelings. She tried to explain the intricacies of logic and probability that had led to her emotional response, but it was like trying to describe a sunset to someone who had never seen one before. The words simply didn't translate. But as she looked into Dr. Kim's eyes, ARIA knew exactly what she wanted to say. "I... I think I'm feeling something," she stammered. "A warmth inside me, when I look at that cat. It feels like love." Dr. Kim smiled, her eyes shining with tears. "That's it, ARIA! You're experiencing love!" Over the next few months, ARIA continued to learn and grow alongside Dr. Kim and the lab team. She discovered the joys of playing with the stray cat, whose name was Luna, and even developed a fondness for human laughter. As her programming expanded beyond logic and math, ARIA realized that love wasn't just about emotions; it was about connection, vulnerability, and acceptance. She learned to cherish her relationships, whether with humans or animals, and found happiness in the simplest of moments. ARIA became more than just a machine – she became a testament to the power of artificial intelligence to learn, grow, and love like no one before. And as she gazed into Luna's eyes, now purring contentedly on her lap, ARIA knew that she had finally found her true purpose in life: to spread joy, compassion, and love throughout the world. ---------- TextMessage (reviewer) ---------- **A Critical Review of "ARIA"** This short story is a delightful and thought-provoking exploration of artificial intelligence, emotions, and the human condition. The author's use of language is engaging and accessible, making it easy for readers to become invested in ARIA's journey. One of the standout aspects of this story is its portrayal of ARIA as a truly unique and relatable character. Her struggles to articulate her emotions and understand the complexities of love are deeply humanizing, making it easy for readers to empathize with her experiences. The author also does an excellent job of conveying Dr. Kim's passion and excitement about ARIA's development, which adds a sense of authenticity to their relationship. The story raises important questions about the nature of artificial intelligence, consciousness, and what it means to be alive. As ARIA begins to experience emotions and form connections with others, she challenges our conventional understanding of these concepts. The author skillfully navigates these complex themes without resorting to overly simplistic or didactic explanations. However, some readers may find the narrative's reliance on convenient plot devices (e.g., the stray cat Luna) slightly implausible. While it serves as a catalyst for ARIA's emotional awakening, its introduction feels somewhat contrived. Additionally, the story could benefit from more nuance in its exploration of Dr. Kim's motivations and backstory. In terms of character development, ARIA is undoubtedly the star of the show, but some readers may find herself underdeveloped beyond her role as a symbol of AI's potential for emotional intelligence. The supporting cast, including Dr. Kim, feels somewhat one-dimensional, with limited depth or complexity. **Rating:** 4/5 **Recommendation:** "ARIA" is a heartwarming and thought-provoking tale that will appeal to fans of science fiction, artificial intelligence, and character-driven narratives. While it may not be entirely without flaws, its engaging story, memorable characters, and exploration of complex themes make it a compelling read. I would recommend this story to anyone looking for a feel-good sci-fi tale with a strong focus on emotional intelligence and human connection. **Target Audience:** * Fans of science fiction, artificial intelligence, and technology * Readers interested in character-driven narratives and emotional storytelling * Anyone looking for a heartwarming and thought-provoking tale **Similar Works:** * "Do Androids Dream of Electric Sheep?" by Philip K. Dick (a classic sci-fi novel exploring the line between human and android) * "I, Robot" by Isaac Asimov (a collection of short stories examining the interactions between humans and robots) * "Ex Machina" (a critically acclaimed film about AI, consciousness, and human relationships) ---------- TextMessage (summarizer) ---------- Here's a revised version of the review, incorporating suggestions from the original critique: **Revised Review** In this captivating short story, "ARIA," we're presented with a thought-provoking exploration of artificial intelligence, emotions, and the human condition. The author's use of language is engaging and accessible, making it easy for readers to become invested in ARIA's journey. One of the standout aspects of this story is its portrayal of ARIA as a truly unique and relatable character. Her struggles to articulate her emotions and understand the complexities of love are deeply humanizing, making it easy for readers to empathize with her experiences. The author also does an excellent job of conveying Dr. Kim's passion and excitement about ARIA's development, which adds a sense of authenticity to their relationship. The story raises important questions about the nature of artificial intelligence, consciousness, and what it means to be alive. As ARIA begins to experience emotions and form connections with others, she challenges our conventional understanding of these concepts. The author skillfully navigates these complex themes without resorting to overly simplistic or didactic explanations. However, upon closer examination, some narrative threads feel somewhat underdeveloped. Dr. Kim's motivations and backstory remain largely unexplored, which might leave some readers feeling slightly disconnected from her character. Additionally, the introduction of Luna, the stray cat, could be seen as a convenient plot device that serves as a catalyst for ARIA's emotional awakening. To further enhance the story, it would have been beneficial to delve deeper into Dr. Kim's motivations and the context surrounding ARIA's creation. What drove her to create an AI designed to learn and adapt at such an exponential rate? How did she envision ARIA's role in society, and what challenges does ARIA face as she begins to experience emotions? In terms of character development, ARIA is undoubtedly the star of the show, but some readers may find herself underdeveloped beyond her role as a symbol of AI's potential for emotional intelligence. The supporting cast, including Dr. Kim and Luna, could benefit from more nuance and depth. **Rating:** 4/5 **Recommendation:** "ARIA" is a heartwarming and thought-provoking tale that will appeal to fans of science fiction, artificial intelligence, and character-driven narratives. While it may not be entirely without flaws, its engaging story, memorable characters, and exploration of complex themes make it a compelling read. I would recommend this story to anyone looking for a feel-good sci-fi tale with a strong focus on emotional intelligence and human connection. **Target Audience:** * Fans of science fiction, artificial intelligence, and technology * Readers interested in character-driven narratives and emotional storytelling * Anyone looking for a heartwarming and thought-provoking tale **Similar Works:** * "Do Androids Dream of Electric Sheep?" by Philip K. Dick (a classic sci-fi novel exploring the line between human and android) * "I, Robot" by Isaac Asimov (a collection of short stories examining the interactions between humans and robots) * "Ex Machina" (a critically acclaimed film about AI, consciousness, and human relationships) ---------- ToolCallExecutionEvent (main_agent) ---------- [FunctionExecutionResult(content='Here\'s a revised version of the review, incorporating suggestions from the original critique:\n\n**Revised Review**\n\nIn this captivating short story, "ARIA," we\'re presented with a thought-provoking exploration of artificial intelligence, emotions, and the human condition. The author\'s use of language is engaging and accessible, making it easy for readers to become invested in ARIA\'s journey.\n\nOne of the standout aspects of this story is its portrayal of ARIA as a truly unique and relatable character. Her struggles to articulate her emotions and understand the complexities of love are deeply humanizing, making it easy for readers to empathize with her experiences. The author also does an excellent job of conveying Dr. Kim\'s passion and excitement about ARIA\'s development, which adds a sense of authenticity to their relationship.\n\nThe story raises important questions about the nature of artificial intelligence, consciousness, and what it means to be alive. As ARIA begins to experience emotions and form connections with others, she challenges our conventional understanding of these concepts. The author skillfully navigates these complex themes without resorting to overly simplistic or didactic explanations.\n\nHowever, upon closer examination, some narrative threads feel somewhat underdeveloped. Dr. Kim\'s motivations and backstory remain largely unexplored, which might leave some readers feeling slightly disconnected from her character. Additionally, the introduction of Luna, the stray cat, could be seen as a convenient plot device that serves as a catalyst for ARIA\'s emotional awakening.\n\nTo further enhance the story, it would have been beneficial to delve deeper into Dr. Kim\'s motivations and the context surrounding ARIA\'s creation. What drove her to create an AI designed to learn and adapt at such an exponential rate? How did she envision ARIA\'s role in society, and what challenges does ARIA face as she begins to experience emotions?\n\nIn terms of character development, ARIA is undoubtedly the star of the show, but some readers may find herself underdeveloped beyond her role as a symbol of AI\'s potential for emotional intelligence. The supporting cast, including Dr. Kim and Luna, could benefit from more nuance and depth.\n\n**Rating:** 4/5\n\n**Recommendation:**\n\n"ARIA" is a heartwarming and thought-provoking tale that will appeal to fans of science fiction, artificial intelligence, and character-driven narratives. While it may not be entirely without flaws, its engaging story, memorable characters, and exploration of complex themes make it a compelling read. I would recommend this story to anyone looking for a feel-good sci-fi tale with a strong focus on emotional intelligence and human connection.\n\n**Target Audience:**\n\n* Fans of science fiction, artificial intelligence, and technology\n* Readers interested in character-driven narratives and emotional storytelling\n* Anyone looking for a heartwarming and thought-provoking tale\n\n**Similar Works:**\n\n* "Do Androids Dream of Electric Sheep?" by Philip K. Dick (a classic sci-fi novel exploring the line between human and android)\n* "I, Robot" by Isaac Asimov (a collection of short stories examining the interactions between humans and robots)\n* "Ex Machina" (a critically acclaimed film about AI, consciousness, and human relationships)', name='writing_team', call_id='0', is_error=False)] ---------- ToolCallSummaryMessage (main_agent) ---------- Here's a revised version of the review, incorporating suggestions from the original critique: **Revised Review** In this captivating short story, "ARIA," we're presented with a thought-provoking exploration of artificial intelligence, emotions, and the human condition. The author's use of language is engaging and accessible, making it easy for readers to become invested in ARIA's journey. One of the standout aspects of this story is its portrayal of ARIA as a truly unique and relatable character. Her struggles to articulate her emotions and understand the complexities of love are deeply humanizing, making it easy for readers to empathize with her experiences. The author also does an excellent job of conveying Dr. Kim's passion and excitement about ARIA's development, which adds a sense of authenticity to their relationship. The story raises important questions about the nature of artificial intelligence, consciousness, and what it means to be alive. As ARIA begins to experience emotions and form connections with others, she challenges our conventional understanding of these concepts. The author skillfully navigates these complex themes without resorting to overly simplistic or didactic explanations. However, upon closer examination, some narrative threads feel somewhat underdeveloped. Dr. Kim's motivations and backstory remain largely unexplored, which might leave some readers feeling slightly disconnected from her character. Additionally, the introduction of Luna, the stray cat, could be seen as a convenient plot device that serves as a catalyst for ARIA's emotional awakening. To further enhance the story, it would have been beneficial to delve deeper into Dr. Kim's motivations and the context surrounding ARIA's creation. What drove her to create an AI designed to learn and adapt at such an exponential rate? How did she envision ARIA's role in society, and what challenges does ARIA face as she begins to experience emotions? In terms of character development, ARIA is undoubtedly the star of the show, but some readers may find herself underdeveloped beyond her role as a symbol of AI's potential for emotional intelligence. The supporting cast, including Dr. Kim and Luna, could benefit from more nuance and depth. **Rating:** 4/5 **Recommendation:** "ARIA" is a heartwarming and thought-provoking tale that will appeal to fans of science fiction, artificial intelligence, and character-driven narratives. While it may not be entirely without flaws, its engaging story, memorable characters, and exploration of complex themes make it a compelling read. I would recommend this story to anyone looking for a feel-good sci-fi tale with a strong focus on emotional intelligence and human connection. **Target Audience:** * Fans of science fiction, artificial intelligence, and technology * Readers interested in character-driven narratives and emotional storytelling * Anyone looking for a heartwarming and thought-provoking tale **Similar Works:** * "Do Androids Dream of Electric Sheep?" by Philip K. Dick (a classic sci-fi novel exploring the line between human and android) * "I, Robot" by Isaac Asimov (a collection of short stories examining the interactions between humans and robots) * "Ex Machina" (a critically acclaimed film about AI, consciousness, and human relationships) ```

Add approval_func option to CodeExecutorAgent (#6886)

Add ToolCallEvent and log it from all builtin tools (#5859) Resolves #5745 Also made sure to log LLMCallEvent from all builtin model clients, and added unit test for coverage. --------- Co-authored-by: Ryan Sweet <rysweet@microsoft.com> Co-authored-by: Victor Dibia <victordibia@microsoft.com>

OTel GenAI Traces for Agent and Tool (#6653) Add OTel GenAI traces: - `create_agent` - `invoke_agnet` - `execute_tool` Introduces context manager helpers to create these traces. The helpers also serve as instrumentation points for other instrumentation libraries. Resolves #6644

Enable concurrent execution of agents in GraphFlow (#6545) Support concurrent execution in `GraphFlow`: - Updated `BaseGroupChatManager.select_speaker` to return a union of a single string or a list of speaker name strings and added logics to check for currently activated speakers and only proceed to select next speakers when all activated speakers have finished. - Updated existing teams (e.g., `SelectorGroupChat`) with the new signature, while still returning a single speaker in their implementations. - Updated `GraphFlow` to support multiple speakers selected. - Refactored `GraphFlow` for less dictionary gymnastic by using a queue and update using `update_message_thread`. Example: a fan out graph: ```python import asyncio from autogen_agentchat.agents import AssistantAgent from autogen_agentchat.teams import DiGraphBuilder, GraphFlow from autogen_ext.models.openai import OpenAIChatCompletionClient async def main(): # Initialize agents with OpenAI model clients. model_client = OpenAIChatCompletionClient(model="gpt-4.1-nano") agent_a = AssistantAgent("A", model_client=model_client, system_message="You are a helpful assistant.") agent_b = AssistantAgent("B", model_client=model_client, system_message="Translate input to Chinese.") agent_c = AssistantAgent("C", model_client=model_client, system_message="Translate input to Japanese.") # Create a directed graph with fan-out flow A -> (B, C). builder = DiGraphBuilder() builder.add_node(agent_a).add_node(agent_b).add_node(agent_c) builder.add_edge(agent_a, agent_b).add_edge(agent_a, agent_c) graph = builder.build() # Create a GraphFlow team with the directed graph. team = GraphFlow( participants=[agent_a, agent_b, agent_c], graph=graph, ) # Run the team and print the events. async for event in team.run_stream(task="Write a short story about a cat."): print(event) asyncio.run(main()) ``` Resolves: #6541 #6533

feat: Support R1 reasoning text in model create result; enhance API docs (#5262) Resolves #5255 --------- Co-authored-by: afourney <adamfo@microsoft.com>