[Improve LLMCouncil] [Improved docs]

pull/1182/head^2
Kye Gomez 3 days ago
parent 3dd8d0604a
commit 74f7bcd2b7

@ -2,61 +2,35 @@
```mermaid ```mermaid
flowchart TD flowchart TD
A[User Query] --> B[LLM Council Initialization] A[User Query] --> B[Council Members]
B --> C{Council Members Provided?}
C -->|No| D[Create Default Council]
C -->|Yes| E[Use Provided Members]
D --> F[Step 1: Parallel Response Generation]
E --> F
subgraph "Default Council Members" subgraph "Council Members"
G1[GPT-5.1-Councilor<br/>Analytical & Comprehensive] C1[GPT-5.1-Councilor]
G2[Gemini-3-Pro-Councilor<br/>Concise & Structured] C2[Gemini-3-Pro-Councilor]
G3[Claude-Sonnet-4.5-Councilor<br/>Thoughtful & Balanced] C3[Claude-Sonnet-4.5-Councilor]
G4[Grok-4-Councilor<br/>Creative & Innovative] C4[Grok-4-Councilor]
end end
F --> G1 B --> C1
F --> G2 B --> C2
F --> G3 B --> C3
F --> G4 B --> C4
G1 --> H[Collect All Responses] C1 --> D[Responses]
G2 --> H C2 --> D
G3 --> H C3 --> D
G4 --> H C4 --> D
H --> I[Step 2: Anonymize Responses] D --> E[Anonymize & Evaluate]
I --> J[Assign Anonymous IDs: A, B, C, D...] E --> F[Chairman Synthesis]
F --> G[Final Response]
J --> K[Step 3: Parallel Evaluation]
subgraph "Evaluation Phase"
K --> L1[Member 1 Evaluates All]
K --> L2[Member 2 Evaluates All]
K --> L3[Member 3 Evaluates All]
K --> L4[Member 4 Evaluates All]
end
L1 --> M[Collect Evaluations & Rankings]
L2 --> M
L3 --> M
L4 --> M
M --> N[Step 4: Chairman Synthesis]
N --> O[Chairman Agent]
O --> P[Final Synthesized Response]
P --> Q[Return Results Dictionary]
style A fill:#e1f5ff
style P fill:#c8e6c9
style Q fill:#c8e6c9
style O fill:#fff9c4
``` ```
The `LLMCouncil` class orchestrates multiple specialized LLM agents to collaboratively answer queries through a structured peer review and synthesis process. Inspired by Andrej Karpathy's llm-council implementation, this architecture demonstrates how different models evaluate and rank each other's work, often selecting responses from other models as superior to their own. The `LLMCouncil` class orchestrates multiple specialized LLM agents to collaboratively answer queries through a structured peer review and synthesis process. Inspired by Andrej Karpathy's llm-council implementation, this architecture demonstrates how different models evaluate and rank each other's work, often selecting responses from other models as superior to their own.
The class automatically tracks all agent messages in a `Conversation` object and formats output using `history_output_formatter`, providing flexible output formats including dictionaries, lists, strings, JSON, YAML, and more.
## Workflow Overview ## Workflow Overview
The LLM Council follows a four-step process: The LLM Council follows a four-step process:
@ -80,6 +54,8 @@ class LLMCouncil:
|-----------|------|-------------|---------| |-----------|------|-------------|---------|
| `council_members` | `List[Agent]` | List of Agent instances representing council members | `None` (creates default council) | | `council_members` | `List[Agent]` | List of Agent instances representing council members | `None` (creates default council) |
| `chairman` | `Agent` | The Chairman agent responsible for synthesizing responses | Created during initialization | | `chairman` | `Agent` | The Chairman agent responsible for synthesizing responses | Created during initialization |
| `conversation` | `Conversation` | Conversation object tracking all messages throughout the workflow | Created during initialization |
| `output_type` | `HistoryOutputType` | Format for the output (e.g., "dict", "list", "string", "json", "yaml") | `"dict"` |
| `verbose` | `bool` | Whether to print progress and intermediate results | `True` | | `verbose` | `bool` | Whether to print progress and intermediate results | `True` |
## Methods ## Methods
@ -92,9 +68,13 @@ Initializes the LLM Council with council members and a Chairman agent.
| Parameter | Type | Default | Description | | Parameter | Type | Default | Description |
|-----------|------|---------|-------------| |-----------|------|---------|-------------|
| `id` | `str` | `swarm_id()` | Unique identifier for the council instance. |
| `name` | `str` | `"LLM Council"` | Name of the council instance. |
| `description` | `str` | `"A collaborative council..."` | Description of the council's purpose. |
| `council_members` | `Optional[List[Agent]]` | `None` | List of Agent instances representing council members. If `None`, creates default council with GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok-4. | | `council_members` | `Optional[List[Agent]]` | `None` | List of Agent instances representing council members. If `None`, creates default council with GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok-4. |
| `chairman_model` | `str` | `"gpt-5.1"` | Model name for the Chairman agent that synthesizes responses. | | `chairman_model` | `str` | `"gpt-5.1"` | Model name for the Chairman agent that synthesizes responses. |
| `verbose` | `bool` | `True` | Whether to print progress and intermediate results. | | `verbose` | `bool` | `True` | Whether to print progress and intermediate results. |
| `output_type` | `HistoryOutputType` | `"dict"` | Format for the output. Options: "list", "dict", "string", "final", "json", "yaml", "xml", "dict-all-except-first", "str-all-except-first", "dict-final", "list-final". |
#### Returns #### Returns
@ -105,12 +85,13 @@ Initializes the LLM Council with council members and a Chairman agent.
#### Description #### Description
Creates an LLM Council instance with specialized council members. If no members are provided, it creates a default council consisting of: Creates an LLM Council instance with specialized council members. If no members are provided, it creates a default council consisting of:
- **GPT-5.1-Councilor**: Analytical and comprehensive responses - **GPT-5.1-Councilor**: Analytical and comprehensive responses
- **Gemini-3-Pro-Councilor**: Concise and well-processed responses - **Gemini-3-Pro-Councilor**: Concise and well-processed responses
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced responses - **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced responses
- **Grok-4-Councilor**: Creative and innovative responses - **Grok-4-Councilor**: Creative and innovative responses
The Chairman agent is automatically created with a specialized prompt for synthesizing responses. The Chairman agent is automatically created with a specialized prompt for synthesizing responses. A `Conversation` object is also initialized to track all messages throughout the workflow, including user queries, council member responses, evaluations, and the final synthesis.
#### Example Usage #### Example Usage
@ -120,7 +101,7 @@ from swarms.structs.llm_council import LLMCouncil
# Create council with default members # Create council with default members
council = LLMCouncil(verbose=True) council = LLMCouncil(verbose=True)
# Create council with custom members # Create council with custom members and output format
from swarms import Agent from swarms import Agent
custom_members = [ custom_members = [
Agent(agent_name="Expert-1", model_name="gpt-4", max_loops=1), Agent(agent_name="Expert-1", model_name="gpt-4", max_loops=1),
@ -129,7 +110,8 @@ custom_members = [
council = LLMCouncil( council = LLMCouncil(
council_members=custom_members, council_members=custom_members,
chairman_model="gpt-4", chairman_model="gpt-4",
verbose=True verbose=True,
output_type="json" # Output as JSON string
) )
``` ```
@ -137,7 +119,7 @@ council = LLMCouncil(
### `run` ### `run`
Executes the full LLM Council workflow: parallel responses, anonymization, peer review, and synthesis. Executes the full LLM Council workflow: parallel responses, anonymization, peer review, and synthesis. All messages are tracked in the conversation object and formatted according to the `output_type` setting.
#### Parameters #### Parameters
@ -149,54 +131,79 @@ Executes the full LLM Council workflow: parallel responses, anonymization, peer
| Type | Description | | Type | Description |
|------|-------------| |------|-------------|
| `Dict` | Dictionary containing the following keys: | | `Union[List, Dict, str]` | Formatted output based on `output_type`. The output contains the conversation history with all messages tracked throughout the workflow. |
#### Return Dictionary Structure #### Output Format
| Key | Type | Description | The return value depends on the `output_type` parameter set during initialization:
|-----|------|-------------|
| `query` | `str` | The original user query. | - **`"dict"`** (default): Returns conversation as a dictionary/list of message dictionaries
| `original_responses` | `Dict[str, str]` | Dictionary mapping council member names to their original responses. | - **`"list"`**: Returns conversation as a list of formatted strings (`"role: content"`)
| `evaluations` | `Dict[str, str]` | Dictionary mapping evaluator names to their evaluation texts (rankings and reasoning). | - **`"string"`** or **`"str"`**: Returns conversation as a formatted string
| `final_response` | `str` | The Chairman's synthesized final answer combining all perspectives. | - **`"final"`** or **`"last"`**: Returns only the content of the final message (Chairman's response)
| `anonymous_mapping` | `Dict[str, str]` | Mapping from anonymous IDs (A, B, C, D) to member names for reference. | - **`"json"`**: Returns conversation as a JSON string
- **`"yaml"`**: Returns conversation as a YAML string
- **`"xml"`**: Returns conversation as an XML string
- **`"dict-all-except-first"`**: Returns all messages except the first as a dictionary
- **`"str-all-except-first"`**: Returns all messages except the first as a string
- **`"dict-final"`**: Returns the final message as a dictionary
- **`"list-final"`**: Returns the final message as a list
#### Conversation Tracking
All messages are automatically tracked in the conversation object with the following roles:
- **`"User"`**: The original user query
- **`"{member_name}"`**: Each council member's response (e.g., "GPT-5.1-Councilor")
- **`"{member_name}-Evaluation"`**: Each council member's evaluation (e.g., "GPT-5.1-Councilor-Evaluation")
- **`"Chairman"`**: The final synthesized response
#### Description #### Description
Executes the complete LLM Council workflow: Executes the complete LLM Council workflow:
1. **Dispatch Phase**: Sends the query to all council members in parallel using `run_agents_concurrently` 1. **User Query Tracking**: Adds the user query to the conversation as "User" role
2. **Collection Phase**: Collects all responses and maps them to member names 2. **Dispatch Phase**: Sends the query to all council members in parallel using `run_agents_concurrently`
3. **Anonymization Phase**: Creates anonymous IDs (A, B, C, D, etc.) and shuffles them to ensure anonymity 3. **Collection Phase**: Collects all responses, maps them to member names, and adds each to the conversation with the member's name as the role
4. **Evaluation Phase**: Each member evaluates and ranks all anonymized responses using `batched_grid_agent_execution` 4. **Anonymization Phase**: Creates anonymous IDs (A, B, C, D, etc.) and shuffles them to ensure anonymity
5. **Synthesis Phase**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer 5. **Evaluation Phase**: Each member evaluates and ranks all anonymized responses using `batched_grid_agent_execution`, then adds evaluations to the conversation with "{member_name}-Evaluation" as the role
6. **Synthesis Phase**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer, which is added to the conversation as "Chairman" role
7. **Output Formatting**: Returns the conversation formatted according to the `output_type` setting using `history_output_formatter`
The method provides verbose output by default, showing progress at each stage. The method provides verbose output by default, showing progress at each stage. All messages are tracked in the `conversation` attribute for later access or export.
#### Example Usage #### Example Usage
```python ```python
from swarms.structs.llm_council import LLMCouncil from swarms.structs.llm_council import LLMCouncil
# Create council with default output format (dict)
council = LLMCouncil(verbose=True) council = LLMCouncil(verbose=True)
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?" query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
# Run the council - returns formatted conversation based on output_type
result = council.run(query) result = council.run(query)
# Access the final synthesized response # With default "dict" output_type, result is a list of message dictionaries
print(result["final_response"]) # Access conversation messages
for message in result:
print(f"{message['role']}: {message['content'][:200]}...")
# Access the conversation object directly for more control
conversation = council.conversation
print("\nFinal message:", conversation.get_final_message_content())
# Access individual member responses # Get conversation as string
for name, response in result["original_responses"].items(): print("\nFull conversation:")
print(f"{name}: {response[:200]}...") print(conversation.get_str())
# Access evaluation rankings # Example with different output types
for evaluator, evaluation in result["evaluations"].items(): council_json = LLMCouncil(output_type="json", verbose=False)
print(f"{evaluator} evaluation:\n{evaluation[:300]}...") result_json = council_json.run(query) # Returns JSON string
# Check anonymous mapping council_final = LLMCouncil(output_type="final", verbose=False)
print("Anonymous IDs:", result["anonymous_mapping"]) result_final = council_final.run(query) # Returns only final response string
``` ```
--- ---
@ -225,6 +232,7 @@ Internal method that creates the default council configuration with four special
- **Grok-4-Councilor** (`model_name="x-ai/grok-4"`): Creative and innovative, temperature=0.8 - **Grok-4-Councilor** (`model_name="x-ai/grok-4"`): Creative and innovative, temperature=0.8
Each agent is configured with: Each agent is configured with:
- Specialized system prompts matching their role - Specialized system prompts matching their role
- `max_loops=1` for single-response generation - `max_loops=1` for single-response generation
- `verbose=False` to reduce noise during parallel execution - `verbose=False` to reduce noise during parallel execution
@ -367,25 +375,40 @@ For comprehensive examples demonstrating various use cases, see the [LLM Council
```python ```python
from swarms.structs.llm_council import LLMCouncil from swarms.structs.llm_council import LLMCouncil
# Create the council # Create the council with default output format
council = LLMCouncil(verbose=True) council = LLMCouncil(verbose=True)
# Example query # Example query
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?" query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
# Run the council # Run the council - returns formatted conversation
result = council.run(query) result = council.run(query)
# Print final response # With default "dict" output_type, result is a list of message dictionaries
print(result["final_response"]) # Print all messages
for message in result:
role = message['role']
content = message['content']
print(f"\n{role}:")
print(content[:500] + "..." if len(content) > 500 else content)
# Optionally print evaluations # Access conversation object directly for more options
print("\n\n" + "="*80) conversation = council.conversation
print("EVALUATIONS")
# Get only the final response
print("\n" + "="*80)
print("FINAL RESPONSE")
print("="*80) print("="*80)
for name, evaluation in result["evaluations"].items(): print(conversation.get_final_message_content())
print(f"\n{name}:")
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation) # Get conversation as formatted string
print("\n" + "="*80)
print("FULL CONVERSATION")
print("="*80)
print(conversation.get_str())
# Export conversation to JSON
conversation.export()
``` ```
## Customization ## Customization
@ -428,6 +451,50 @@ council = LLMCouncil(
) )
``` ```
### Custom Output Format
You can control the output format using the `output_type` parameter:
```python
# Get output as JSON string
council = LLMCouncil(output_type="json")
result = council.run(query) # Returns JSON string
# Get only the final response
council = LLMCouncil(output_type="final")
result = council.run(query) # Returns only final response string
# Get as YAML
council = LLMCouncil(output_type="yaml")
result = council.run(query) # Returns YAML string
# Get as formatted string
council = LLMCouncil(output_type="string")
result = council.run(query) # Returns formatted conversation string
```
### Accessing Conversation History
The conversation object is accessible for advanced usage:
```python
council = LLMCouncil()
council.run(query)
# Access conversation directly
conversation = council.conversation
# Get conversation history
history = conversation.conversation_history
# Export to file
conversation.export() # Saves to default location
# Get specific format
json_output = conversation.to_json()
yaml_output = conversation.return_messages_as_dictionary()
```
## Architecture Benefits ## Architecture Benefits
1. **Diversity**: Multiple models provide varied perspectives and approaches 1. **Diversity**: Multiple models provide varied perspectives and approaches
@ -436,6 +503,8 @@ council = LLMCouncil(
4. **Transparency**: Full visibility into individual responses and evaluation rankings 4. **Transparency**: Full visibility into individual responses and evaluation rankings
5. **Scalability**: Easy to add or remove council members 5. **Scalability**: Easy to add or remove council members
6. **Flexibility**: Supports custom agents and models 6. **Flexibility**: Supports custom agents and models
7. **Conversation Tracking**: All messages are automatically tracked in a Conversation object for history and export
8. **Flexible Output**: Multiple output formats supported via `history_output_formatter` (dict, list, string, JSON, YAML, XML, etc.)
## Performance Considerations ## Performance Considerations
@ -443,11 +512,14 @@ council = LLMCouncil(
- **Anonymization**: Responses are anonymized to prevent bias in evaluation - **Anonymization**: Responses are anonymized to prevent bias in evaluation
- **Model Selection**: Different models can be used for different roles based on their strengths - **Model Selection**: Different models can be used for different roles based on their strengths
- **Verbose Mode**: Can be disabled for production use to reduce output - **Verbose Mode**: Can be disabled for production use to reduce output
- **Conversation Management**: Conversation object efficiently tracks all messages in memory and supports export to JSON/YAML files
- **Output Formatting**: Choose lightweight output formats (e.g., "final") for production to reduce memory usage
## Related Documentation ## Related Documentation
- [Multi-Agent Architectures Overview](overview.md) - [Multi-Agent Architectures Overview](overview.md)
- [Council of Judges](council_of_judges.md) - Similar peer review pattern - [Council of Judges](council_of_judges.md) - Similar peer review pattern
- [Agent Class Reference](agent.md) - Understanding individual agents - [Agent Class Reference](agent.md) - Understanding individual agents
- [Conversation Class Reference](conversation.md) - Understanding conversation tracking and management
- [Multi-Agent Execution Utilities](various_execution_methods.md) - Underlying execution methods - [Multi-Agent Execution Utilities](various_execution_methods.md) - Underlying execution methods
- [History Output Formatter](../../../swarms/utils/history_output_formatter.py) - Output formatting utilities

@ -29,4 +29,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -27,4 +27,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -27,4 +27,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -29,4 +29,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -26,4 +26,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -34,4 +34,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -28,4 +28,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -29,4 +29,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -29,4 +29,3 @@ result = council.run(query)
# Print final response # Print final response
print(result["final_response"]) print(result["final_response"])

@ -15,5 +15,8 @@ print(result["final_response"])
# Optionally print evaluations # Optionally print evaluations
for name, evaluation in result["evaluations"].items(): for name, evaluation in result["evaluations"].items():
print(f"\n{name}:") print(f"\n{name}:")
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation) print(
evaluation[:500] + "..."
if len(evaluation) > 500
else evaluation
)

@ -88,9 +88,7 @@ class ReasoningAgentRouter:
eval: bool = False, eval: bool = False,
random_models_on: bool = False, random_models_on: bool = False,
majority_voting_prompt: Optional[str] = None, majority_voting_prompt: Optional[str] = None,
reasoning_model_name: Optional[ reasoning_model_name: Optional[str] = "gpt-4o",
str
] = "gpt-4o",
): ):
""" """
Initialize the ReasoningAgentRouter with the specified configuration. Initialize the ReasoningAgentRouter with the specified configuration.

@ -35,9 +35,7 @@ class ReasoningDuo:
model_names: list[str] = ["gpt-4o-mini", "gpt-4.1"], model_names: list[str] = ["gpt-4o-mini", "gpt-4.1"],
system_prompt: str = "You are a helpful assistant that can answer questions and help with tasks.", system_prompt: str = "You are a helpful assistant that can answer questions and help with tasks.",
output_type: OutputType = "dict-all-except-first", output_type: OutputType = "dict-all-except-first",
reasoning_model_name: Optional[ reasoning_model_name: Optional[str] = "gpt-4o",
str
] = "gpt-4o",
max_loops: int = 1, max_loops: int = 1,
*args, *args,
**kwargs, **kwargs,

@ -679,7 +679,7 @@ class AOP:
self.tool_configs: Dict[str, AgentToolConfig] = {} self.tool_configs: Dict[str, AgentToolConfig] = {}
self.task_queues: Dict[str, TaskQueue] = {} self.task_queues: Dict[str, TaskQueue] = {}
self.transport = transport self.transport = transport
self.mcp_server = FastMCP( self.mcp_server = FastMCP(
name=server_name, name=server_name,
port=port, port=port,

@ -17,12 +17,14 @@ from swarms.structs.multi_agent_exec import (
run_agents_concurrently, run_agents_concurrently,
batched_grid_agent_execution, batched_grid_agent_execution,
) )
from swarms.utils.history_output_formatter import HistoryOutputType, history_output_formatter
from swarms.structs.conversation import Conversation
from swarms.structs.swarm_id import swarm_id
def get_gpt_councilor_prompt() -> str: def get_gpt_councilor_prompt() -> str:
""" """
Get system prompt for GPT-5.1 councilor. Get system prompt for GPT-5.1 councilor.
Returns: Returns:
System prompt string for GPT-5.1 councilor agent. System prompt string for GPT-5.1 councilor agent.
""" """
@ -46,7 +48,7 @@ Remember: You are part of a council where multiple AI models will respond to the
def get_gemini_councilor_prompt() -> str: def get_gemini_councilor_prompt() -> str:
""" """
Get system prompt for Gemini 3 Pro councilor. Get system prompt for Gemini 3 Pro councilor.
Returns: Returns:
System prompt string for Gemini 3 Pro councilor agent. System prompt string for Gemini 3 Pro councilor agent.
""" """
@ -70,7 +72,7 @@ Remember: You are part of a council where multiple AI models will respond to the
def get_claude_councilor_prompt() -> str: def get_claude_councilor_prompt() -> str:
""" """
Get system prompt for Claude Sonnet 4.5 councilor. Get system prompt for Claude Sonnet 4.5 councilor.
Returns: Returns:
System prompt string for Claude Sonnet 4.5 councilor agent. System prompt string for Claude Sonnet 4.5 councilor agent.
""" """
@ -94,7 +96,7 @@ Remember: You are part of a council where multiple AI models will respond to the
def get_grok_councilor_prompt() -> str: def get_grok_councilor_prompt() -> str:
""" """
Get system prompt for Grok-4 councilor. Get system prompt for Grok-4 councilor.
Returns: Returns:
System prompt string for Grok-4 councilor agent. System prompt string for Grok-4 councilor agent.
""" """
@ -118,7 +120,7 @@ Remember: You are part of a council where multiple AI models will respond to the
def get_chairman_prompt() -> str: def get_chairman_prompt() -> str:
""" """
Get system prompt for the Chairman agent. Get system prompt for the Chairman agent.
Returns: Returns:
System prompt string for the Chairman agent. System prompt string for the Chairman agent.
""" """
@ -141,23 +143,27 @@ Your approach:
Remember: You have access to all original responses and all evaluations. Use this rich context to create the best possible final answer.""" Remember: You have access to all original responses and all evaluations. Use this rich context to create the best possible final answer."""
def get_evaluation_prompt(query: str, responses: Dict[str, str], evaluator_name: str) -> str: def get_evaluation_prompt(
query: str, responses: Dict[str, str], evaluator_name: str
) -> str:
""" """
Create evaluation prompt for council members to review and rank responses. Create evaluation prompt for council members to review and rank responses.
Args: Args:
query: The original user query query: The original user query
responses: Dictionary mapping anonymous IDs to response texts responses: Dictionary mapping anonymous IDs to response texts
evaluator_name: Name of the agent doing the evaluation evaluator_name: Name of the agent doing the evaluation
Returns: Returns:
Formatted evaluation prompt string Formatted evaluation prompt string
""" """
responses_text = "\n\n".join([ responses_text = "\n\n".join(
f"Response {response_id}:\n{response_text}" [
for response_id, response_text in responses.items() f"Response {response_id}:\n{response_text}"
]) for response_id, response_text in responses.items()
]
)
return f"""You are evaluating responses from your fellow LLM Council members to the following query: return f"""You are evaluating responses from your fellow LLM Council members to the following query:
QUERY: {query} QUERY: {query}
@ -191,30 +197,34 @@ def get_synthesis_prompt(
query: str, query: str,
original_responses: Dict[str, str], original_responses: Dict[str, str],
evaluations: Dict[str, str], evaluations: Dict[str, str],
id_to_member: Dict[str, str] id_to_member: Dict[str, str],
) -> str: ) -> str:
""" """
Create synthesis prompt for the Chairman. Create synthesis prompt for the Chairman.
Args: Args:
query: Original user query query: Original user query
original_responses: Dict mapping member names to their responses original_responses: Dict mapping member names to their responses
evaluations: Dict mapping evaluator names to their evaluation texts evaluations: Dict mapping evaluator names to their evaluation texts
id_to_member: Mapping from anonymous IDs to member names id_to_member: Mapping from anonymous IDs to member names
Returns: Returns:
Formatted synthesis prompt Formatted synthesis prompt
""" """
responses_section = "\n\n".join([ responses_section = "\n\n".join(
f"=== {name} ===\n{response}" [
for name, response in original_responses.items() f"=== {name} ===\n{response}"
]) for name, response in original_responses.items()
]
evaluations_section = "\n\n".join([ )
f"=== Evaluation by {name} ===\n{evaluation}"
for name, evaluation in evaluations.items() evaluations_section = "\n\n".join(
]) [
f"=== Evaluation by {name} ===\n{evaluation}"
for name, evaluation in evaluations.items()
]
)
return f"""As the Chairman of the LLM Council, synthesize the following information into a final, comprehensive answer. return f"""As the Chairman of the LLM Council, synthesize the following information into a final, comprehensive answer.
ORIGINAL QUERY: ORIGINAL QUERY:
@ -246,38 +256,46 @@ class LLMCouncil:
""" """
An LLM Council that orchestrates multiple specialized agents to collaboratively An LLM Council that orchestrates multiple specialized agents to collaboratively
answer queries through independent responses, peer review, and synthesis. answer queries through independent responses, peer review, and synthesis.
The council follows this workflow: The council follows this workflow:
1. Dispatch query to all council members in parallel 1. Dispatch query to all council members in parallel
2. Collect all responses (anonymized) 2. Collect all responses (anonymized)
3. Have each member review and rank all responses 3. Have each member review and rank all responses
4. Chairman synthesizes everything into final response 4. Chairman synthesizes everything into final response
""" """
def __init__( def __init__(
self, self,
id: str = swarm_id(),
name: str = "LLM Council",
description: str = "A collaborative council of LLM agents where each member independently answers a query, reviews and ranks anonymized peer responses, and a chairman synthesizes the best elements into a final answer.",
council_members: Optional[List[Agent]] = None, council_members: Optional[List[Agent]] = None,
chairman_model: str = "gpt-5.1", chairman_model: str = "gpt-5.1",
verbose: bool = True, verbose: bool = True,
output_type: HistoryOutputType = "dict",
): ):
""" """
Initialize the LLM Council. Initialize the LLM Council.
Args: Args:
council_members: List of Agent instances representing council members. council_members: List of Agent instances representing council members.
If None, creates default council with GPT-5.1, Gemini 3 Pro, If None, creates default council with GPT-5.1, Gemini 3 Pro,
Claude Sonnet 4.5, and Grok-4. Claude Sonnet 4.5, and Grok-4.
chairman_model: Model name for the Chairman agent that synthesizes responses. chairman_model: Model name for the Chairman agent that synthesizes responses.
verbose: Whether to print progress and intermediate results. verbose: Whether to print progress and intermediate results.
output_type: Format for the output. Options: "list", "dict", "string", "final", "json", "yaml", etc.
""" """
self.name = name
self.description = description
self.verbose = verbose self.verbose = verbose
self.output_type = output_type
# Create default council members if none provided # Create default council members if none provided
if council_members is None: if council_members is None:
self.council_members = self._create_default_council() self.council_members = self._create_default_council()
else: else:
self.council_members = council_members self.council_members = council_members
# Create Chairman agent # Create Chairman agent
self.chairman = Agent( self.chairman = Agent(
agent_name="Chairman", agent_name="Chairman",
@ -289,19 +307,25 @@ class LLMCouncil:
temperature=0.7, temperature=0.7,
) )
self.conversation = Conversation(name=f"[LLM Council] [Conversation][{name}]")
if self.verbose: if self.verbose:
print(f"🏛️ LLM Council initialized with {len(self.council_members)} members") print(
f"🏛️ LLM Council initialized with {len(self.council_members)} members"
)
for i, member in enumerate(self.council_members, 1): for i, member in enumerate(self.council_members, 1):
print(f" {i}. {member.agent_name} ({member.model_name})") print(
f" {i}. {member.agent_name} ({member.model_name})"
)
def _create_default_council(self) -> List[Agent]: def _create_default_council(self) -> List[Agent]:
""" """
Create default council members with specialized prompts and models. Create default council members with specialized prompts and models.
Returns: Returns:
List of Agent instances configured as council members. List of Agent instances configured as council members.
""" """
# GPT-5.1 Agent - Analytical and comprehensive # GPT-5.1 Agent - Analytical and comprehensive
gpt_agent = Agent( gpt_agent = Agent(
agent_name="GPT-5.1-Councilor", agent_name="GPT-5.1-Councilor",
@ -312,7 +336,7 @@ class LLMCouncil:
verbose=False, verbose=False,
temperature=0.7, temperature=0.7,
) )
# Gemini 3 Pro Agent - Concise and processed # Gemini 3 Pro Agent - Concise and processed
gemini_agent = Agent( gemini_agent = Agent(
agent_name="Gemini-3-Pro-Councilor", agent_name="Gemini-3-Pro-Councilor",
@ -323,7 +347,7 @@ class LLMCouncil:
verbose=False, verbose=False,
temperature=0.7, temperature=0.7,
) )
# Claude Sonnet 4.5 Agent - Balanced and thoughtful # Claude Sonnet 4.5 Agent - Balanced and thoughtful
claude_agent = Agent( claude_agent = Agent(
agent_name="Claude-Sonnet-4.5-Councilor", agent_name="Claude-Sonnet-4.5-Councilor",
@ -335,7 +359,7 @@ class LLMCouncil:
temperature=0.0, temperature=0.0,
top_p=None, top_p=None,
) )
# Grok-4 Agent - Creative and innovative # Grok-4 Agent - Creative and innovative
grok_agent = Agent( grok_agent = Agent(
agent_name="Grok-4-Councilor", agent_name="Grok-4-Councilor",
@ -346,114 +370,135 @@ class LLMCouncil:
verbose=False, verbose=False,
temperature=0.8, temperature=0.8,
) )
members = [gpt_agent, gemini_agent, claude_agent, grok_agent] members = [gpt_agent, gemini_agent, claude_agent, grok_agent]
return members return members
def run(self, query: str) -> Dict: def run(self, query: str):
""" """
Execute the full LLM Council workflow. Execute the full LLM Council workflow.
Args: Args:
query: The user's query to process query: The user's query to process
Returns: Returns:
Dictionary containing: Formatted output based on output_type, containing conversation history
- original_responses: Dict mapping member names to their responses with all council member responses, evaluations, and final synthesis.
- evaluations: Dict mapping evaluator names to their rankings
- final_response: The Chairman's synthesized final answer
""" """
if self.verbose: if self.verbose:
print(f"\n{'='*80}") print(f"\n{'='*80}")
print("🏛️ LLM COUNCIL SESSION") print("🏛️ LLM COUNCIL SESSION")
print("="*80) print("=" * 80)
print(f"\n📝 Query: {query}\n") print(f"\n📝 Query: {query}\n")
# Add user query to conversation
self.conversation.add(role="User", content=query)
# Step 1: Get responses from all council members in parallel # Step 1: Get responses from all council members in parallel
if self.verbose: if self.verbose:
print("📤 Dispatching query to all council members...") print("📤 Dispatching query to all council members...")
results_dict = run_agents_concurrently( results_dict = run_agents_concurrently(
self.council_members, self.council_members,
task=query, task=query,
return_agent_output_dict=True return_agent_output_dict=True,
) )
# Map results to member names # Map results to member names
original_responses = { original_responses = {
member.agent_name: response member.agent_name: response
for member, response in zip(self.council_members, for member, response in zip(
[results_dict.get(member.agent_name, "") self.council_members,
for member in self.council_members]) [
results_dict.get(member.agent_name, "")
for member in self.council_members
],
)
} }
# Add each council member's response to conversation
for member_name, response in original_responses.items():
self.conversation.add(role=member_name, content=response)
if self.verbose: if self.verbose:
print(f"✅ Received {len(original_responses)} responses\n") print(
f"✅ Received {len(original_responses)} responses\n"
)
for name, response in original_responses.items(): for name, response in original_responses.items():
print(f" {name}: {response[:100]}...") print(f" {name}: {response[:100]}...")
# Step 2: Anonymize responses for evaluation # Step 2: Anonymize responses for evaluation
# Create anonymous IDs (A, B, C, D, etc.) # Create anonymous IDs (A, B, C, D, etc.)
anonymous_ids = [chr(65 + i) for i in range(len(self.council_members))] anonymous_ids = [
chr(65 + i) for i in range(len(self.council_members))
]
random.shuffle(anonymous_ids) # Shuffle to ensure anonymity random.shuffle(anonymous_ids) # Shuffle to ensure anonymity
anonymous_responses = { anonymous_responses = {
anonymous_ids[i]: original_responses[member.agent_name] anonymous_ids[i]: original_responses[member.agent_name]
for i, member in enumerate(self.council_members) for i, member in enumerate(self.council_members)
} }
# Create mapping from anonymous ID to member name (for later reference) # Create mapping from anonymous ID to member name (for later reference)
id_to_member = { id_to_member = {
anonymous_ids[i]: member.agent_name anonymous_ids[i]: member.agent_name
for i, member in enumerate(self.council_members) for i, member in enumerate(self.council_members)
} }
if self.verbose: if self.verbose:
print("\n🔍 Council members evaluating each other's responses...") print(
"\n🔍 Council members evaluating each other's responses..."
)
# Step 3: Have each member evaluate and rank all responses concurrently # Step 3: Have each member evaluate and rank all responses concurrently
# Create evaluation tasks for each member # Create evaluation tasks for each member
evaluation_tasks = [ evaluation_tasks = [
get_evaluation_prompt(query, anonymous_responses, member.agent_name) get_evaluation_prompt(
query, anonymous_responses, member.agent_name
)
for member in self.council_members for member in self.council_members
] ]
# Run evaluations concurrently using batched_grid_agent_execution # Run evaluations concurrently using batched_grid_agent_execution
evaluation_results = batched_grid_agent_execution( evaluation_results = batched_grid_agent_execution(
self.council_members, self.council_members, evaluation_tasks
evaluation_tasks
) )
# Map results to member names # Map results to member names
evaluations = { evaluations = {
member.agent_name: evaluation_results[i] member.agent_name: evaluation_results[i]
for i, member in enumerate(self.council_members) for i, member in enumerate(self.council_members)
} }
# Add each council member's evaluation to conversation
for member_name, evaluation in evaluations.items():
self.conversation.add(
role=f"{member_name}-Evaluation", content=evaluation
)
if self.verbose: if self.verbose:
print(f"✅ Received {len(evaluations)} evaluations\n") print(f"✅ Received {len(evaluations)} evaluations\n")
# Step 4: Chairman synthesizes everything # Step 4: Chairman synthesizes everything
if self.verbose: if self.verbose:
print("👔 Chairman synthesizing final response...\n") print("👔 Chairman synthesizing final response...\n")
synthesis_prompt = get_synthesis_prompt( synthesis_prompt = get_synthesis_prompt(
query, original_responses, evaluations, id_to_member query, original_responses, evaluations, id_to_member
) )
final_response = self.chairman.run(task=synthesis_prompt) final_response = self.chairman.run(task=synthesis_prompt)
# Add chairman's final response to conversation
self.conversation.add(role="Chairman", content=final_response)
if self.verbose: if self.verbose:
print(f"{'='*80}") print(f"{'='*80}")
print("✅ FINAL RESPONSE") print("✅ FINAL RESPONSE")
print(f"{'='*80}\n") print(f"{'='*80}\n")
return {
"query": query,
"original_responses": original_responses,
"evaluations": evaluations,
"final_response": final_response,
"anonymous_mapping": id_to_member,
}
# Format and return output using history_output_formatter
return history_output_formatter(
conversation=self.conversation, type=self.output_type
)

@ -168,7 +168,9 @@ def test_error_handling():
# Test with invalid agent configuration # Test with invalid agent configuration
print("Testing invalid agent configuration...") print("Testing invalid agent configuration...")
try: try:
swarm.create_agents_from_specs({"agents": [{"agent_name": ""}]}) swarm.create_agents_from_specs(
{"agents": [{"agent_name": ""}]}
)
print( print(
"✗ Should have raised an error for empty agent configuration" "✗ Should have raised an error for empty agent configuration"
) )

@ -1,5 +1,3 @@
import pytest
from swarms.agents.i_agent import IterativeReflectiveExpansion from swarms.agents.i_agent import IterativeReflectiveExpansion

@ -3,7 +3,6 @@ import pytest
from swarms import Agent, SequentialWorkflow from swarms import Agent, SequentialWorkflow
def test_sequential_workflow_initialization_with_agents(): def test_sequential_workflow_initialization_with_agents():
"""Test SequentialWorkflow initialization with agents""" """Test SequentialWorkflow initialization with agents"""
agent1 = Agent( agent1 = Agent(

Loading…
Cancel
Save