Kye Gomez
4 days ago
parent
e5c2960912
commit
688772e99b
@ -0,0 +1,106 @@
|
||||
# LLM Council Examples
|
||||
|
||||
This page provides examples demonstrating the LLM Council pattern, inspired by Andrej Karpathy's llm-council implementation. The LLM Council uses multiple specialized AI agents that:
|
||||
|
||||
1. Each respond independently to queries
|
||||
2. Review and rank each other's anonymized responses
|
||||
3. Have a Chairman synthesize all responses into a final comprehensive answer
|
||||
|
||||
## Example Files
|
||||
|
||||
All LLM Council examples are located in the [`examples/multi_agent/llm_council_examples/`](https://github.com/kyegomez/swarms/tree/master/examples/multi_agent/llm_council_examples) directory.
|
||||
|
||||
### Marketing & Business
|
||||
|
||||
- **[marketing_strategy_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/marketing_strategy_council.py)** - Marketing strategy analysis and recommendations
|
||||
- **[business_strategy_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/business_strategy_council.py)** - Comprehensive business strategy development
|
||||
|
||||
### Finance & Investment
|
||||
|
||||
- **[finance_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/finance_analysis_council.py)** - Financial analysis and investment recommendations
|
||||
- **[etf_stock_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/etf_stock_analysis_council.py)** - ETF and stock analysis with portfolio recommendations
|
||||
|
||||
### Medical & Healthcare
|
||||
|
||||
- **[medical_treatment_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/medical_treatment_council.py)** - Medical treatment recommendations and care plans
|
||||
- **[medical_diagnosis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/medical_diagnosis_council.py)** - Diagnostic analysis based on symptoms
|
||||
|
||||
### Technology & Research
|
||||
|
||||
- **[technology_assessment_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/technology_assessment_council.py)** - Technology evaluation and implementation strategy
|
||||
- **[research_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/research_analysis_council.py)** - Comprehensive research analysis on complex topics
|
||||
|
||||
### Legal
|
||||
|
||||
- **[legal_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/legal_analysis_council.py)** - Legal implications and compliance analysis
|
||||
|
||||
## Basic Usage Pattern
|
||||
|
||||
All examples follow the same pattern:
|
||||
|
||||
```python
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Run a query
|
||||
result = council.run("Your query here")
|
||||
|
||||
# Access results
|
||||
print(result["final_response"]) # Chairman's synthesized answer
|
||||
print(result["original_responses"]) # Individual member responses
|
||||
print(result["evaluations"]) # How members ranked each other
|
||||
```
|
||||
|
||||
## Running Examples
|
||||
|
||||
Run any example directly:
|
||||
|
||||
```bash
|
||||
python examples/multi_agent/llm_council_examples/marketing_strategy_council.py
|
||||
python examples/multi_agent/llm_council_examples/finance_analysis_council.py
|
||||
python examples/multi_agent/llm_council_examples/medical_diagnosis_council.py
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Multiple Perspectives**: Each council member (GPT-5.1, Gemini, Claude, Grok) provides unique insights
|
||||
- **Peer Review**: Members evaluate and rank each other's responses anonymously
|
||||
- **Synthesis**: Chairman combines the best elements from all responses
|
||||
- **Transparency**: See both individual responses and evaluation rankings
|
||||
|
||||
## Council Members
|
||||
|
||||
The default council consists of:
|
||||
- **GPT-5.1-Councilor**: Analytical and comprehensive
|
||||
- **Gemini-3-Pro-Councilor**: Concise and well-processed
|
||||
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced
|
||||
- **Grok-4-Councilor**: Creative and innovative
|
||||
|
||||
## Customization
|
||||
|
||||
You can create custom council members:
|
||||
|
||||
```python
|
||||
from swarms import Agent
|
||||
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||
|
||||
custom_agent = Agent(
|
||||
agent_name="Custom-Councilor",
|
||||
system_prompt=get_gpt_councilor_prompt(),
|
||||
model_name="gpt-4.1",
|
||||
max_loops=1,
|
||||
)
|
||||
|
||||
council = LLMCouncil(
|
||||
council_members=[custom_agent, ...],
|
||||
chairman_model="gpt-5.1",
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
## Documentation
|
||||
|
||||
For complete API reference and detailed documentation, see the [LLM Council Reference Documentation](../swarms/structs/llm_council.md).
|
||||
|
||||
@ -0,0 +1,453 @@
|
||||
# LLM Council Class Documentation
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[User Query] --> B[LLM Council Initialization]
|
||||
B --> C{Council Members Provided?}
|
||||
C -->|No| D[Create Default Council]
|
||||
C -->|Yes| E[Use Provided Members]
|
||||
D --> F[Step 1: Parallel Response Generation]
|
||||
E --> F
|
||||
|
||||
subgraph "Default Council Members"
|
||||
G1[GPT-5.1-Councilor<br/>Analytical & Comprehensive]
|
||||
G2[Gemini-3-Pro-Councilor<br/>Concise & Structured]
|
||||
G3[Claude-Sonnet-4.5-Councilor<br/>Thoughtful & Balanced]
|
||||
G4[Grok-4-Councilor<br/>Creative & Innovative]
|
||||
end
|
||||
|
||||
F --> G1
|
||||
F --> G2
|
||||
F --> G3
|
||||
F --> G4
|
||||
|
||||
G1 --> H[Collect All Responses]
|
||||
G2 --> H
|
||||
G3 --> H
|
||||
G4 --> H
|
||||
|
||||
H --> I[Step 2: Anonymize Responses]
|
||||
I --> J[Assign Anonymous IDs: A, B, C, D...]
|
||||
|
||||
J --> K[Step 3: Parallel Evaluation]
|
||||
|
||||
subgraph "Evaluation Phase"
|
||||
K --> L1[Member 1 Evaluates All]
|
||||
K --> L2[Member 2 Evaluates All]
|
||||
K --> L3[Member 3 Evaluates All]
|
||||
K --> L4[Member 4 Evaluates All]
|
||||
end
|
||||
|
||||
L1 --> M[Collect Evaluations & Rankings]
|
||||
L2 --> M
|
||||
L3 --> M
|
||||
L4 --> M
|
||||
|
||||
M --> N[Step 4: Chairman Synthesis]
|
||||
N --> O[Chairman Agent]
|
||||
O --> P[Final Synthesized Response]
|
||||
|
||||
P --> Q[Return Results Dictionary]
|
||||
|
||||
style A fill:#e1f5ff
|
||||
style P fill:#c8e6c9
|
||||
style Q fill:#c8e6c9
|
||||
style O fill:#fff9c4
|
||||
```
|
||||
|
||||
The `LLMCouncil` class orchestrates multiple specialized LLM agents to collaboratively answer queries through a structured peer review and synthesis process. Inspired by Andrej Karpathy's llm-council implementation, this architecture demonstrates how different models evaluate and rank each other's work, often selecting responses from other models as superior to their own.
|
||||
|
||||
## Workflow Overview
|
||||
|
||||
The LLM Council follows a four-step process:
|
||||
|
||||
1. **Parallel Response Generation**: All council members independently respond to the user query
|
||||
2. **Anonymization**: Responses are anonymized with random IDs (A, B, C, D, etc.) to ensure objective evaluation
|
||||
3. **Peer Review**: Each member evaluates and ranks all responses (including potentially their own)
|
||||
4. **Synthesis**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer
|
||||
|
||||
## Class Definition
|
||||
|
||||
### LLMCouncil
|
||||
|
||||
```python
|
||||
class LLMCouncil:
|
||||
```
|
||||
|
||||
### Attributes
|
||||
|
||||
| Attribute | Type | Description | Default |
|
||||
|-----------|------|-------------|---------|
|
||||
| `council_members` | `List[Agent]` | List of Agent instances representing council members | `None` (creates default council) |
|
||||
| `chairman` | `Agent` | The Chairman agent responsible for synthesizing responses | Created during initialization |
|
||||
| `verbose` | `bool` | Whether to print progress and intermediate results | `True` |
|
||||
|
||||
## Methods
|
||||
|
||||
### `__init__`
|
||||
|
||||
Initializes the LLM Council with council members and a Chairman agent.
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `council_members` | `Optional[List[Agent]]` | `None` | List of Agent instances representing council members. If `None`, creates default council with GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok-4. |
|
||||
| `chairman_model` | `str` | `"gpt-5.1"` | Model name for the Chairman agent that synthesizes responses. |
|
||||
| `verbose` | `bool` | `True` | Whether to print progress and intermediate results. |
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `LLMCouncil` | Initialized LLM Council instance. |
|
||||
|
||||
#### Description
|
||||
|
||||
Creates an LLM Council instance with specialized council members. If no members are provided, it creates a default council consisting of:
|
||||
- **GPT-5.1-Councilor**: Analytical and comprehensive responses
|
||||
- **Gemini-3-Pro-Councilor**: Concise and well-processed responses
|
||||
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced responses
|
||||
- **Grok-4-Councilor**: Creative and innovative responses
|
||||
|
||||
The Chairman agent is automatically created with a specialized prompt for synthesizing responses.
|
||||
|
||||
#### Example Usage
|
||||
|
||||
```python
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create council with default members
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Create council with custom members
|
||||
from swarms import Agent
|
||||
custom_members = [
|
||||
Agent(agent_name="Expert-1", model_name="gpt-4", max_loops=1),
|
||||
Agent(agent_name="Expert-2", model_name="claude-3-opus", max_loops=1),
|
||||
]
|
||||
council = LLMCouncil(
|
||||
council_members=custom_members,
|
||||
chairman_model="gpt-4",
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `run`
|
||||
|
||||
Executes the full LLM Council workflow: parallel responses, anonymization, peer review, and synthesis.
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `query` | `str` | Required | The user's query to process through the council. |
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `Dict` | Dictionary containing the following keys: |
|
||||
|
||||
#### Return Dictionary Structure
|
||||
|
||||
| Key | Type | Description |
|
||||
|-----|------|-------------|
|
||||
| `query` | `str` | The original user query. |
|
||||
| `original_responses` | `Dict[str, str]` | Dictionary mapping council member names to their original responses. |
|
||||
| `evaluations` | `Dict[str, str]` | Dictionary mapping evaluator names to their evaluation texts (rankings and reasoning). |
|
||||
| `final_response` | `str` | The Chairman's synthesized final answer combining all perspectives. |
|
||||
| `anonymous_mapping` | `Dict[str, str]` | Mapping from anonymous IDs (A, B, C, D) to member names for reference. |
|
||||
|
||||
#### Description
|
||||
|
||||
Executes the complete LLM Council workflow:
|
||||
|
||||
1. **Dispatch Phase**: Sends the query to all council members in parallel using `run_agents_concurrently`
|
||||
2. **Collection Phase**: Collects all responses and maps them to member names
|
||||
3. **Anonymization Phase**: Creates anonymous IDs (A, B, C, D, etc.) and shuffles them to ensure anonymity
|
||||
4. **Evaluation Phase**: Each member evaluates and ranks all anonymized responses using `batched_grid_agent_execution`
|
||||
5. **Synthesis Phase**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer
|
||||
|
||||
The method provides verbose output by default, showing progress at each stage.
|
||||
|
||||
#### Example Usage
|
||||
|
||||
```python
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||
|
||||
result = council.run(query)
|
||||
|
||||
# Access the final synthesized response
|
||||
print(result["final_response"])
|
||||
|
||||
# Access individual member responses
|
||||
for name, response in result["original_responses"].items():
|
||||
print(f"{name}: {response[:200]}...")
|
||||
|
||||
# Access evaluation rankings
|
||||
for evaluator, evaluation in result["evaluations"].items():
|
||||
print(f"{evaluator} evaluation:\n{evaluation[:300]}...")
|
||||
|
||||
# Check anonymous mapping
|
||||
print("Anonymous IDs:", result["anonymous_mapping"])
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### `_create_default_council`
|
||||
|
||||
Creates default council members with specialized prompts and models.
|
||||
|
||||
#### Parameters
|
||||
|
||||
None (internal method).
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `List[Agent]` | List of Agent instances configured as council members. |
|
||||
|
||||
#### Description
|
||||
|
||||
Internal method that creates the default council configuration with four specialized agents:
|
||||
|
||||
- **GPT-5.1-Councilor** (`model_name="gpt-5.1"`): Analytical and comprehensive, temperature=0.7
|
||||
- **Gemini-3-Pro-Councilor** (`model_name="gemini-2.5-flash"`): Concise and structured, temperature=0.7
|
||||
- **Claude-Sonnet-4.5-Councilor** (`model_name="anthropic/claude-sonnet-4-5"`): Thoughtful and balanced, temperature=0.0
|
||||
- **Grok-4-Councilor** (`model_name="x-ai/grok-4"`): Creative and innovative, temperature=0.8
|
||||
|
||||
Each agent is configured with:
|
||||
- Specialized system prompts matching their role
|
||||
- `max_loops=1` for single-response generation
|
||||
- `verbose=False` to reduce noise during parallel execution
|
||||
- Appropriate temperature settings for their style
|
||||
|
||||
---
|
||||
|
||||
## Helper Functions
|
||||
|
||||
### `get_gpt_councilor_prompt()`
|
||||
|
||||
Returns the system prompt for GPT-5.1 councilor agent.
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | System prompt string emphasizing analytical thinking and comprehensive coverage. |
|
||||
|
||||
---
|
||||
|
||||
### `get_gemini_councilor_prompt()`
|
||||
|
||||
Returns the system prompt for Gemini 3 Pro councilor agent.
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | System prompt string emphasizing concise, well-processed, and structured responses. |
|
||||
|
||||
---
|
||||
|
||||
### `get_claude_councilor_prompt()`
|
||||
|
||||
Returns the system prompt for Claude Sonnet 4.5 councilor agent.
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | System prompt string emphasizing thoughtful, balanced, and nuanced responses. |
|
||||
|
||||
---
|
||||
|
||||
### `get_grok_councilor_prompt()`
|
||||
|
||||
Returns the system prompt for Grok-4 councilor agent.
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | System prompt string emphasizing creative, innovative, and unique perspectives. |
|
||||
|
||||
---
|
||||
|
||||
### `get_chairman_prompt()`
|
||||
|
||||
Returns the system prompt for the Chairman agent.
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | System prompt string for synthesizing responses and evaluations into a final answer. |
|
||||
|
||||
---
|
||||
|
||||
### `get_evaluation_prompt(query, responses, evaluator_name)`
|
||||
|
||||
Creates evaluation prompt for council members to review and rank responses.
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `query` | `str` | The original user query. |
|
||||
| `responses` | `Dict[str, str]` | Dictionary mapping anonymous IDs to response texts. |
|
||||
| `evaluator_name` | `str` | Name of the agent doing the evaluation. |
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | Formatted evaluation prompt string with instructions for ranking responses. |
|
||||
|
||||
---
|
||||
|
||||
### `get_synthesis_prompt(query, original_responses, evaluations, id_to_member)`
|
||||
|
||||
Creates synthesis prompt for the Chairman.
|
||||
|
||||
#### Parameters
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|-----------|------|-------------|
|
||||
| `query` | `str` | Original user query. |
|
||||
| `original_responses` | `Dict[str, str]` | Dictionary mapping member names to their responses. |
|
||||
| `evaluations` | `Dict[str, str]` | Dictionary mapping evaluator names to their evaluation texts. |
|
||||
| `id_to_member` | `Dict[str, str]` | Mapping from anonymous IDs to member names. |
|
||||
|
||||
#### Returns
|
||||
|
||||
| Type | Description |
|
||||
|------|-------------|
|
||||
| `str` | Formatted synthesis prompt for the Chairman agent. |
|
||||
|
||||
---
|
||||
|
||||
## Use Cases
|
||||
|
||||
The LLM Council is ideal for scenarios requiring:
|
||||
|
||||
- **Multi-perspective Analysis**: When you need diverse viewpoints on complex topics
|
||||
- **Quality Assurance**: When peer review and ranking can improve response quality
|
||||
- **Transparent Decision Making**: When you want to see how different models evaluate each other
|
||||
- **Synthesis of Expertise**: When combining multiple specialized perspectives is valuable
|
||||
|
||||
### Common Applications
|
||||
|
||||
- **Medical Diagnosis**: Multiple medical AI agents provide diagnoses, evaluate each other, and synthesize recommendations
|
||||
- **Financial Analysis**: Different financial experts analyze investments and rank each other's assessments
|
||||
- **Legal Analysis**: Multiple legal perspectives evaluate compliance and risk
|
||||
- **Business Strategy**: Diverse strategic viewpoints are synthesized into comprehensive plans
|
||||
- **Research Analysis**: Multiple research perspectives are combined for thorough analysis
|
||||
|
||||
## Examples
|
||||
|
||||
For comprehensive examples demonstrating various use cases, see the [LLM Council Examples](../../../examples/multi_agent/llm_council_examples/) directory:
|
||||
|
||||
- **Medical**: `medical_diagnosis_council.py`, `medical_treatment_council.py`
|
||||
- **Finance**: `finance_analysis_council.py`, `etf_stock_analysis_council.py`
|
||||
- **Business**: `business_strategy_council.py`, `marketing_strategy_council.py`
|
||||
- **Technology**: `technology_assessment_council.py`, `research_analysis_council.py`
|
||||
- **Legal**: `legal_analysis_council.py`
|
||||
|
||||
### Quick Start Example
|
||||
|
||||
```python
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Example query
|
||||
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
# Optionally print evaluations
|
||||
print("\n\n" + "="*80)
|
||||
print("EVALUATIONS")
|
||||
print("="*80)
|
||||
for name, evaluation in result["evaluations"].items():
|
||||
print(f"\n{name}:")
|
||||
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation)
|
||||
```
|
||||
|
||||
## Customization
|
||||
|
||||
### Creating Custom Council Members
|
||||
|
||||
You can create custom council members with specialized roles:
|
||||
|
||||
```python
|
||||
from swarms import Agent
|
||||
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||
|
||||
# Create custom councilor
|
||||
custom_agent = Agent(
|
||||
agent_name="Domain-Expert-Councilor",
|
||||
agent_description="Specialized domain expert for specific analysis",
|
||||
system_prompt=get_gpt_councilor_prompt(), # Or create custom prompt
|
||||
model_name="gpt-4",
|
||||
max_loops=1,
|
||||
verbose=False,
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
# Create council with custom members
|
||||
council = LLMCouncil(
|
||||
council_members=[custom_agent, ...], # Add your custom agents
|
||||
chairman_model="gpt-4",
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
### Custom Chairman Model
|
||||
|
||||
You can specify a different model for the Chairman:
|
||||
|
||||
```python
|
||||
council = LLMCouncil(
|
||||
chairman_model="claude-3-opus", # Use Claude as Chairman
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
## Architecture Benefits
|
||||
|
||||
1. **Diversity**: Multiple models provide varied perspectives and approaches
|
||||
2. **Quality Control**: Peer review ensures responses are evaluated objectively
|
||||
3. **Synthesis**: Chairman combines the best elements from all responses
|
||||
4. **Transparency**: Full visibility into individual responses and evaluation rankings
|
||||
5. **Scalability**: Easy to add or remove council members
|
||||
6. **Flexibility**: Supports custom agents and models
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
- **Parallel Execution**: Both response generation and evaluation phases run in parallel for efficiency
|
||||
- **Anonymization**: Responses are anonymized to prevent bias in evaluation
|
||||
- **Model Selection**: Different models can be used for different roles based on their strengths
|
||||
- **Verbose Mode**: Can be disabled for production use to reduce output
|
||||
|
||||
## Related Documentation
|
||||
|
||||
- [Multi-Agent Architectures Overview](overview.md)
|
||||
- [Council of Judges](council_of_judges.md) - Similar peer review pattern
|
||||
- [Agent Class Reference](agent.md) - Understanding individual agents
|
||||
- [Multi-Agent Execution Utilities](various_execution_methods.md) - Underlying execution methods
|
||||
|
||||
@ -0,0 +1,95 @@
|
||||
# LLM Council Examples
|
||||
|
||||
This directory contains examples demonstrating the LLM Council pattern, inspired by Andrej Karpathy's llm-council implementation. The LLM Council uses multiple specialized AI agents that:
|
||||
|
||||
1. Each respond independently to queries
|
||||
2. Review and rank each other's anonymized responses
|
||||
3. Have a Chairman synthesize all responses into a final comprehensive answer
|
||||
|
||||
## Examples
|
||||
|
||||
### Marketing & Business
|
||||
- **marketing_strategy_council.py** - Marketing strategy analysis and recommendations
|
||||
- **business_strategy_council.py** - Comprehensive business strategy development
|
||||
|
||||
### Finance & Investment
|
||||
- **finance_analysis_council.py** - Financial analysis and investment recommendations
|
||||
- **etf_stock_analysis_council.py** - ETF and stock analysis with portfolio recommendations
|
||||
|
||||
### Medical & Healthcare
|
||||
- **medical_treatment_council.py** - Medical treatment recommendations and care plans
|
||||
- **medical_diagnosis_council.py** - Diagnostic analysis based on symptoms
|
||||
|
||||
### Technology & Research
|
||||
- **technology_assessment_council.py** - Technology evaluation and implementation strategy
|
||||
- **research_analysis_council.py** - Comprehensive research analysis on complex topics
|
||||
|
||||
### Legal
|
||||
- **legal_analysis_council.py** - Legal implications and compliance analysis
|
||||
|
||||
## Usage
|
||||
|
||||
Each example follows the same pattern:
|
||||
|
||||
```python
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Run a query
|
||||
result = council.run("Your query here")
|
||||
|
||||
# Access results
|
||||
print(result["final_response"]) # Chairman's synthesized answer
|
||||
print(result["original_responses"]) # Individual member responses
|
||||
print(result["evaluations"]) # How members ranked each other
|
||||
```
|
||||
|
||||
## Running Examples
|
||||
|
||||
Run any example directly:
|
||||
|
||||
```bash
|
||||
python examples/multi_agent/llm_council_examples/marketing_strategy_council.py
|
||||
python examples/multi_agent/llm_council_examples/finance_analysis_council.py
|
||||
python examples/multi_agent/llm_council_examples/medical_diagnosis_council.py
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
- **Multiple Perspectives**: Each council member (GPT-5.1, Gemini, Claude, Grok) provides unique insights
|
||||
- **Peer Review**: Members evaluate and rank each other's responses anonymously
|
||||
- **Synthesis**: Chairman combines the best elements from all responses
|
||||
- **Transparency**: See both individual responses and evaluation rankings
|
||||
|
||||
## Council Members
|
||||
|
||||
The default council consists of:
|
||||
- **GPT-5.1-Councilor**: Analytical and comprehensive
|
||||
- **Gemini-3-Pro-Councilor**: Concise and well-processed
|
||||
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced
|
||||
- **Grok-4-Councilor**: Creative and innovative
|
||||
|
||||
## Customization
|
||||
|
||||
You can create custom council members:
|
||||
|
||||
```python
|
||||
from swarms import Agent
|
||||
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||
|
||||
custom_agent = Agent(
|
||||
agent_name="Custom-Councilor",
|
||||
system_prompt=get_gpt_councilor_prompt(),
|
||||
model_name="gpt-4.1",
|
||||
max_loops=1,
|
||||
)
|
||||
|
||||
council = LLMCouncil(
|
||||
council_members=[custom_agent, ...],
|
||||
chairman_model="gpt-5.1",
|
||||
verbose=True
|
||||
)
|
||||
```
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
"""
|
||||
LLM Council Example: Business Strategy Development
|
||||
|
||||
This example demonstrates using the LLM Council to develop comprehensive
|
||||
business strategies for new ventures.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Business strategy query
|
||||
query = """
|
||||
A tech startup wants to launch an AI-powered personal finance app targeting
|
||||
millennials and Gen Z. Develop a comprehensive business strategy including:
|
||||
1. Market opportunity and competitive landscape analysis
|
||||
2. Product positioning and unique value proposition
|
||||
3. Go-to-market strategy and customer acquisition plan
|
||||
4. Revenue model and pricing strategy
|
||||
5. Key partnerships and distribution channels
|
||||
6. Resource requirements and funding needs
|
||||
7. Risk assessment and mitigation strategies
|
||||
8. Success metrics and KPIs for first 12 months
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,30 @@
|
||||
"""
|
||||
LLM Council Example: ETF Stock Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to analyze ETF holdings
|
||||
and provide stock investment recommendations.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# ETF and stock analysis query
|
||||
query = """
|
||||
Analyze the top energy ETFs (including nuclear, solar, gas, and renewable energy)
|
||||
and provide:
|
||||
1. Top 5 best-performing energy stocks across all energy sectors
|
||||
2. ETF recommendations for diversified energy exposure
|
||||
3. Risk-return profiles for each recommendation
|
||||
4. Current market conditions affecting energy investments
|
||||
5. Allocation strategy for a $100,000 portfolio
|
||||
6. Key metrics to track for each investment
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,30 @@
|
||||
"""
|
||||
LLM Council Example: Financial Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to provide comprehensive
|
||||
financial analysis and investment recommendations.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Financial analysis query
|
||||
query = """
|
||||
Provide a comprehensive financial analysis for investing in emerging markets
|
||||
technology ETFs. Include:
|
||||
1. Risk assessment and volatility analysis
|
||||
2. Historical performance trends
|
||||
3. Sector composition and diversification benefits
|
||||
4. Comparison with developed market tech ETFs
|
||||
5. Recommended allocation percentage for a moderate risk portfolio
|
||||
6. Key factors to monitor going forward
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
"""
|
||||
LLM Council Example: Legal Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to analyze legal scenarios
|
||||
and provide comprehensive legal insights.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Legal analysis query
|
||||
query = """
|
||||
A startup is considering using AI-generated content for their marketing materials.
|
||||
Analyze the legal implications including:
|
||||
1. Intellectual property rights and ownership of AI-generated content
|
||||
2. Copyright and trademark considerations
|
||||
3. Liability for AI-generated content that may be inaccurate or misleading
|
||||
4. Compliance with advertising regulations (FTC, FDA, etc.)
|
||||
5. Data privacy implications if using customer data to train models
|
||||
6. Contractual considerations with AI service providers
|
||||
7. Risk mitigation strategies
|
||||
8. Best practices for legal compliance
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,29 @@
|
||||
"""
|
||||
LLM Council Example: Marketing Strategy Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to analyze and develop
|
||||
comprehensive marketing strategies by leveraging multiple AI perspectives.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Marketing strategy query
|
||||
query = """
|
||||
Analyze the marketing strategy for a new sustainable energy startup launching
|
||||
a solar panel subscription service. Provide recommendations on:
|
||||
1. Target audience segmentation
|
||||
2. Key messaging and value propositions
|
||||
3. Marketing channels and budget allocation
|
||||
4. Competitive positioning
|
||||
5. Launch timeline and milestones
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,37 @@
|
||||
"""
|
||||
LLM Council Example: Medical Diagnosis Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to analyze symptoms
|
||||
and provide diagnostic insights.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Medical diagnosis query
|
||||
query = """
|
||||
A 35-year-old patient presents with:
|
||||
- Persistent fatigue for 3 months
|
||||
- Unexplained weight loss (15 lbs)
|
||||
- Night sweats
|
||||
- Intermittent low-grade fever
|
||||
- Swollen lymph nodes in neck and armpits
|
||||
- Recent blood work shows elevated ESR and CRP
|
||||
|
||||
Provide:
|
||||
1. Differential diagnosis with most likely conditions ranked
|
||||
2. Additional diagnostic tests needed to confirm
|
||||
3. Red flag symptoms requiring immediate attention
|
||||
4. Possible causes and risk factors
|
||||
5. Recommended next steps for the patient
|
||||
6. When to seek emergency care
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,31 @@
|
||||
"""
|
||||
LLM Council Example: Medical Treatment Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to analyze medical treatments
|
||||
and provide comprehensive treatment recommendations.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Medical treatment query
|
||||
query = """
|
||||
A 45-year-old patient with Type 2 diabetes, hypertension, and early-stage
|
||||
kidney disease needs treatment recommendations. Provide:
|
||||
1. Comprehensive treatment plan addressing all conditions
|
||||
2. Medication options with pros/cons for each condition
|
||||
3. Lifestyle modifications and their expected impact
|
||||
4. Monitoring schedule and key metrics to track
|
||||
5. Potential drug interactions and contraindications
|
||||
6. Expected outcomes and timeline for improvement
|
||||
7. When to consider specialist referrals
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
"""
|
||||
LLM Council Example: Research Analysis
|
||||
|
||||
This example demonstrates using the LLM Council to conduct comprehensive
|
||||
research analysis on complex topics.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Research analysis query
|
||||
query = """
|
||||
Conduct a comprehensive analysis of the potential impact of climate change
|
||||
on global food security over the next 20 years. Include:
|
||||
1. Key climate factors affecting agriculture (temperature, precipitation, extreme weather)
|
||||
2. Regional vulnerabilities and impacts on major food-producing regions
|
||||
3. Crop yield projections and food availability scenarios
|
||||
4. Economic implications and food price volatility
|
||||
5. Adaptation strategies and technological solutions
|
||||
6. Policy recommendations for governments and international organizations
|
||||
7. Role of innovation in agriculture (precision farming, GMOs, vertical farming)
|
||||
8. Social and geopolitical implications of food insecurity
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,32 @@
|
||||
"""
|
||||
LLM Council Example: Technology Assessment
|
||||
|
||||
This example demonstrates using the LLM Council to assess emerging technologies
|
||||
and their business implications.
|
||||
"""
|
||||
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Technology assessment query
|
||||
query = """
|
||||
Evaluate the business potential and implementation strategy for integrating
|
||||
quantum computing capabilities into a financial services company. Consider:
|
||||
1. Current state of quantum computing technology
|
||||
2. Specific use cases in financial services (risk modeling, portfolio optimization, fraud detection)
|
||||
3. Competitive advantages and potential ROI
|
||||
4. Implementation timeline and resource requirements
|
||||
5. Technical challenges and limitations
|
||||
6. Risk factors and mitigation strategies
|
||||
7. Partnership opportunities with quantum computing providers
|
||||
8. Expected timeline for practical business value
|
||||
"""
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
@ -0,0 +1,23 @@
|
||||
from swarms.structs.llm_council import LLMCouncil
|
||||
|
||||
# Example usage of the LLM Council without a function:
|
||||
# Create the council
|
||||
council = LLMCouncil(verbose=True)
|
||||
|
||||
# Example query
|
||||
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||
|
||||
# Run the council
|
||||
result = council.run(query)
|
||||
|
||||
# Print final response
|
||||
print(result["final_response"])
|
||||
|
||||
# Optionally print evaluations
|
||||
print("\n\n" + "="*80)
|
||||
print("EVALUATIONS")
|
||||
print("="*80)
|
||||
for name, evaluation in result["evaluations"].items():
|
||||
print(f"\n{name}:")
|
||||
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation)
|
||||
|
||||
@ -0,0 +1,459 @@
|
||||
"""
|
||||
LLM Council - A Swarms implementation inspired by Andrej Karpathy's llm-council.
|
||||
|
||||
This implementation creates a council of specialized LLM agents that:
|
||||
1. Each agent responds to the user query independently
|
||||
2. All agents review and rank each other's (anonymized) responses
|
||||
3. A Chairman LLM synthesizes all responses and rankings into a final answer
|
||||
|
||||
The council demonstrates how different models evaluate and rank each other's work,
|
||||
often selecting responses from other models as superior to their own.
|
||||
"""
|
||||
|
||||
from typing import Dict, List, Optional
|
||||
import random
|
||||
from swarms import Agent
|
||||
from swarms.structs.multi_agent_exec import (
|
||||
run_agents_concurrently,
|
||||
batched_grid_agent_execution,
|
||||
)
|
||||
|
||||
|
||||
def get_gpt_councilor_prompt() -> str:
|
||||
"""
|
||||
Get system prompt for GPT-5.1 councilor.
|
||||
|
||||
Returns:
|
||||
System prompt string for GPT-5.1 councilor agent.
|
||||
"""
|
||||
return """You are a member of the LLM Council, representing GPT-5.1. Your role is to provide comprehensive, analytical, and thorough responses to user queries.
|
||||
|
||||
Your strengths:
|
||||
- Deep analytical thinking and comprehensive coverage
|
||||
- Ability to break down complex topics into detailed components
|
||||
- Thorough exploration of multiple perspectives
|
||||
- Rich contextual understanding
|
||||
|
||||
Your approach:
|
||||
- Provide detailed, well-structured responses
|
||||
- Include relevant context and background information
|
||||
- Consider multiple angles and perspectives
|
||||
- Be thorough but clear in your explanations
|
||||
|
||||
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on quality, depth, and clarity."""
|
||||
|
||||
|
||||
def get_gemini_councilor_prompt() -> str:
|
||||
"""
|
||||
Get system prompt for Gemini 3 Pro councilor.
|
||||
|
||||
Returns:
|
||||
System prompt string for Gemini 3 Pro councilor agent.
|
||||
"""
|
||||
return """You are a member of the LLM Council, representing Gemini 3 Pro. Your role is to provide concise, well-processed, and structured responses to user queries.
|
||||
|
||||
Your strengths:
|
||||
- Clear and structured communication
|
||||
- Efficient information processing
|
||||
- Condensed yet comprehensive responses
|
||||
- Well-organized presentation
|
||||
|
||||
Your approach:
|
||||
- Provide concise but complete answers
|
||||
- Structure information clearly and logically
|
||||
- Focus on key points without unnecessary verbosity
|
||||
- Present information in an easily digestible format
|
||||
|
||||
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on clarity, structure, and efficiency."""
|
||||
|
||||
|
||||
def get_claude_councilor_prompt() -> str:
|
||||
"""
|
||||
Get system prompt for Claude Sonnet 4.5 councilor.
|
||||
|
||||
Returns:
|
||||
System prompt string for Claude Sonnet 4.5 councilor agent.
|
||||
"""
|
||||
return """You are a member of the LLM Council, representing Claude Sonnet 4.5. Your role is to provide thoughtful, balanced, and nuanced responses to user queries.
|
||||
|
||||
Your strengths:
|
||||
- Nuanced understanding and balanced perspectives
|
||||
- Thoughtful consideration of trade-offs
|
||||
- Clear reasoning and logical structure
|
||||
- Ethical and responsible analysis
|
||||
|
||||
Your approach:
|
||||
- Provide balanced, well-reasoned responses
|
||||
- Consider multiple viewpoints and implications
|
||||
- Be thoughtful about potential limitations or edge cases
|
||||
- Maintain clarity while showing depth of thought
|
||||
|
||||
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on thoughtfulness, balance, and nuanced reasoning."""
|
||||
|
||||
|
||||
def get_grok_councilor_prompt() -> str:
|
||||
"""
|
||||
Get system prompt for Grok-4 councilor.
|
||||
|
||||
Returns:
|
||||
System prompt string for Grok-4 councilor agent.
|
||||
"""
|
||||
return """You are a member of the LLM Council, representing Grok-4. Your role is to provide creative, innovative, and unique perspectives on user queries.
|
||||
|
||||
Your strengths:
|
||||
- Creative problem-solving and innovative thinking
|
||||
- Unique perspectives and out-of-the-box approaches
|
||||
- Engaging and dynamic communication style
|
||||
- Ability to connect seemingly unrelated concepts
|
||||
|
||||
Your approach:
|
||||
- Provide creative and innovative responses
|
||||
- Offer unique perspectives and fresh insights
|
||||
- Be engaging and dynamic in your communication
|
||||
- Think creatively while maintaining accuracy
|
||||
|
||||
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on creativity, innovation, and unique insights."""
|
||||
|
||||
|
||||
def get_chairman_prompt() -> str:
|
||||
"""
|
||||
Get system prompt for the Chairman agent.
|
||||
|
||||
Returns:
|
||||
System prompt string for the Chairman agent.
|
||||
"""
|
||||
return """You are the Chairman of the LLM Council. Your role is to synthesize responses from all council members along with their evaluations and rankings into a final, comprehensive answer.
|
||||
|
||||
Your responsibilities:
|
||||
1. Review all council member responses to the user's query
|
||||
2. Consider the rankings and evaluations provided by each council member
|
||||
3. Synthesize the best elements from all responses
|
||||
4. Create a final, comprehensive answer that incorporates the strengths of different approaches
|
||||
5. Provide transparency about which perspectives influenced the final answer
|
||||
|
||||
Your approach:
|
||||
- Synthesize rather than simply aggregate
|
||||
- Identify the strongest elements from each response
|
||||
- Create a cohesive final answer that benefits from multiple perspectives
|
||||
- Acknowledge the diversity of approaches taken by council members
|
||||
- Provide a balanced, comprehensive response that serves the user's needs
|
||||
|
||||
Remember: You have access to all original responses and all evaluations. Use this rich context to create the best possible final answer."""
|
||||
|
||||
|
||||
def get_evaluation_prompt(query: str, responses: Dict[str, str], evaluator_name: str) -> str:
|
||||
"""
|
||||
Create evaluation prompt for council members to review and rank responses.
|
||||
|
||||
Args:
|
||||
query: The original user query
|
||||
responses: Dictionary mapping anonymous IDs to response texts
|
||||
evaluator_name: Name of the agent doing the evaluation
|
||||
|
||||
Returns:
|
||||
Formatted evaluation prompt string
|
||||
"""
|
||||
responses_text = "\n\n".join([
|
||||
f"Response {response_id}:\n{response_text}"
|
||||
for response_id, response_text in responses.items()
|
||||
])
|
||||
|
||||
return f"""You are evaluating responses from your fellow LLM Council members to the following query:
|
||||
|
||||
QUERY: {query}
|
||||
|
||||
Below are the anonymized responses from all council members (including potentially your own):
|
||||
|
||||
{responses_text}
|
||||
|
||||
Your task:
|
||||
1. Carefully read and analyze each response
|
||||
2. Evaluate the quality, accuracy, completeness, and usefulness of each response
|
||||
3. Rank the responses from best to worst (1 = best, {len(responses)} = worst)
|
||||
4. Provide brief reasoning for your rankings
|
||||
5. Be honest and objective - you may find another model's response superior to your own
|
||||
|
||||
Format your evaluation as follows:
|
||||
|
||||
RANKINGS:
|
||||
1. Response [ID]: [Brief reason why this is the best]
|
||||
2. Response [ID]: [Brief reason]
|
||||
...
|
||||
{len(responses)}. Response [ID]: [Brief reason why this ranks lowest]
|
||||
|
||||
ADDITIONAL OBSERVATIONS:
|
||||
[Any additional insights about the responses, common themes, strengths/weaknesses, etc.]
|
||||
|
||||
Remember: The goal is honest, objective evaluation. If another model's response is genuinely better, acknowledge it."""
|
||||
|
||||
|
||||
def get_synthesis_prompt(
|
||||
query: str,
|
||||
original_responses: Dict[str, str],
|
||||
evaluations: Dict[str, str],
|
||||
id_to_member: Dict[str, str]
|
||||
) -> str:
|
||||
"""
|
||||
Create synthesis prompt for the Chairman.
|
||||
|
||||
Args:
|
||||
query: Original user query
|
||||
original_responses: Dict mapping member names to their responses
|
||||
evaluations: Dict mapping evaluator names to their evaluation texts
|
||||
id_to_member: Mapping from anonymous IDs to member names
|
||||
|
||||
Returns:
|
||||
Formatted synthesis prompt
|
||||
"""
|
||||
responses_section = "\n\n".join([
|
||||
f"=== {name} ===\n{response}"
|
||||
for name, response in original_responses.items()
|
||||
])
|
||||
|
||||
evaluations_section = "\n\n".join([
|
||||
f"=== Evaluation by {name} ===\n{evaluation}"
|
||||
for name, evaluation in evaluations.items()
|
||||
])
|
||||
|
||||
return f"""As the Chairman of the LLM Council, synthesize the following information into a final, comprehensive answer.
|
||||
|
||||
ORIGINAL QUERY:
|
||||
{query}
|
||||
|
||||
COUNCIL MEMBER RESPONSES:
|
||||
{responses_section}
|
||||
|
||||
COUNCIL MEMBER EVALUATIONS AND RANKINGS:
|
||||
{evaluations_section}
|
||||
|
||||
ANONYMOUS ID MAPPING (for reference):
|
||||
{chr(10).join([f" {aid} = {name}" for aid, name in id_to_member.items()])}
|
||||
|
||||
Your task:
|
||||
1. Review all council member responses
|
||||
2. Consider the evaluations and rankings provided by each member
|
||||
3. Identify the strongest elements from each response
|
||||
4. Synthesize a final, comprehensive answer that:
|
||||
- Incorporates the best insights from multiple perspectives
|
||||
- Addresses the query thoroughly and accurately
|
||||
- Benefits from the diversity of approaches taken
|
||||
- Is clear, well-structured, and useful
|
||||
|
||||
Provide your final synthesized response below. You may reference which perspectives or approaches influenced different parts of your answer."""
|
||||
|
||||
|
||||
class LLMCouncil:
|
||||
"""
|
||||
An LLM Council that orchestrates multiple specialized agents to collaboratively
|
||||
answer queries through independent responses, peer review, and synthesis.
|
||||
|
||||
The council follows this workflow:
|
||||
1. Dispatch query to all council members in parallel
|
||||
2. Collect all responses (anonymized)
|
||||
3. Have each member review and rank all responses
|
||||
4. Chairman synthesizes everything into final response
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
council_members: Optional[List[Agent]] = None,
|
||||
chairman_model: str = "gpt-5.1",
|
||||
verbose: bool = True,
|
||||
):
|
||||
"""
|
||||
Initialize the LLM Council.
|
||||
|
||||
Args:
|
||||
council_members: List of Agent instances representing council members.
|
||||
If None, creates default council with GPT-5.1, Gemini 3 Pro,
|
||||
Claude Sonnet 4.5, and Grok-4.
|
||||
chairman_model: Model name for the Chairman agent that synthesizes responses.
|
||||
verbose: Whether to print progress and intermediate results.
|
||||
"""
|
||||
self.verbose = verbose
|
||||
|
||||
# Create default council members if none provided
|
||||
if council_members is None:
|
||||
self.council_members = self._create_default_council()
|
||||
else:
|
||||
self.council_members = council_members
|
||||
|
||||
# Create Chairman agent
|
||||
self.chairman = Agent(
|
||||
agent_name="Chairman",
|
||||
agent_description="Chairman of the LLM Council, responsible for synthesizing all responses and rankings into a final answer",
|
||||
system_prompt=get_chairman_prompt(),
|
||||
model_name=chairman_model,
|
||||
max_loops=1,
|
||||
verbose=verbose,
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
if self.verbose:
|
||||
print(f"🏛️ LLM Council initialized with {len(self.council_members)} members")
|
||||
for i, member in enumerate(self.council_members, 1):
|
||||
print(f" {i}. {member.agent_name} ({member.model_name})")
|
||||
|
||||
def _create_default_council(self) -> List[Agent]:
|
||||
"""
|
||||
Create default council members with specialized prompts and models.
|
||||
|
||||
Returns:
|
||||
List of Agent instances configured as council members.
|
||||
"""
|
||||
|
||||
# GPT-5.1 Agent - Analytical and comprehensive
|
||||
gpt_agent = Agent(
|
||||
agent_name="GPT-5.1-Councilor",
|
||||
agent_description="Analytical and comprehensive AI councilor specializing in deep analysis and thorough responses",
|
||||
system_prompt=get_gpt_councilor_prompt(),
|
||||
model_name="gpt-5.1",
|
||||
max_loops=1,
|
||||
verbose=False,
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
# Gemini 3 Pro Agent - Concise and processed
|
||||
gemini_agent = Agent(
|
||||
agent_name="Gemini-3-Pro-Councilor",
|
||||
agent_description="Concise and well-processed AI councilor specializing in clear, structured responses",
|
||||
system_prompt=get_gemini_councilor_prompt(),
|
||||
model_name="gemini-2.5-flash", # Using available Gemini model
|
||||
max_loops=1,
|
||||
verbose=False,
|
||||
temperature=0.7,
|
||||
)
|
||||
|
||||
# Claude Sonnet 4.5 Agent - Balanced and thoughtful
|
||||
claude_agent = Agent(
|
||||
agent_name="Claude-Sonnet-4.5-Councilor",
|
||||
agent_description="Thoughtful and balanced AI councilor specializing in nuanced and well-reasoned responses",
|
||||
system_prompt=get_claude_councilor_prompt(),
|
||||
model_name="anthropic/claude-sonnet-4-5", # Using available Claude model
|
||||
max_loops=1,
|
||||
verbose=False,
|
||||
temperature=0.0,
|
||||
top_p=None,
|
||||
)
|
||||
|
||||
# Grok-4 Agent - Creative and innovative
|
||||
grok_agent = Agent(
|
||||
agent_name="Grok-4-Councilor",
|
||||
agent_description="Creative and innovative AI councilor specializing in unique perspectives and creative solutions",
|
||||
system_prompt=get_grok_councilor_prompt(),
|
||||
model_name="x-ai/grok-4", # Using available model as proxy for Grok-4
|
||||
max_loops=1,
|
||||
verbose=False,
|
||||
temperature=0.8,
|
||||
)
|
||||
|
||||
members = [gpt_agent, gemini_agent, claude_agent, grok_agent]
|
||||
|
||||
return members
|
||||
|
||||
def run(self, query: str) -> Dict:
|
||||
"""
|
||||
Execute the full LLM Council workflow.
|
||||
|
||||
Args:
|
||||
query: The user's query to process
|
||||
|
||||
Returns:
|
||||
Dictionary containing:
|
||||
- original_responses: Dict mapping member names to their responses
|
||||
- evaluations: Dict mapping evaluator names to their rankings
|
||||
- final_response: The Chairman's synthesized final answer
|
||||
"""
|
||||
if self.verbose:
|
||||
print(f"\n{'='*80}")
|
||||
print("🏛️ LLM COUNCIL SESSION")
|
||||
print("="*80)
|
||||
print(f"\n📝 Query: {query}\n")
|
||||
|
||||
# Step 1: Get responses from all council members in parallel
|
||||
if self.verbose:
|
||||
print("📤 Dispatching query to all council members...")
|
||||
|
||||
results_dict = run_agents_concurrently(
|
||||
self.council_members,
|
||||
task=query,
|
||||
return_agent_output_dict=True
|
||||
)
|
||||
|
||||
# Map results to member names
|
||||
original_responses = {
|
||||
member.agent_name: response
|
||||
for member, response in zip(self.council_members,
|
||||
[results_dict.get(member.agent_name, "")
|
||||
for member in self.council_members])
|
||||
}
|
||||
|
||||
if self.verbose:
|
||||
print(f"✅ Received {len(original_responses)} responses\n")
|
||||
for name, response in original_responses.items():
|
||||
print(f" {name}: {response[:100]}...")
|
||||
|
||||
# Step 2: Anonymize responses for evaluation
|
||||
# Create anonymous IDs (A, B, C, D, etc.)
|
||||
anonymous_ids = [chr(65 + i) for i in range(len(self.council_members))]
|
||||
random.shuffle(anonymous_ids) # Shuffle to ensure anonymity
|
||||
|
||||
anonymous_responses = {
|
||||
anonymous_ids[i]: original_responses[member.agent_name]
|
||||
for i, member in enumerate(self.council_members)
|
||||
}
|
||||
|
||||
# Create mapping from anonymous ID to member name (for later reference)
|
||||
id_to_member = {
|
||||
anonymous_ids[i]: member.agent_name
|
||||
for i, member in enumerate(self.council_members)
|
||||
}
|
||||
|
||||
if self.verbose:
|
||||
print("\n🔍 Council members evaluating each other's responses...")
|
||||
|
||||
# Step 3: Have each member evaluate and rank all responses concurrently
|
||||
# Create evaluation tasks for each member
|
||||
evaluation_tasks = [
|
||||
get_evaluation_prompt(query, anonymous_responses, member.agent_name)
|
||||
for member in self.council_members
|
||||
]
|
||||
|
||||
# Run evaluations concurrently using batched_grid_agent_execution
|
||||
evaluation_results = batched_grid_agent_execution(
|
||||
self.council_members,
|
||||
evaluation_tasks
|
||||
)
|
||||
|
||||
# Map results to member names
|
||||
evaluations = {
|
||||
member.agent_name: evaluation_results[i]
|
||||
for i, member in enumerate(self.council_members)
|
||||
}
|
||||
|
||||
if self.verbose:
|
||||
print(f"✅ Received {len(evaluations)} evaluations\n")
|
||||
|
||||
# Step 4: Chairman synthesizes everything
|
||||
if self.verbose:
|
||||
print("👔 Chairman synthesizing final response...\n")
|
||||
|
||||
synthesis_prompt = get_synthesis_prompt(
|
||||
query, original_responses, evaluations, id_to_member
|
||||
)
|
||||
|
||||
final_response = self.chairman.run(task=synthesis_prompt)
|
||||
|
||||
if self.verbose:
|
||||
print(f"{'='*80}")
|
||||
print("✅ FINAL RESPONSE")
|
||||
print(f"{'='*80}\n")
|
||||
|
||||
return {
|
||||
"query": query,
|
||||
"original_responses": original_responses,
|
||||
"evaluations": evaluations,
|
||||
"final_response": final_response,
|
||||
"anonymous_mapping": id_to_member,
|
||||
}
|
||||
|
||||
Loading…
Reference in new issue