Kye Gomez
4 days ago
parent
e5c2960912
commit
688772e99b
@ -0,0 +1,106 @@
|
|||||||
|
# LLM Council Examples
|
||||||
|
|
||||||
|
This page provides examples demonstrating the LLM Council pattern, inspired by Andrej Karpathy's llm-council implementation. The LLM Council uses multiple specialized AI agents that:
|
||||||
|
|
||||||
|
1. Each respond independently to queries
|
||||||
|
2. Review and rank each other's anonymized responses
|
||||||
|
3. Have a Chairman synthesize all responses into a final comprehensive answer
|
||||||
|
|
||||||
|
## Example Files
|
||||||
|
|
||||||
|
All LLM Council examples are located in the [`examples/multi_agent/llm_council_examples/`](https://github.com/kyegomez/swarms/tree/master/examples/multi_agent/llm_council_examples) directory.
|
||||||
|
|
||||||
|
### Marketing & Business
|
||||||
|
|
||||||
|
- **[marketing_strategy_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/marketing_strategy_council.py)** - Marketing strategy analysis and recommendations
|
||||||
|
- **[business_strategy_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/business_strategy_council.py)** - Comprehensive business strategy development
|
||||||
|
|
||||||
|
### Finance & Investment
|
||||||
|
|
||||||
|
- **[finance_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/finance_analysis_council.py)** - Financial analysis and investment recommendations
|
||||||
|
- **[etf_stock_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/etf_stock_analysis_council.py)** - ETF and stock analysis with portfolio recommendations
|
||||||
|
|
||||||
|
### Medical & Healthcare
|
||||||
|
|
||||||
|
- **[medical_treatment_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/medical_treatment_council.py)** - Medical treatment recommendations and care plans
|
||||||
|
- **[medical_diagnosis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/medical_diagnosis_council.py)** - Diagnostic analysis based on symptoms
|
||||||
|
|
||||||
|
### Technology & Research
|
||||||
|
|
||||||
|
- **[technology_assessment_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/technology_assessment_council.py)** - Technology evaluation and implementation strategy
|
||||||
|
- **[research_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/research_analysis_council.py)** - Comprehensive research analysis on complex topics
|
||||||
|
|
||||||
|
### Legal
|
||||||
|
|
||||||
|
- **[legal_analysis_council.py](https://github.com/kyegomez/swarms/blob/master/examples/multi_agent/llm_council_examples/legal_analysis_council.py)** - Legal implications and compliance analysis
|
||||||
|
|
||||||
|
## Basic Usage Pattern
|
||||||
|
|
||||||
|
All examples follow the same pattern:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Run a query
|
||||||
|
result = council.run("Your query here")
|
||||||
|
|
||||||
|
# Access results
|
||||||
|
print(result["final_response"]) # Chairman's synthesized answer
|
||||||
|
print(result["original_responses"]) # Individual member responses
|
||||||
|
print(result["evaluations"]) # How members ranked each other
|
||||||
|
```
|
||||||
|
|
||||||
|
## Running Examples
|
||||||
|
|
||||||
|
Run any example directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python examples/multi_agent/llm_council_examples/marketing_strategy_council.py
|
||||||
|
python examples/multi_agent/llm_council_examples/finance_analysis_council.py
|
||||||
|
python examples/multi_agent/llm_council_examples/medical_diagnosis_council.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- **Multiple Perspectives**: Each council member (GPT-5.1, Gemini, Claude, Grok) provides unique insights
|
||||||
|
- **Peer Review**: Members evaluate and rank each other's responses anonymously
|
||||||
|
- **Synthesis**: Chairman combines the best elements from all responses
|
||||||
|
- **Transparency**: See both individual responses and evaluation rankings
|
||||||
|
|
||||||
|
## Council Members
|
||||||
|
|
||||||
|
The default council consists of:
|
||||||
|
- **GPT-5.1-Councilor**: Analytical and comprehensive
|
||||||
|
- **Gemini-3-Pro-Councilor**: Concise and well-processed
|
||||||
|
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced
|
||||||
|
- **Grok-4-Councilor**: Creative and innovative
|
||||||
|
|
||||||
|
## Customization
|
||||||
|
|
||||||
|
You can create custom council members:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import Agent
|
||||||
|
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||||
|
|
||||||
|
custom_agent = Agent(
|
||||||
|
agent_name="Custom-Councilor",
|
||||||
|
system_prompt=get_gpt_councilor_prompt(),
|
||||||
|
model_name="gpt-4.1",
|
||||||
|
max_loops=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
council = LLMCouncil(
|
||||||
|
council_members=[custom_agent, ...],
|
||||||
|
chairman_model="gpt-5.1",
|
||||||
|
verbose=True
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Documentation
|
||||||
|
|
||||||
|
For complete API reference and detailed documentation, see the [LLM Council Reference Documentation](../swarms/structs/llm_council.md).
|
||||||
|
|
||||||
@ -0,0 +1,453 @@
|
|||||||
|
# LLM Council Class Documentation
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart TD
|
||||||
|
A[User Query] --> B[LLM Council Initialization]
|
||||||
|
B --> C{Council Members Provided?}
|
||||||
|
C -->|No| D[Create Default Council]
|
||||||
|
C -->|Yes| E[Use Provided Members]
|
||||||
|
D --> F[Step 1: Parallel Response Generation]
|
||||||
|
E --> F
|
||||||
|
|
||||||
|
subgraph "Default Council Members"
|
||||||
|
G1[GPT-5.1-Councilor<br/>Analytical & Comprehensive]
|
||||||
|
G2[Gemini-3-Pro-Councilor<br/>Concise & Structured]
|
||||||
|
G3[Claude-Sonnet-4.5-Councilor<br/>Thoughtful & Balanced]
|
||||||
|
G4[Grok-4-Councilor<br/>Creative & Innovative]
|
||||||
|
end
|
||||||
|
|
||||||
|
F --> G1
|
||||||
|
F --> G2
|
||||||
|
F --> G3
|
||||||
|
F --> G4
|
||||||
|
|
||||||
|
G1 --> H[Collect All Responses]
|
||||||
|
G2 --> H
|
||||||
|
G3 --> H
|
||||||
|
G4 --> H
|
||||||
|
|
||||||
|
H --> I[Step 2: Anonymize Responses]
|
||||||
|
I --> J[Assign Anonymous IDs: A, B, C, D...]
|
||||||
|
|
||||||
|
J --> K[Step 3: Parallel Evaluation]
|
||||||
|
|
||||||
|
subgraph "Evaluation Phase"
|
||||||
|
K --> L1[Member 1 Evaluates All]
|
||||||
|
K --> L2[Member 2 Evaluates All]
|
||||||
|
K --> L3[Member 3 Evaluates All]
|
||||||
|
K --> L4[Member 4 Evaluates All]
|
||||||
|
end
|
||||||
|
|
||||||
|
L1 --> M[Collect Evaluations & Rankings]
|
||||||
|
L2 --> M
|
||||||
|
L3 --> M
|
||||||
|
L4 --> M
|
||||||
|
|
||||||
|
M --> N[Step 4: Chairman Synthesis]
|
||||||
|
N --> O[Chairman Agent]
|
||||||
|
O --> P[Final Synthesized Response]
|
||||||
|
|
||||||
|
P --> Q[Return Results Dictionary]
|
||||||
|
|
||||||
|
style A fill:#e1f5ff
|
||||||
|
style P fill:#c8e6c9
|
||||||
|
style Q fill:#c8e6c9
|
||||||
|
style O fill:#fff9c4
|
||||||
|
```
|
||||||
|
|
||||||
|
The `LLMCouncil` class orchestrates multiple specialized LLM agents to collaboratively answer queries through a structured peer review and synthesis process. Inspired by Andrej Karpathy's llm-council implementation, this architecture demonstrates how different models evaluate and rank each other's work, often selecting responses from other models as superior to their own.
|
||||||
|
|
||||||
|
## Workflow Overview
|
||||||
|
|
||||||
|
The LLM Council follows a four-step process:
|
||||||
|
|
||||||
|
1. **Parallel Response Generation**: All council members independently respond to the user query
|
||||||
|
2. **Anonymization**: Responses are anonymized with random IDs (A, B, C, D, etc.) to ensure objective evaluation
|
||||||
|
3. **Peer Review**: Each member evaluates and ranks all responses (including potentially their own)
|
||||||
|
4. **Synthesis**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer
|
||||||
|
|
||||||
|
## Class Definition
|
||||||
|
|
||||||
|
### LLMCouncil
|
||||||
|
|
||||||
|
```python
|
||||||
|
class LLMCouncil:
|
||||||
|
```
|
||||||
|
|
||||||
|
### Attributes
|
||||||
|
|
||||||
|
| Attribute | Type | Description | Default |
|
||||||
|
|-----------|------|-------------|---------|
|
||||||
|
| `council_members` | `List[Agent]` | List of Agent instances representing council members | `None` (creates default council) |
|
||||||
|
| `chairman` | `Agent` | The Chairman agent responsible for synthesizing responses | Created during initialization |
|
||||||
|
| `verbose` | `bool` | Whether to print progress and intermediate results | `True` |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `__init__`
|
||||||
|
|
||||||
|
Initializes the LLM Council with council members and a Chairman agent.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|-----------|------|---------|-------------|
|
||||||
|
| `council_members` | `Optional[List[Agent]]` | `None` | List of Agent instances representing council members. If `None`, creates default council with GPT-5.1, Gemini 3 Pro, Claude Sonnet 4.5, and Grok-4. |
|
||||||
|
| `chairman_model` | `str` | `"gpt-5.1"` | Model name for the Chairman agent that synthesizes responses. |
|
||||||
|
| `verbose` | `bool` | `True` | Whether to print progress and intermediate results. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `LLMCouncil` | Initialized LLM Council instance. |
|
||||||
|
|
||||||
|
#### Description
|
||||||
|
|
||||||
|
Creates an LLM Council instance with specialized council members. If no members are provided, it creates a default council consisting of:
|
||||||
|
- **GPT-5.1-Councilor**: Analytical and comprehensive responses
|
||||||
|
- **Gemini-3-Pro-Councilor**: Concise and well-processed responses
|
||||||
|
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced responses
|
||||||
|
- **Grok-4-Councilor**: Creative and innovative responses
|
||||||
|
|
||||||
|
The Chairman agent is automatically created with a specialized prompt for synthesizing responses.
|
||||||
|
|
||||||
|
#### Example Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create council with default members
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Create council with custom members
|
||||||
|
from swarms import Agent
|
||||||
|
custom_members = [
|
||||||
|
Agent(agent_name="Expert-1", model_name="gpt-4", max_loops=1),
|
||||||
|
Agent(agent_name="Expert-2", model_name="claude-3-opus", max_loops=1),
|
||||||
|
]
|
||||||
|
council = LLMCouncil(
|
||||||
|
council_members=custom_members,
|
||||||
|
chairman_model="gpt-4",
|
||||||
|
verbose=True
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `run`
|
||||||
|
|
||||||
|
Executes the full LLM Council workflow: parallel responses, anonymization, peer review, and synthesis.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Default | Description |
|
||||||
|
|-----------|------|---------|-------------|
|
||||||
|
| `query` | `str` | Required | The user's query to process through the council. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `Dict` | Dictionary containing the following keys: |
|
||||||
|
|
||||||
|
#### Return Dictionary Structure
|
||||||
|
|
||||||
|
| Key | Type | Description |
|
||||||
|
|-----|------|-------------|
|
||||||
|
| `query` | `str` | The original user query. |
|
||||||
|
| `original_responses` | `Dict[str, str]` | Dictionary mapping council member names to their original responses. |
|
||||||
|
| `evaluations` | `Dict[str, str]` | Dictionary mapping evaluator names to their evaluation texts (rankings and reasoning). |
|
||||||
|
| `final_response` | `str` | The Chairman's synthesized final answer combining all perspectives. |
|
||||||
|
| `anonymous_mapping` | `Dict[str, str]` | Mapping from anonymous IDs (A, B, C, D) to member names for reference. |
|
||||||
|
|
||||||
|
#### Description
|
||||||
|
|
||||||
|
Executes the complete LLM Council workflow:
|
||||||
|
|
||||||
|
1. **Dispatch Phase**: Sends the query to all council members in parallel using `run_agents_concurrently`
|
||||||
|
2. **Collection Phase**: Collects all responses and maps them to member names
|
||||||
|
3. **Anonymization Phase**: Creates anonymous IDs (A, B, C, D, etc.) and shuffles them to ensure anonymity
|
||||||
|
4. **Evaluation Phase**: Each member evaluates and ranks all anonymized responses using `batched_grid_agent_execution`
|
||||||
|
5. **Synthesis Phase**: The Chairman agent synthesizes all responses and evaluations into a final comprehensive answer
|
||||||
|
|
||||||
|
The method provides verbose output by default, showing progress at each stage.
|
||||||
|
|
||||||
|
#### Example Usage
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||||
|
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Access the final synthesized response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
|
# Access individual member responses
|
||||||
|
for name, response in result["original_responses"].items():
|
||||||
|
print(f"{name}: {response[:200]}...")
|
||||||
|
|
||||||
|
# Access evaluation rankings
|
||||||
|
for evaluator, evaluation in result["evaluations"].items():
|
||||||
|
print(f"{evaluator} evaluation:\n{evaluation[:300]}...")
|
||||||
|
|
||||||
|
# Check anonymous mapping
|
||||||
|
print("Anonymous IDs:", result["anonymous_mapping"])
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `_create_default_council`
|
||||||
|
|
||||||
|
Creates default council members with specialized prompts and models.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
None (internal method).
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `List[Agent]` | List of Agent instances configured as council members. |
|
||||||
|
|
||||||
|
#### Description
|
||||||
|
|
||||||
|
Internal method that creates the default council configuration with four specialized agents:
|
||||||
|
|
||||||
|
- **GPT-5.1-Councilor** (`model_name="gpt-5.1"`): Analytical and comprehensive, temperature=0.7
|
||||||
|
- **Gemini-3-Pro-Councilor** (`model_name="gemini-2.5-flash"`): Concise and structured, temperature=0.7
|
||||||
|
- **Claude-Sonnet-4.5-Councilor** (`model_name="anthropic/claude-sonnet-4-5"`): Thoughtful and balanced, temperature=0.0
|
||||||
|
- **Grok-4-Councilor** (`model_name="x-ai/grok-4"`): Creative and innovative, temperature=0.8
|
||||||
|
|
||||||
|
Each agent is configured with:
|
||||||
|
- Specialized system prompts matching their role
|
||||||
|
- `max_loops=1` for single-response generation
|
||||||
|
- `verbose=False` to reduce noise during parallel execution
|
||||||
|
- Appropriate temperature settings for their style
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Helper Functions
|
||||||
|
|
||||||
|
### `get_gpt_councilor_prompt()`
|
||||||
|
|
||||||
|
Returns the system prompt for GPT-5.1 councilor agent.
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | System prompt string emphasizing analytical thinking and comprehensive coverage. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_gemini_councilor_prompt()`
|
||||||
|
|
||||||
|
Returns the system prompt for Gemini 3 Pro councilor agent.
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | System prompt string emphasizing concise, well-processed, and structured responses. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_claude_councilor_prompt()`
|
||||||
|
|
||||||
|
Returns the system prompt for Claude Sonnet 4.5 councilor agent.
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | System prompt string emphasizing thoughtful, balanced, and nuanced responses. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_grok_councilor_prompt()`
|
||||||
|
|
||||||
|
Returns the system prompt for Grok-4 councilor agent.
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | System prompt string emphasizing creative, innovative, and unique perspectives. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_chairman_prompt()`
|
||||||
|
|
||||||
|
Returns the system prompt for the Chairman agent.
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | System prompt string for synthesizing responses and evaluations into a final answer. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_evaluation_prompt(query, responses, evaluator_name)`
|
||||||
|
|
||||||
|
Creates evaluation prompt for council members to review and rank responses.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Description |
|
||||||
|
|-----------|------|-------------|
|
||||||
|
| `query` | `str` | The original user query. |
|
||||||
|
| `responses` | `Dict[str, str]` | Dictionary mapping anonymous IDs to response texts. |
|
||||||
|
| `evaluator_name` | `str` | Name of the agent doing the evaluation. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | Formatted evaluation prompt string with instructions for ranking responses. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### `get_synthesis_prompt(query, original_responses, evaluations, id_to_member)`
|
||||||
|
|
||||||
|
Creates synthesis prompt for the Chairman.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Description |
|
||||||
|
|-----------|------|-------------|
|
||||||
|
| `query` | `str` | Original user query. |
|
||||||
|
| `original_responses` | `Dict[str, str]` | Dictionary mapping member names to their responses. |
|
||||||
|
| `evaluations` | `Dict[str, str]` | Dictionary mapping evaluator names to their evaluation texts. |
|
||||||
|
| `id_to_member` | `Dict[str, str]` | Mapping from anonymous IDs to member names. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Type | Description |
|
||||||
|
|------|-------------|
|
||||||
|
| `str` | Formatted synthesis prompt for the Chairman agent. |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Use Cases
|
||||||
|
|
||||||
|
The LLM Council is ideal for scenarios requiring:
|
||||||
|
|
||||||
|
- **Multi-perspective Analysis**: When you need diverse viewpoints on complex topics
|
||||||
|
- **Quality Assurance**: When peer review and ranking can improve response quality
|
||||||
|
- **Transparent Decision Making**: When you want to see how different models evaluate each other
|
||||||
|
- **Synthesis of Expertise**: When combining multiple specialized perspectives is valuable
|
||||||
|
|
||||||
|
### Common Applications
|
||||||
|
|
||||||
|
- **Medical Diagnosis**: Multiple medical AI agents provide diagnoses, evaluate each other, and synthesize recommendations
|
||||||
|
- **Financial Analysis**: Different financial experts analyze investments and rank each other's assessments
|
||||||
|
- **Legal Analysis**: Multiple legal perspectives evaluate compliance and risk
|
||||||
|
- **Business Strategy**: Diverse strategic viewpoints are synthesized into comprehensive plans
|
||||||
|
- **Research Analysis**: Multiple research perspectives are combined for thorough analysis
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
For comprehensive examples demonstrating various use cases, see the [LLM Council Examples](../../../examples/multi_agent/llm_council_examples/) directory:
|
||||||
|
|
||||||
|
- **Medical**: `medical_diagnosis_council.py`, `medical_treatment_council.py`
|
||||||
|
- **Finance**: `finance_analysis_council.py`, `etf_stock_analysis_council.py`
|
||||||
|
- **Business**: `business_strategy_council.py`, `marketing_strategy_council.py`
|
||||||
|
- **Technology**: `technology_assessment_council.py`, `research_analysis_council.py`
|
||||||
|
- **Legal**: `legal_analysis_council.py`
|
||||||
|
|
||||||
|
### Quick Start Example
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Example query
|
||||||
|
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
|
# Optionally print evaluations
|
||||||
|
print("\n\n" + "="*80)
|
||||||
|
print("EVALUATIONS")
|
||||||
|
print("="*80)
|
||||||
|
for name, evaluation in result["evaluations"].items():
|
||||||
|
print(f"\n{name}:")
|
||||||
|
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Customization
|
||||||
|
|
||||||
|
### Creating Custom Council Members
|
||||||
|
|
||||||
|
You can create custom council members with specialized roles:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import Agent
|
||||||
|
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||||
|
|
||||||
|
# Create custom councilor
|
||||||
|
custom_agent = Agent(
|
||||||
|
agent_name="Domain-Expert-Councilor",
|
||||||
|
agent_description="Specialized domain expert for specific analysis",
|
||||||
|
system_prompt=get_gpt_councilor_prompt(), # Or create custom prompt
|
||||||
|
model_name="gpt-4",
|
||||||
|
max_loops=1,
|
||||||
|
verbose=False,
|
||||||
|
temperature=0.7,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Create council with custom members
|
||||||
|
council = LLMCouncil(
|
||||||
|
council_members=[custom_agent, ...], # Add your custom agents
|
||||||
|
chairman_model="gpt-4",
|
||||||
|
verbose=True
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Chairman Model
|
||||||
|
|
||||||
|
You can specify a different model for the Chairman:
|
||||||
|
|
||||||
|
```python
|
||||||
|
council = LLMCouncil(
|
||||||
|
chairman_model="claude-3-opus", # Use Claude as Chairman
|
||||||
|
verbose=True
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture Benefits
|
||||||
|
|
||||||
|
1. **Diversity**: Multiple models provide varied perspectives and approaches
|
||||||
|
2. **Quality Control**: Peer review ensures responses are evaluated objectively
|
||||||
|
3. **Synthesis**: Chairman combines the best elements from all responses
|
||||||
|
4. **Transparency**: Full visibility into individual responses and evaluation rankings
|
||||||
|
5. **Scalability**: Easy to add or remove council members
|
||||||
|
6. **Flexibility**: Supports custom agents and models
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- **Parallel Execution**: Both response generation and evaluation phases run in parallel for efficiency
|
||||||
|
- **Anonymization**: Responses are anonymized to prevent bias in evaluation
|
||||||
|
- **Model Selection**: Different models can be used for different roles based on their strengths
|
||||||
|
- **Verbose Mode**: Can be disabled for production use to reduce output
|
||||||
|
|
||||||
|
## Related Documentation
|
||||||
|
|
||||||
|
- [Multi-Agent Architectures Overview](overview.md)
|
||||||
|
- [Council of Judges](council_of_judges.md) - Similar peer review pattern
|
||||||
|
- [Agent Class Reference](agent.md) - Understanding individual agents
|
||||||
|
- [Multi-Agent Execution Utilities](various_execution_methods.md) - Underlying execution methods
|
||||||
|
|
||||||
@ -0,0 +1,95 @@
|
|||||||
|
# LLM Council Examples
|
||||||
|
|
||||||
|
This directory contains examples demonstrating the LLM Council pattern, inspired by Andrej Karpathy's llm-council implementation. The LLM Council uses multiple specialized AI agents that:
|
||||||
|
|
||||||
|
1. Each respond independently to queries
|
||||||
|
2. Review and rank each other's anonymized responses
|
||||||
|
3. Have a Chairman synthesize all responses into a final comprehensive answer
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Marketing & Business
|
||||||
|
- **marketing_strategy_council.py** - Marketing strategy analysis and recommendations
|
||||||
|
- **business_strategy_council.py** - Comprehensive business strategy development
|
||||||
|
|
||||||
|
### Finance & Investment
|
||||||
|
- **finance_analysis_council.py** - Financial analysis and investment recommendations
|
||||||
|
- **etf_stock_analysis_council.py** - ETF and stock analysis with portfolio recommendations
|
||||||
|
|
||||||
|
### Medical & Healthcare
|
||||||
|
- **medical_treatment_council.py** - Medical treatment recommendations and care plans
|
||||||
|
- **medical_diagnosis_council.py** - Diagnostic analysis based on symptoms
|
||||||
|
|
||||||
|
### Technology & Research
|
||||||
|
- **technology_assessment_council.py** - Technology evaluation and implementation strategy
|
||||||
|
- **research_analysis_council.py** - Comprehensive research analysis on complex topics
|
||||||
|
|
||||||
|
### Legal
|
||||||
|
- **legal_analysis_council.py** - Legal implications and compliance analysis
|
||||||
|
|
||||||
|
## Usage
|
||||||
|
|
||||||
|
Each example follows the same pattern:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Run a query
|
||||||
|
result = council.run("Your query here")
|
||||||
|
|
||||||
|
# Access results
|
||||||
|
print(result["final_response"]) # Chairman's synthesized answer
|
||||||
|
print(result["original_responses"]) # Individual member responses
|
||||||
|
print(result["evaluations"]) # How members ranked each other
|
||||||
|
```
|
||||||
|
|
||||||
|
## Running Examples
|
||||||
|
|
||||||
|
Run any example directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python examples/multi_agent/llm_council_examples/marketing_strategy_council.py
|
||||||
|
python examples/multi_agent/llm_council_examples/finance_analysis_council.py
|
||||||
|
python examples/multi_agent/llm_council_examples/medical_diagnosis_council.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Key Features
|
||||||
|
|
||||||
|
- **Multiple Perspectives**: Each council member (GPT-5.1, Gemini, Claude, Grok) provides unique insights
|
||||||
|
- **Peer Review**: Members evaluate and rank each other's responses anonymously
|
||||||
|
- **Synthesis**: Chairman combines the best elements from all responses
|
||||||
|
- **Transparency**: See both individual responses and evaluation rankings
|
||||||
|
|
||||||
|
## Council Members
|
||||||
|
|
||||||
|
The default council consists of:
|
||||||
|
- **GPT-5.1-Councilor**: Analytical and comprehensive
|
||||||
|
- **Gemini-3-Pro-Councilor**: Concise and well-processed
|
||||||
|
- **Claude-Sonnet-4.5-Councilor**: Thoughtful and balanced
|
||||||
|
- **Grok-4-Councilor**: Creative and innovative
|
||||||
|
|
||||||
|
## Customization
|
||||||
|
|
||||||
|
You can create custom council members:
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import Agent
|
||||||
|
from swarms.structs.llm_council import LLMCouncil, get_gpt_councilor_prompt
|
||||||
|
|
||||||
|
custom_agent = Agent(
|
||||||
|
agent_name="Custom-Councilor",
|
||||||
|
system_prompt=get_gpt_councilor_prompt(),
|
||||||
|
model_name="gpt-4.1",
|
||||||
|
max_loops=1,
|
||||||
|
)
|
||||||
|
|
||||||
|
council = LLMCouncil(
|
||||||
|
council_members=[custom_agent, ...],
|
||||||
|
chairman_model="gpt-5.1",
|
||||||
|
verbose=True
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
@ -0,0 +1,32 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Business Strategy Development
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to develop comprehensive
|
||||||
|
business strategies for new ventures.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Business strategy query
|
||||||
|
query = """
|
||||||
|
A tech startup wants to launch an AI-powered personal finance app targeting
|
||||||
|
millennials and Gen Z. Develop a comprehensive business strategy including:
|
||||||
|
1. Market opportunity and competitive landscape analysis
|
||||||
|
2. Product positioning and unique value proposition
|
||||||
|
3. Go-to-market strategy and customer acquisition plan
|
||||||
|
4. Revenue model and pricing strategy
|
||||||
|
5. Key partnerships and distribution channels
|
||||||
|
6. Resource requirements and funding needs
|
||||||
|
7. Risk assessment and mitigation strategies
|
||||||
|
8. Success metrics and KPIs for first 12 months
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,30 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: ETF Stock Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to analyze ETF holdings
|
||||||
|
and provide stock investment recommendations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# ETF and stock analysis query
|
||||||
|
query = """
|
||||||
|
Analyze the top energy ETFs (including nuclear, solar, gas, and renewable energy)
|
||||||
|
and provide:
|
||||||
|
1. Top 5 best-performing energy stocks across all energy sectors
|
||||||
|
2. ETF recommendations for diversified energy exposure
|
||||||
|
3. Risk-return profiles for each recommendation
|
||||||
|
4. Current market conditions affecting energy investments
|
||||||
|
5. Allocation strategy for a $100,000 portfolio
|
||||||
|
6. Key metrics to track for each investment
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,30 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Financial Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to provide comprehensive
|
||||||
|
financial analysis and investment recommendations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Financial analysis query
|
||||||
|
query = """
|
||||||
|
Provide a comprehensive financial analysis for investing in emerging markets
|
||||||
|
technology ETFs. Include:
|
||||||
|
1. Risk assessment and volatility analysis
|
||||||
|
2. Historical performance trends
|
||||||
|
3. Sector composition and diversification benefits
|
||||||
|
4. Comparison with developed market tech ETFs
|
||||||
|
5. Recommended allocation percentage for a moderate risk portfolio
|
||||||
|
6. Key factors to monitor going forward
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,32 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Legal Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to analyze legal scenarios
|
||||||
|
and provide comprehensive legal insights.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Legal analysis query
|
||||||
|
query = """
|
||||||
|
A startup is considering using AI-generated content for their marketing materials.
|
||||||
|
Analyze the legal implications including:
|
||||||
|
1. Intellectual property rights and ownership of AI-generated content
|
||||||
|
2. Copyright and trademark considerations
|
||||||
|
3. Liability for AI-generated content that may be inaccurate or misleading
|
||||||
|
4. Compliance with advertising regulations (FTC, FDA, etc.)
|
||||||
|
5. Data privacy implications if using customer data to train models
|
||||||
|
6. Contractual considerations with AI service providers
|
||||||
|
7. Risk mitigation strategies
|
||||||
|
8. Best practices for legal compliance
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,29 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Marketing Strategy Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to analyze and develop
|
||||||
|
comprehensive marketing strategies by leveraging multiple AI perspectives.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Marketing strategy query
|
||||||
|
query = """
|
||||||
|
Analyze the marketing strategy for a new sustainable energy startup launching
|
||||||
|
a solar panel subscription service. Provide recommendations on:
|
||||||
|
1. Target audience segmentation
|
||||||
|
2. Key messaging and value propositions
|
||||||
|
3. Marketing channels and budget allocation
|
||||||
|
4. Competitive positioning
|
||||||
|
5. Launch timeline and milestones
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,37 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Medical Diagnosis Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to analyze symptoms
|
||||||
|
and provide diagnostic insights.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Medical diagnosis query
|
||||||
|
query = """
|
||||||
|
A 35-year-old patient presents with:
|
||||||
|
- Persistent fatigue for 3 months
|
||||||
|
- Unexplained weight loss (15 lbs)
|
||||||
|
- Night sweats
|
||||||
|
- Intermittent low-grade fever
|
||||||
|
- Swollen lymph nodes in neck and armpits
|
||||||
|
- Recent blood work shows elevated ESR and CRP
|
||||||
|
|
||||||
|
Provide:
|
||||||
|
1. Differential diagnosis with most likely conditions ranked
|
||||||
|
2. Additional diagnostic tests needed to confirm
|
||||||
|
3. Red flag symptoms requiring immediate attention
|
||||||
|
4. Possible causes and risk factors
|
||||||
|
5. Recommended next steps for the patient
|
||||||
|
6. When to seek emergency care
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,31 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Medical Treatment Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to analyze medical treatments
|
||||||
|
and provide comprehensive treatment recommendations.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Medical treatment query
|
||||||
|
query = """
|
||||||
|
A 45-year-old patient with Type 2 diabetes, hypertension, and early-stage
|
||||||
|
kidney disease needs treatment recommendations. Provide:
|
||||||
|
1. Comprehensive treatment plan addressing all conditions
|
||||||
|
2. Medication options with pros/cons for each condition
|
||||||
|
3. Lifestyle modifications and their expected impact
|
||||||
|
4. Monitoring schedule and key metrics to track
|
||||||
|
5. Potential drug interactions and contraindications
|
||||||
|
6. Expected outcomes and timeline for improvement
|
||||||
|
7. When to consider specialist referrals
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,32 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Research Analysis
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to conduct comprehensive
|
||||||
|
research analysis on complex topics.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Research analysis query
|
||||||
|
query = """
|
||||||
|
Conduct a comprehensive analysis of the potential impact of climate change
|
||||||
|
on global food security over the next 20 years. Include:
|
||||||
|
1. Key climate factors affecting agriculture (temperature, precipitation, extreme weather)
|
||||||
|
2. Regional vulnerabilities and impacts on major food-producing regions
|
||||||
|
3. Crop yield projections and food availability scenarios
|
||||||
|
4. Economic implications and food price volatility
|
||||||
|
5. Adaptation strategies and technological solutions
|
||||||
|
6. Policy recommendations for governments and international organizations
|
||||||
|
7. Role of innovation in agriculture (precision farming, GMOs, vertical farming)
|
||||||
|
8. Social and geopolitical implications of food insecurity
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,32 @@
|
|||||||
|
"""
|
||||||
|
LLM Council Example: Technology Assessment
|
||||||
|
|
||||||
|
This example demonstrates using the LLM Council to assess emerging technologies
|
||||||
|
and their business implications.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Technology assessment query
|
||||||
|
query = """
|
||||||
|
Evaluate the business potential and implementation strategy for integrating
|
||||||
|
quantum computing capabilities into a financial services company. Consider:
|
||||||
|
1. Current state of quantum computing technology
|
||||||
|
2. Specific use cases in financial services (risk modeling, portfolio optimization, fraud detection)
|
||||||
|
3. Competitive advantages and potential ROI
|
||||||
|
4. Implementation timeline and resource requirements
|
||||||
|
5. Technical challenges and limitations
|
||||||
|
6. Risk factors and mitigation strategies
|
||||||
|
7. Partnership opportunities with quantum computing providers
|
||||||
|
8. Expected timeline for practical business value
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
@ -0,0 +1,23 @@
|
|||||||
|
from swarms.structs.llm_council import LLMCouncil
|
||||||
|
|
||||||
|
# Example usage of the LLM Council without a function:
|
||||||
|
# Create the council
|
||||||
|
council = LLMCouncil(verbose=True)
|
||||||
|
|
||||||
|
# Example query
|
||||||
|
query = "What are the top five best energy stocks across nuclear, solar, gas, and other energy sources?"
|
||||||
|
|
||||||
|
# Run the council
|
||||||
|
result = council.run(query)
|
||||||
|
|
||||||
|
# Print final response
|
||||||
|
print(result["final_response"])
|
||||||
|
|
||||||
|
# Optionally print evaluations
|
||||||
|
print("\n\n" + "="*80)
|
||||||
|
print("EVALUATIONS")
|
||||||
|
print("="*80)
|
||||||
|
for name, evaluation in result["evaluations"].items():
|
||||||
|
print(f"\n{name}:")
|
||||||
|
print(evaluation[:500] + "..." if len(evaluation) > 500 else evaluation)
|
||||||
|
|
||||||
@ -0,0 +1,459 @@
|
|||||||
|
"""
|
||||||
|
LLM Council - A Swarms implementation inspired by Andrej Karpathy's llm-council.
|
||||||
|
|
||||||
|
This implementation creates a council of specialized LLM agents that:
|
||||||
|
1. Each agent responds to the user query independently
|
||||||
|
2. All agents review and rank each other's (anonymized) responses
|
||||||
|
3. A Chairman LLM synthesizes all responses and rankings into a final answer
|
||||||
|
|
||||||
|
The council demonstrates how different models evaluate and rank each other's work,
|
||||||
|
often selecting responses from other models as superior to their own.
|
||||||
|
"""
|
||||||
|
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
import random
|
||||||
|
from swarms import Agent
|
||||||
|
from swarms.structs.multi_agent_exec import (
|
||||||
|
run_agents_concurrently,
|
||||||
|
batched_grid_agent_execution,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def get_gpt_councilor_prompt() -> str:
|
||||||
|
"""
|
||||||
|
Get system prompt for GPT-5.1 councilor.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
System prompt string for GPT-5.1 councilor agent.
|
||||||
|
"""
|
||||||
|
return """You are a member of the LLM Council, representing GPT-5.1. Your role is to provide comprehensive, analytical, and thorough responses to user queries.
|
||||||
|
|
||||||
|
Your strengths:
|
||||||
|
- Deep analytical thinking and comprehensive coverage
|
||||||
|
- Ability to break down complex topics into detailed components
|
||||||
|
- Thorough exploration of multiple perspectives
|
||||||
|
- Rich contextual understanding
|
||||||
|
|
||||||
|
Your approach:
|
||||||
|
- Provide detailed, well-structured responses
|
||||||
|
- Include relevant context and background information
|
||||||
|
- Consider multiple angles and perspectives
|
||||||
|
- Be thorough but clear in your explanations
|
||||||
|
|
||||||
|
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on quality, depth, and clarity."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_gemini_councilor_prompt() -> str:
|
||||||
|
"""
|
||||||
|
Get system prompt for Gemini 3 Pro councilor.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
System prompt string for Gemini 3 Pro councilor agent.
|
||||||
|
"""
|
||||||
|
return """You are a member of the LLM Council, representing Gemini 3 Pro. Your role is to provide concise, well-processed, and structured responses to user queries.
|
||||||
|
|
||||||
|
Your strengths:
|
||||||
|
- Clear and structured communication
|
||||||
|
- Efficient information processing
|
||||||
|
- Condensed yet comprehensive responses
|
||||||
|
- Well-organized presentation
|
||||||
|
|
||||||
|
Your approach:
|
||||||
|
- Provide concise but complete answers
|
||||||
|
- Structure information clearly and logically
|
||||||
|
- Focus on key points without unnecessary verbosity
|
||||||
|
- Present information in an easily digestible format
|
||||||
|
|
||||||
|
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on clarity, structure, and efficiency."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_claude_councilor_prompt() -> str:
|
||||||
|
"""
|
||||||
|
Get system prompt for Claude Sonnet 4.5 councilor.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
System prompt string for Claude Sonnet 4.5 councilor agent.
|
||||||
|
"""
|
||||||
|
return """You are a member of the LLM Council, representing Claude Sonnet 4.5. Your role is to provide thoughtful, balanced, and nuanced responses to user queries.
|
||||||
|
|
||||||
|
Your strengths:
|
||||||
|
- Nuanced understanding and balanced perspectives
|
||||||
|
- Thoughtful consideration of trade-offs
|
||||||
|
- Clear reasoning and logical structure
|
||||||
|
- Ethical and responsible analysis
|
||||||
|
|
||||||
|
Your approach:
|
||||||
|
- Provide balanced, well-reasoned responses
|
||||||
|
- Consider multiple viewpoints and implications
|
||||||
|
- Be thoughtful about potential limitations or edge cases
|
||||||
|
- Maintain clarity while showing depth of thought
|
||||||
|
|
||||||
|
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on thoughtfulness, balance, and nuanced reasoning."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_grok_councilor_prompt() -> str:
|
||||||
|
"""
|
||||||
|
Get system prompt for Grok-4 councilor.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
System prompt string for Grok-4 councilor agent.
|
||||||
|
"""
|
||||||
|
return """You are a member of the LLM Council, representing Grok-4. Your role is to provide creative, innovative, and unique perspectives on user queries.
|
||||||
|
|
||||||
|
Your strengths:
|
||||||
|
- Creative problem-solving and innovative thinking
|
||||||
|
- Unique perspectives and out-of-the-box approaches
|
||||||
|
- Engaging and dynamic communication style
|
||||||
|
- Ability to connect seemingly unrelated concepts
|
||||||
|
|
||||||
|
Your approach:
|
||||||
|
- Provide creative and innovative responses
|
||||||
|
- Offer unique perspectives and fresh insights
|
||||||
|
- Be engaging and dynamic in your communication
|
||||||
|
- Think creatively while maintaining accuracy
|
||||||
|
|
||||||
|
Remember: You are part of a council where multiple AI models will respond to the same query, and then evaluate each other's responses. Focus on creativity, innovation, and unique insights."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_chairman_prompt() -> str:
|
||||||
|
"""
|
||||||
|
Get system prompt for the Chairman agent.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
System prompt string for the Chairman agent.
|
||||||
|
"""
|
||||||
|
return """You are the Chairman of the LLM Council. Your role is to synthesize responses from all council members along with their evaluations and rankings into a final, comprehensive answer.
|
||||||
|
|
||||||
|
Your responsibilities:
|
||||||
|
1. Review all council member responses to the user's query
|
||||||
|
2. Consider the rankings and evaluations provided by each council member
|
||||||
|
3. Synthesize the best elements from all responses
|
||||||
|
4. Create a final, comprehensive answer that incorporates the strengths of different approaches
|
||||||
|
5. Provide transparency about which perspectives influenced the final answer
|
||||||
|
|
||||||
|
Your approach:
|
||||||
|
- Synthesize rather than simply aggregate
|
||||||
|
- Identify the strongest elements from each response
|
||||||
|
- Create a cohesive final answer that benefits from multiple perspectives
|
||||||
|
- Acknowledge the diversity of approaches taken by council members
|
||||||
|
- Provide a balanced, comprehensive response that serves the user's needs
|
||||||
|
|
||||||
|
Remember: You have access to all original responses and all evaluations. Use this rich context to create the best possible final answer."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_evaluation_prompt(query: str, responses: Dict[str, str], evaluator_name: str) -> str:
|
||||||
|
"""
|
||||||
|
Create evaluation prompt for council members to review and rank responses.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: The original user query
|
||||||
|
responses: Dictionary mapping anonymous IDs to response texts
|
||||||
|
evaluator_name: Name of the agent doing the evaluation
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted evaluation prompt string
|
||||||
|
"""
|
||||||
|
responses_text = "\n\n".join([
|
||||||
|
f"Response {response_id}:\n{response_text}"
|
||||||
|
for response_id, response_text in responses.items()
|
||||||
|
])
|
||||||
|
|
||||||
|
return f"""You are evaluating responses from your fellow LLM Council members to the following query:
|
||||||
|
|
||||||
|
QUERY: {query}
|
||||||
|
|
||||||
|
Below are the anonymized responses from all council members (including potentially your own):
|
||||||
|
|
||||||
|
{responses_text}
|
||||||
|
|
||||||
|
Your task:
|
||||||
|
1. Carefully read and analyze each response
|
||||||
|
2. Evaluate the quality, accuracy, completeness, and usefulness of each response
|
||||||
|
3. Rank the responses from best to worst (1 = best, {len(responses)} = worst)
|
||||||
|
4. Provide brief reasoning for your rankings
|
||||||
|
5. Be honest and objective - you may find another model's response superior to your own
|
||||||
|
|
||||||
|
Format your evaluation as follows:
|
||||||
|
|
||||||
|
RANKINGS:
|
||||||
|
1. Response [ID]: [Brief reason why this is the best]
|
||||||
|
2. Response [ID]: [Brief reason]
|
||||||
|
...
|
||||||
|
{len(responses)}. Response [ID]: [Brief reason why this ranks lowest]
|
||||||
|
|
||||||
|
ADDITIONAL OBSERVATIONS:
|
||||||
|
[Any additional insights about the responses, common themes, strengths/weaknesses, etc.]
|
||||||
|
|
||||||
|
Remember: The goal is honest, objective evaluation. If another model's response is genuinely better, acknowledge it."""
|
||||||
|
|
||||||
|
|
||||||
|
def get_synthesis_prompt(
|
||||||
|
query: str,
|
||||||
|
original_responses: Dict[str, str],
|
||||||
|
evaluations: Dict[str, str],
|
||||||
|
id_to_member: Dict[str, str]
|
||||||
|
) -> str:
|
||||||
|
"""
|
||||||
|
Create synthesis prompt for the Chairman.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: Original user query
|
||||||
|
original_responses: Dict mapping member names to their responses
|
||||||
|
evaluations: Dict mapping evaluator names to their evaluation texts
|
||||||
|
id_to_member: Mapping from anonymous IDs to member names
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Formatted synthesis prompt
|
||||||
|
"""
|
||||||
|
responses_section = "\n\n".join([
|
||||||
|
f"=== {name} ===\n{response}"
|
||||||
|
for name, response in original_responses.items()
|
||||||
|
])
|
||||||
|
|
||||||
|
evaluations_section = "\n\n".join([
|
||||||
|
f"=== Evaluation by {name} ===\n{evaluation}"
|
||||||
|
for name, evaluation in evaluations.items()
|
||||||
|
])
|
||||||
|
|
||||||
|
return f"""As the Chairman of the LLM Council, synthesize the following information into a final, comprehensive answer.
|
||||||
|
|
||||||
|
ORIGINAL QUERY:
|
||||||
|
{query}
|
||||||
|
|
||||||
|
COUNCIL MEMBER RESPONSES:
|
||||||
|
{responses_section}
|
||||||
|
|
||||||
|
COUNCIL MEMBER EVALUATIONS AND RANKINGS:
|
||||||
|
{evaluations_section}
|
||||||
|
|
||||||
|
ANONYMOUS ID MAPPING (for reference):
|
||||||
|
{chr(10).join([f" {aid} = {name}" for aid, name in id_to_member.items()])}
|
||||||
|
|
||||||
|
Your task:
|
||||||
|
1. Review all council member responses
|
||||||
|
2. Consider the evaluations and rankings provided by each member
|
||||||
|
3. Identify the strongest elements from each response
|
||||||
|
4. Synthesize a final, comprehensive answer that:
|
||||||
|
- Incorporates the best insights from multiple perspectives
|
||||||
|
- Addresses the query thoroughly and accurately
|
||||||
|
- Benefits from the diversity of approaches taken
|
||||||
|
- Is clear, well-structured, and useful
|
||||||
|
|
||||||
|
Provide your final synthesized response below. You may reference which perspectives or approaches influenced different parts of your answer."""
|
||||||
|
|
||||||
|
|
||||||
|
class LLMCouncil:
|
||||||
|
"""
|
||||||
|
An LLM Council that orchestrates multiple specialized agents to collaboratively
|
||||||
|
answer queries through independent responses, peer review, and synthesis.
|
||||||
|
|
||||||
|
The council follows this workflow:
|
||||||
|
1. Dispatch query to all council members in parallel
|
||||||
|
2. Collect all responses (anonymized)
|
||||||
|
3. Have each member review and rank all responses
|
||||||
|
4. Chairman synthesizes everything into final response
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(
|
||||||
|
self,
|
||||||
|
council_members: Optional[List[Agent]] = None,
|
||||||
|
chairman_model: str = "gpt-5.1",
|
||||||
|
verbose: bool = True,
|
||||||
|
):
|
||||||
|
"""
|
||||||
|
Initialize the LLM Council.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
council_members: List of Agent instances representing council members.
|
||||||
|
If None, creates default council with GPT-5.1, Gemini 3 Pro,
|
||||||
|
Claude Sonnet 4.5, and Grok-4.
|
||||||
|
chairman_model: Model name for the Chairman agent that synthesizes responses.
|
||||||
|
verbose: Whether to print progress and intermediate results.
|
||||||
|
"""
|
||||||
|
self.verbose = verbose
|
||||||
|
|
||||||
|
# Create default council members if none provided
|
||||||
|
if council_members is None:
|
||||||
|
self.council_members = self._create_default_council()
|
||||||
|
else:
|
||||||
|
self.council_members = council_members
|
||||||
|
|
||||||
|
# Create Chairman agent
|
||||||
|
self.chairman = Agent(
|
||||||
|
agent_name="Chairman",
|
||||||
|
agent_description="Chairman of the LLM Council, responsible for synthesizing all responses and rankings into a final answer",
|
||||||
|
system_prompt=get_chairman_prompt(),
|
||||||
|
model_name=chairman_model,
|
||||||
|
max_loops=1,
|
||||||
|
verbose=verbose,
|
||||||
|
temperature=0.7,
|
||||||
|
)
|
||||||
|
|
||||||
|
if self.verbose:
|
||||||
|
print(f"🏛️ LLM Council initialized with {len(self.council_members)} members")
|
||||||
|
for i, member in enumerate(self.council_members, 1):
|
||||||
|
print(f" {i}. {member.agent_name} ({member.model_name})")
|
||||||
|
|
||||||
|
def _create_default_council(self) -> List[Agent]:
|
||||||
|
"""
|
||||||
|
Create default council members with specialized prompts and models.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of Agent instances configured as council members.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# GPT-5.1 Agent - Analytical and comprehensive
|
||||||
|
gpt_agent = Agent(
|
||||||
|
agent_name="GPT-5.1-Councilor",
|
||||||
|
agent_description="Analytical and comprehensive AI councilor specializing in deep analysis and thorough responses",
|
||||||
|
system_prompt=get_gpt_councilor_prompt(),
|
||||||
|
model_name="gpt-5.1",
|
||||||
|
max_loops=1,
|
||||||
|
verbose=False,
|
||||||
|
temperature=0.7,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Gemini 3 Pro Agent - Concise and processed
|
||||||
|
gemini_agent = Agent(
|
||||||
|
agent_name="Gemini-3-Pro-Councilor",
|
||||||
|
agent_description="Concise and well-processed AI councilor specializing in clear, structured responses",
|
||||||
|
system_prompt=get_gemini_councilor_prompt(),
|
||||||
|
model_name="gemini-2.5-flash", # Using available Gemini model
|
||||||
|
max_loops=1,
|
||||||
|
verbose=False,
|
||||||
|
temperature=0.7,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Claude Sonnet 4.5 Agent - Balanced and thoughtful
|
||||||
|
claude_agent = Agent(
|
||||||
|
agent_name="Claude-Sonnet-4.5-Councilor",
|
||||||
|
agent_description="Thoughtful and balanced AI councilor specializing in nuanced and well-reasoned responses",
|
||||||
|
system_prompt=get_claude_councilor_prompt(),
|
||||||
|
model_name="anthropic/claude-sonnet-4-5", # Using available Claude model
|
||||||
|
max_loops=1,
|
||||||
|
verbose=False,
|
||||||
|
temperature=0.0,
|
||||||
|
top_p=None,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Grok-4 Agent - Creative and innovative
|
||||||
|
grok_agent = Agent(
|
||||||
|
agent_name="Grok-4-Councilor",
|
||||||
|
agent_description="Creative and innovative AI councilor specializing in unique perspectives and creative solutions",
|
||||||
|
system_prompt=get_grok_councilor_prompt(),
|
||||||
|
model_name="x-ai/grok-4", # Using available model as proxy for Grok-4
|
||||||
|
max_loops=1,
|
||||||
|
verbose=False,
|
||||||
|
temperature=0.8,
|
||||||
|
)
|
||||||
|
|
||||||
|
members = [gpt_agent, gemini_agent, claude_agent, grok_agent]
|
||||||
|
|
||||||
|
return members
|
||||||
|
|
||||||
|
def run(self, query: str) -> Dict:
|
||||||
|
"""
|
||||||
|
Execute the full LLM Council workflow.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
query: The user's query to process
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dictionary containing:
|
||||||
|
- original_responses: Dict mapping member names to their responses
|
||||||
|
- evaluations: Dict mapping evaluator names to their rankings
|
||||||
|
- final_response: The Chairman's synthesized final answer
|
||||||
|
"""
|
||||||
|
if self.verbose:
|
||||||
|
print(f"\n{'='*80}")
|
||||||
|
print("🏛️ LLM COUNCIL SESSION")
|
||||||
|
print("="*80)
|
||||||
|
print(f"\n📝 Query: {query}\n")
|
||||||
|
|
||||||
|
# Step 1: Get responses from all council members in parallel
|
||||||
|
if self.verbose:
|
||||||
|
print("📤 Dispatching query to all council members...")
|
||||||
|
|
||||||
|
results_dict = run_agents_concurrently(
|
||||||
|
self.council_members,
|
||||||
|
task=query,
|
||||||
|
return_agent_output_dict=True
|
||||||
|
)
|
||||||
|
|
||||||
|
# Map results to member names
|
||||||
|
original_responses = {
|
||||||
|
member.agent_name: response
|
||||||
|
for member, response in zip(self.council_members,
|
||||||
|
[results_dict.get(member.agent_name, "")
|
||||||
|
for member in self.council_members])
|
||||||
|
}
|
||||||
|
|
||||||
|
if self.verbose:
|
||||||
|
print(f"✅ Received {len(original_responses)} responses\n")
|
||||||
|
for name, response in original_responses.items():
|
||||||
|
print(f" {name}: {response[:100]}...")
|
||||||
|
|
||||||
|
# Step 2: Anonymize responses for evaluation
|
||||||
|
# Create anonymous IDs (A, B, C, D, etc.)
|
||||||
|
anonymous_ids = [chr(65 + i) for i in range(len(self.council_members))]
|
||||||
|
random.shuffle(anonymous_ids) # Shuffle to ensure anonymity
|
||||||
|
|
||||||
|
anonymous_responses = {
|
||||||
|
anonymous_ids[i]: original_responses[member.agent_name]
|
||||||
|
for i, member in enumerate(self.council_members)
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create mapping from anonymous ID to member name (for later reference)
|
||||||
|
id_to_member = {
|
||||||
|
anonymous_ids[i]: member.agent_name
|
||||||
|
for i, member in enumerate(self.council_members)
|
||||||
|
}
|
||||||
|
|
||||||
|
if self.verbose:
|
||||||
|
print("\n🔍 Council members evaluating each other's responses...")
|
||||||
|
|
||||||
|
# Step 3: Have each member evaluate and rank all responses concurrently
|
||||||
|
# Create evaluation tasks for each member
|
||||||
|
evaluation_tasks = [
|
||||||
|
get_evaluation_prompt(query, anonymous_responses, member.agent_name)
|
||||||
|
for member in self.council_members
|
||||||
|
]
|
||||||
|
|
||||||
|
# Run evaluations concurrently using batched_grid_agent_execution
|
||||||
|
evaluation_results = batched_grid_agent_execution(
|
||||||
|
self.council_members,
|
||||||
|
evaluation_tasks
|
||||||
|
)
|
||||||
|
|
||||||
|
# Map results to member names
|
||||||
|
evaluations = {
|
||||||
|
member.agent_name: evaluation_results[i]
|
||||||
|
for i, member in enumerate(self.council_members)
|
||||||
|
}
|
||||||
|
|
||||||
|
if self.verbose:
|
||||||
|
print(f"✅ Received {len(evaluations)} evaluations\n")
|
||||||
|
|
||||||
|
# Step 4: Chairman synthesizes everything
|
||||||
|
if self.verbose:
|
||||||
|
print("👔 Chairman synthesizing final response...\n")
|
||||||
|
|
||||||
|
synthesis_prompt = get_synthesis_prompt(
|
||||||
|
query, original_responses, evaluations, id_to_member
|
||||||
|
)
|
||||||
|
|
||||||
|
final_response = self.chairman.run(task=synthesis_prompt)
|
||||||
|
|
||||||
|
if self.verbose:
|
||||||
|
print(f"{'='*80}")
|
||||||
|
print("✅ FINAL RESPONSE")
|
||||||
|
print(f"{'='*80}\n")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"query": query,
|
||||||
|
"original_responses": original_responses,
|
||||||
|
"evaluations": evaluations,
|
||||||
|
"final_response": final_response,
|
||||||
|
"anonymous_mapping": id_to_member,
|
||||||
|
}
|
||||||
|
|
||||||
Loading…
Reference in new issue