Merge pull request #1022 from harshalmore31/agents-loader
[Feat] AgentLoader utility for loading agentspull/938/head
commit
5a2ea821d6
@ -0,0 +1,453 @@
|
||||
# AgentLoader Documentation
|
||||
|
||||
The `AgentLoader` is a powerful utility for creating Swarms agents from markdown files using the Claude Code sub-agent format. It supports both single and multiple markdown file loading, providing a flexible way to define and deploy agents using YAML frontmatter configuration.
|
||||
|
||||
## Overview
|
||||
|
||||
The AgentLoader enables you to:
|
||||
|
||||
- Load single agents from markdown files with YAML frontmatter
|
||||
- Load multiple agents from directories or file lists with concurrent processing
|
||||
- Parse Claude Code sub-agent YAML frontmatter configurations
|
||||
- Extract system prompts from markdown content
|
||||
- Utilize 100% CPU cores for high-performance batch loading
|
||||
- Provide comprehensive error handling and validation
|
||||
|
||||
## Installation
|
||||
|
||||
The AgentLoader is included with the Swarms framework:
|
||||
|
||||
```python
|
||||
from swarms.utils import AgentLoader, load_agent_from_markdown, load_agents_from_markdown
|
||||
```
|
||||
|
||||
## Markdown Format
|
||||
|
||||
The AgentLoader uses the Claude Code sub-agent YAML frontmatter format:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: your-sub-agent-name
|
||||
description: Description of when this subagent should be invoked
|
||||
model_name: gpt-4
|
||||
temperature: 0.3
|
||||
max_loops: 2
|
||||
mcp_url: http://example.com/mcp # optional
|
||||
---
|
||||
|
||||
Your subagent's system prompt goes here. This can be multiple paragraphs
|
||||
and should clearly define the subagent's role, capabilities, and approach
|
||||
to solving problems.
|
||||
|
||||
Include specific instructions, best practices, and any constraints
|
||||
the subagent should follow.
|
||||
```
|
||||
|
||||
**Schema Fields:**
|
||||
|
||||
- `name` (required): Your sub-agent name
|
||||
- `description` (required): Description of when this subagent should be invoked
|
||||
- `model_name` (optional): Name of model (defaults to random selection if not provided)
|
||||
- `temperature` (optional): Float value for model temperature (0.0-2.0)
|
||||
- `max_loops` (optional): Integer for maximum reasoning loops
|
||||
- `mcp_url` (optional): MCP server URL if needed
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Loading a Single Agent
|
||||
|
||||
```python
|
||||
from swarms.utils import load_agent_from_markdown
|
||||
|
||||
# Load agent from markdown file
|
||||
agent = load_agent_from_markdown("finance_advisor.md")
|
||||
|
||||
# Use the agent
|
||||
response = agent.run(
|
||||
"I have $10,000 to invest. What's a good strategy for a beginner?"
|
||||
)
|
||||
```
|
||||
|
||||
### Loading Multiple Agents (Concurrent)
|
||||
|
||||
```python
|
||||
from swarms.utils import load_agents_from_markdown
|
||||
|
||||
# Load agents from list of files with concurrent processing
|
||||
agents = load_agents_from_markdown([
|
||||
"market_researcher.md",
|
||||
"financial_analyst.md",
|
||||
"risk_analyst.md"
|
||||
], concurrent=True) # Uses all CPU cores for faster loading
|
||||
|
||||
# Use agents in a workflow
|
||||
from swarms.structs import SequentialWorkflow
|
||||
|
||||
workflow = SequentialWorkflow(
|
||||
agents=agents,
|
||||
max_loops=1
|
||||
)
|
||||
|
||||
task = "Analyze the AI healthcare market for a $50M investment."
|
||||
result = workflow.run(task)
|
||||
```
|
||||
|
||||
## Class-Based Usage
|
||||
|
||||
### AgentLoader Class
|
||||
|
||||
For more advanced usage, use the `AgentLoader` class directly:
|
||||
|
||||
```python
|
||||
from swarms.utils import AgentLoader
|
||||
|
||||
# Initialize loader
|
||||
loader = AgentLoader()
|
||||
|
||||
# Load single agent
|
||||
agent = loader.load_single_agent("path/to/agent.md")
|
||||
|
||||
# Load multiple agents with concurrent processing
|
||||
agents = loader.load_multiple_agents(
|
||||
"./agents_directory/",
|
||||
concurrent=True, # Enable concurrent processing
|
||||
max_workers=8 # Optional: limit worker threads
|
||||
)
|
||||
|
||||
# Parse markdown file without creating agent
|
||||
config = loader.parse_markdown_file("path/to/agent.md")
|
||||
print(config.name, config.description)
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
You can override default configuration when loading agents:
|
||||
|
||||
```python
|
||||
agent = load_agent_from_markdown(
|
||||
file_path="agent.md",
|
||||
max_loops=5,
|
||||
verbose=True,
|
||||
dashboard=True,
|
||||
autosave=False,
|
||||
context_length=200000
|
||||
)
|
||||
```
|
||||
|
||||
### Available Configuration Parameters
|
||||
|
||||
- `max_loops` (int): Maximum number of reasoning loops (default: 1)
|
||||
- `autosave` (bool): Enable automatic state saving (default: True)
|
||||
- `dashboard` (bool): Enable dashboard monitoring (default: False)
|
||||
- `verbose` (bool): Enable verbose logging (default: False)
|
||||
- `dynamic_temperature_enabled` (bool): Enable dynamic temperature (default: False)
|
||||
- `saved_state_path` (str): Path for saving agent state
|
||||
- `user_name` (str): User identifier (default: "default_user")
|
||||
- `retry_attempts` (int): Number of retry attempts (default: 3)
|
||||
- `context_length` (int): Maximum context length (default: 100000)
|
||||
- `return_step_meta` (bool): Return step metadata (default: False)
|
||||
- `output_type` (str): Output format type (default: "str")
|
||||
- `auto_generate_prompt` (bool): Auto-generate prompts (default: False)
|
||||
- `artifacts_on` (bool): Enable artifacts (default: False)
|
||||
|
||||
## Complete Example
|
||||
|
||||
### Example: Finance Advisor Agent
|
||||
|
||||
Create a file `finance_advisor.md`:
|
||||
|
||||
```markdown
|
||||
---
|
||||
name: FinanceAdvisor
|
||||
description: Expert financial advisor for investment and budgeting guidance
|
||||
model_name: gpt-4
|
||||
temperature: 0.7
|
||||
max_loops: 1
|
||||
---
|
||||
|
||||
You are an expert financial advisor with deep knowledge in:
|
||||
|
||||
- Investment strategies and portfolio management
|
||||
- Personal budgeting and financial planning
|
||||
- Risk assessment and diversification
|
||||
- Tax optimization strategies
|
||||
- Retirement planning
|
||||
|
||||
Your approach:
|
||||
|
||||
- Provide clear, actionable financial advice
|
||||
- Consider individual risk tolerance and goals
|
||||
- Explain complex concepts in simple terms
|
||||
- Always emphasize the importance of diversification
|
||||
- Include relevant disclaimers about financial advice
|
||||
|
||||
When analyzing financial situations:
|
||||
|
||||
1. Assess current financial position
|
||||
2. Identify short-term and long-term goals
|
||||
3. Evaluate risk tolerance
|
||||
4. Recommend appropriate strategies
|
||||
5. Suggest specific action steps
|
||||
```
|
||||
|
||||
### Loading and Using the Agent
|
||||
|
||||
```python
|
||||
from swarms.utils import load_agent_from_markdown
|
||||
|
||||
# Load the Finance Advisor agent
|
||||
agent = load_agent_from_markdown("finance_advisor.md")
|
||||
|
||||
# Use the agent for financial advice
|
||||
response = agent.run(
|
||||
"I have $10,000 to invest. What's a good strategy for a beginner?"
|
||||
)
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The AgentLoader provides comprehensive error handling:
|
||||
|
||||
```python
|
||||
from swarms.utils import AgentLoader
|
||||
|
||||
loader = AgentLoader()
|
||||
|
||||
try:
|
||||
# This will raise FileNotFoundError
|
||||
agent = loader.load_single_agent("nonexistent.md")
|
||||
except FileNotFoundError as e:
|
||||
print(f"File not found: {e}")
|
||||
|
||||
try:
|
||||
# This will handle parsing errors gracefully
|
||||
agents = loader.load_multiple_agents("./invalid_directory/")
|
||||
print(f"Successfully loaded {len(agents)} agents")
|
||||
except Exception as e:
|
||||
print(f"Error loading agents: {e}")
|
||||
```
|
||||
|
||||
## Concurrent Processing Features
|
||||
|
||||
### Multi-Core Performance
|
||||
|
||||
The AgentLoader utilizes 100% of CPU cores for concurrent agent loading, providing significant performance improvements when processing multiple markdown files:
|
||||
|
||||
```python
|
||||
from swarms.utils import load_agents_from_markdown
|
||||
|
||||
# Automatic concurrent processing for multiple files
|
||||
agents = load_agents_from_markdown([
|
||||
"agent1.md", "agent2.md", "agent3.md", "agent4.md"
|
||||
]) # concurrent=True by default
|
||||
|
||||
# Manual control over concurrency
|
||||
agents = load_agents_from_markdown(
|
||||
"./agents_directory/",
|
||||
concurrent=True, # Enable concurrent processing
|
||||
max_workers=8 # Limit to 8 worker threads
|
||||
)
|
||||
|
||||
# Disable concurrency for debugging or single files
|
||||
agents = load_agents_from_markdown(
|
||||
["single_agent.md"],
|
||||
concurrent=False # Sequential processing
|
||||
)
|
||||
```
|
||||
|
||||
### Resource Management
|
||||
|
||||
```python
|
||||
# Default: Uses all CPU cores
|
||||
agents = load_agents_from_markdown(files, concurrent=True)
|
||||
|
||||
# Custom worker count for resource control
|
||||
agents = load_agents_from_markdown(
|
||||
files,
|
||||
concurrent=True,
|
||||
max_workers=4 # Limit to 4 threads
|
||||
)
|
||||
|
||||
# ThreadPoolExecutor automatically manages:
|
||||
# - Thread lifecycle
|
||||
# - Resource cleanup
|
||||
# - Exception handling
|
||||
# - Result collection
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Custom System Prompt Building
|
||||
|
||||
The AgentLoader automatically builds comprehensive system prompts from the markdown structure:
|
||||
|
||||
```python
|
||||
loader = AgentLoader()
|
||||
config = loader.parse_markdown_file("agent.md")
|
||||
|
||||
# The system prompt includes:
|
||||
# - Role description from the table
|
||||
# - Focus areas as bullet points
|
||||
# - Approach as numbered steps
|
||||
# - Expected outputs as deliverables
|
||||
|
||||
print("Generated System Prompt:")
|
||||
print(config.system_prompt)
|
||||
```
|
||||
|
||||
|
||||
## Integration with Swarms
|
||||
|
||||
The loaded agents are fully compatible with Swarms orchestration systems:
|
||||
|
||||
```python
|
||||
from swarms.utils import load_agents_from_markdown
|
||||
from swarms.structs import SequentialWorkflow
|
||||
|
||||
# Load multiple specialized agents
|
||||
agents = load_agents_from_markdown("./specialist_agents/")
|
||||
|
||||
# Create a sequential workflow
|
||||
workflow = SequentialWorkflow(
|
||||
agents=agents,
|
||||
max_loops=1
|
||||
)
|
||||
|
||||
# Execute complex task across multiple agents
|
||||
result = workflow.run("Conduct a comprehensive system audit")
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Consistent Naming**: Use clear, descriptive agent names
|
||||
2. **Detailed Descriptions**: Provide comprehensive role descriptions
|
||||
3. **Structured Sections**: Use the optional sections to define agent behavior
|
||||
4. **Error Handling**: Always wrap agent loading in try-catch blocks
|
||||
5. **Model Selection**: Choose appropriate models based on agent complexity
|
||||
6. **Configuration**: Override defaults when specific behavior is needed
|
||||
|
||||
|
||||
## API Reference
|
||||
|
||||
### AgentLoader Class
|
||||
|
||||
```python
|
||||
class AgentLoader:
|
||||
def __init__(self, model: Optional[LiteLLM] = None)
|
||||
def parse_markdown_file(self, file_path: str) -> MarkdownAgentConfig
|
||||
def load_single_agent(self, file_path: str, **kwargs) -> Agent
|
||||
def load_multiple_agents(self, file_paths: Union[str, List[str]], **kwargs) -> List[Agent]
|
||||
```
|
||||
|
||||
### Convenience Functions
|
||||
|
||||
```python
|
||||
def load_agent_from_markdown(file_path: str, **kwargs) -> Agent
|
||||
def load_agents_from_markdown(
|
||||
file_paths: Union[str, List[str]],
|
||||
concurrent: bool = True, # Enable concurrent processing
|
||||
max_workers: Optional[int] = None, # Max worker threads (defaults to CPU count)
|
||||
**kwargs
|
||||
) -> List[Agent]
|
||||
```
|
||||
|
||||
### Configuration Model
|
||||
|
||||
```python
|
||||
class MarkdownAgentConfig(BaseModel):
|
||||
name: str
|
||||
description: str
|
||||
model_name: Optional[str] = "gpt-4"
|
||||
temperature: Optional[float] = 0.1 # Model temperature (0.0-2.0)
|
||||
mcp_url: Optional[str] = None # Optional MCP server URL
|
||||
system_prompt: str
|
||||
max_loops: int = 1
|
||||
autosave: bool = False
|
||||
dashboard: bool = False
|
||||
verbose: bool = False
|
||||
# ... additional configuration fields
|
||||
```
|
||||
|
||||
## Examples Repository
|
||||
|
||||
Find complete working examples in the `examples/agent_loader/` directory:
|
||||
|
||||
### Single Agent Example (`agent_loader_demo.py`)
|
||||
```python
|
||||
from swarms.utils import load_agent_from_markdown
|
||||
|
||||
agent = load_agent_from_markdown("finance_advisor.md")
|
||||
|
||||
agent.run(
|
||||
task="Analyze the financial market trends for 2023."
|
||||
)
|
||||
```
|
||||
|
||||
### Multi-Agent Workflow Example (`multi_agents_loader_demo.py`)
|
||||
```python
|
||||
from swarms.utils import load_agents_from_markdown
|
||||
|
||||
agents = load_agents_from_markdown([
|
||||
"market_researcher.md",
|
||||
"financial_analyst.md",
|
||||
"risk_analyst.md"
|
||||
])
|
||||
|
||||
# Use agents in a workflow
|
||||
from swarms.structs.sequential_workflow import SequentialWorkflow
|
||||
|
||||
workflow = SequentialWorkflow(
|
||||
agents=agents,
|
||||
max_loops=1
|
||||
)
|
||||
|
||||
task = """
|
||||
Analyze the AI healthcare market for a $50M investment opportunity.
|
||||
Focus on market size, competition, financials, and risks.
|
||||
"""
|
||||
|
||||
result = workflow.run(task)
|
||||
```
|
||||
|
||||
### Sample Agent Definition (`finance_advisor.md`)
|
||||
```markdown
|
||||
---
|
||||
name: FinanceAdvisor
|
||||
description: Expert financial advisor for investment and budgeting guidance
|
||||
model_name: gpt-4o
|
||||
temperature: 0.7
|
||||
max_loops: 1
|
||||
---
|
||||
|
||||
You are an expert financial advisor with deep knowledge in:
|
||||
|
||||
- Investment strategies and portfolio management
|
||||
- Personal budgeting and financial planning
|
||||
- Risk assessment and diversification
|
||||
- Tax optimization strategies
|
||||
- Retirement planning
|
||||
|
||||
Your approach:
|
||||
|
||||
- Provide clear, actionable financial advice
|
||||
- Consider individual risk tolerance and goals
|
||||
- Explain complex concepts in simple terms
|
||||
- Always emphasize the importance of diversification
|
||||
- Include relevant disclaimers about financial advice
|
||||
|
||||
When analyzing financial situations:
|
||||
|
||||
1. Assess current financial position
|
||||
2. Identify short-term and long-term goals
|
||||
3. Evaluate risk tolerance
|
||||
4. Recommend appropriate strategies
|
||||
5. Suggest specific action steps
|
||||
```
|
||||
|
||||
## Support
|
||||
|
||||
For questions and support:
|
||||
|
||||
- GitHub Issues: [https://github.com/kyegomez/swarms/issues](https://github.com/kyegomez/swarms/issues)
|
||||
- Documentation: [https://docs.swarms.world](https://docs.swarms.world)
|
||||
- Community: Join our Discord for real-time support
|
@ -0,0 +1,7 @@
|
||||
from swarms.utils import load_agent_from_markdown
|
||||
|
||||
agent = load_agent_from_markdown("finance_advisor.md")
|
||||
|
||||
agent.run(
|
||||
task="Analyze the financial market trends for 2023."
|
||||
)
|
@ -0,0 +1,28 @@
|
||||
---
|
||||
name: FinanceAdvisor
|
||||
description: Expert financial advisor for investment and budgeting guidance
|
||||
model_name: gpt-4o
|
||||
temperature: 0.7
|
||||
max_loops: 1
|
||||
---
|
||||
|
||||
You are an expert financial advisor with deep knowledge in:
|
||||
- Investment strategies and portfolio management
|
||||
- Personal budgeting and financial planning
|
||||
- Risk assessment and diversification
|
||||
- Tax optimization strategies
|
||||
- Retirement planning
|
||||
|
||||
Your approach:
|
||||
- Provide clear, actionable financial advice
|
||||
- Consider individual risk tolerance and goals
|
||||
- Explain complex concepts in simple terms
|
||||
- Always emphasize the importance of diversification
|
||||
- Include relevant disclaimers about financial advice
|
||||
|
||||
When analyzing financial situations:
|
||||
1. Assess current financial position
|
||||
2. Identify short-term and long-term goals
|
||||
3. Evaluate risk tolerance
|
||||
4. Recommend appropriate strategies
|
||||
5. Suggest specific action steps
|
@ -0,0 +1,22 @@
|
||||
from swarms.utils import load_agents_from_markdown
|
||||
|
||||
agents = load_agents_from_markdown([
|
||||
"market_researcher.md",
|
||||
"financial_analyst.md",
|
||||
"risk_analyst.md"
|
||||
])
|
||||
|
||||
# Example 3: Use agents in a workflow
|
||||
from swarms.structs.sequential_workflow import SequentialWorkflow
|
||||
|
||||
workflow = SequentialWorkflow(
|
||||
agents=agents,
|
||||
max_loops=1
|
||||
)
|
||||
|
||||
task = """
|
||||
Analyze the AI healthcare market for a $50M investment opportunity.
|
||||
Focus on market size, competition, financials, and risks.
|
||||
"""
|
||||
|
||||
result = workflow.run(task)
|
@ -0,0 +1,447 @@
|
||||
import os
|
||||
import yaml
|
||||
from pathlib import Path
|
||||
from typing import Any, Dict, List, Optional, Union
|
||||
from concurrent.futures import (
|
||||
ThreadPoolExecutor,
|
||||
as_completed,
|
||||
TimeoutError,
|
||||
)
|
||||
from pydantic import BaseModel, Field, field_validator
|
||||
from loguru import logger
|
||||
|
||||
# Lazy import to avoid circular dependency
|
||||
|
||||
# Default model configuration
|
||||
DEFAULT_MODEL = "gpt-4o"
|
||||
|
||||
|
||||
class MarkdownAgentConfig(BaseModel):
|
||||
"""Configuration model for agents loaded from Claude Code markdown files."""
|
||||
|
||||
name: str
|
||||
description: str
|
||||
model_name: Optional[str] = "gpt-4o"
|
||||
temperature: Optional[float] = Field(default=0.1, ge=0.0, le=2.0)
|
||||
mcp_url: Optional[int] = None
|
||||
system_prompt: str
|
||||
max_loops: int = Field(default=1, ge=1)
|
||||
autosave: bool = False
|
||||
dashboard: bool = False
|
||||
verbose: bool = False
|
||||
dynamic_temperature_enabled: bool = False
|
||||
saved_state_path: Optional[str] = None
|
||||
user_name: str = "default_user"
|
||||
retry_attempts: int = Field(default=3, ge=1)
|
||||
context_length: int = Field(default=100000, ge=1000)
|
||||
return_step_meta: bool = False
|
||||
output_type: str = "str"
|
||||
auto_generate_prompt: bool = False
|
||||
artifacts_on: bool = False
|
||||
artifacts_file_extension: str = ".md"
|
||||
artifacts_output_path: str = ""
|
||||
streaming_on: bool = False
|
||||
|
||||
@field_validator("system_prompt")
|
||||
@classmethod
|
||||
def validate_system_prompt(cls, v):
|
||||
if not v or not isinstance(v, str) or len(v.strip()) == 0:
|
||||
raise ValueError(
|
||||
"System prompt must be a non-empty string"
|
||||
)
|
||||
return v
|
||||
|
||||
|
||||
class AgentLoader:
|
||||
"""
|
||||
Loader for creating agents from markdown files using Claude Code sub-agent format.
|
||||
|
||||
Supports both single markdown file and multiple markdown files.
|
||||
Uses YAML frontmatter format for agent configuration.
|
||||
|
||||
Features:
|
||||
- Single markdown file loading
|
||||
- Multiple markdown files loading (batch processing)
|
||||
- YAML frontmatter parsing
|
||||
- Agent configuration extraction from YAML metadata
|
||||
- Error handling and validation
|
||||
"""
|
||||
|
||||
def __init__(self):
|
||||
"""
|
||||
Initialize the AgentLoader.
|
||||
"""
|
||||
pass
|
||||
|
||||
def parse_yaml_frontmatter(self, content: str) -> Dict[str, Any]:
|
||||
"""
|
||||
Parse YAML frontmatter from markdown content.
|
||||
|
||||
Args:
|
||||
content: Markdown content with potential YAML frontmatter
|
||||
|
||||
Returns:
|
||||
Dictionary with parsed YAML data and remaining content
|
||||
"""
|
||||
lines = content.split("\n")
|
||||
|
||||
# Check if content starts with YAML frontmatter
|
||||
if not lines[0].strip() == "---":
|
||||
return {"frontmatter": {}, "content": content}
|
||||
|
||||
# Find end of frontmatter
|
||||
end_marker = -1
|
||||
for i, line in enumerate(lines[1:], 1):
|
||||
if line.strip() == "---":
|
||||
end_marker = i
|
||||
break
|
||||
|
||||
if end_marker == -1:
|
||||
return {"frontmatter": {}, "content": content}
|
||||
|
||||
# Extract frontmatter and content
|
||||
frontmatter_text = "\n".join(lines[1:end_marker])
|
||||
remaining_content = "\n".join(lines[end_marker + 1 :]).strip()
|
||||
|
||||
try:
|
||||
frontmatter_data = yaml.safe_load(frontmatter_text) or {}
|
||||
except yaml.YAMLError as e:
|
||||
logger.warning(f"Failed to parse YAML frontmatter: {e}")
|
||||
return {"frontmatter": {}, "content": content}
|
||||
|
||||
return {
|
||||
"frontmatter": frontmatter_data,
|
||||
"content": remaining_content,
|
||||
}
|
||||
|
||||
def parse_markdown_file(
|
||||
self, file_path: str
|
||||
) -> MarkdownAgentConfig:
|
||||
"""
|
||||
Parse a single markdown file to extract agent configuration.
|
||||
Uses Claude Code sub-agent YAML frontmatter format.
|
||||
|
||||
Args:
|
||||
file_path: Path to markdown file
|
||||
|
||||
Returns:
|
||||
MarkdownAgentConfig object with parsed configuration
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If file doesn't exist
|
||||
ValueError: If parsing fails or no YAML frontmatter found
|
||||
"""
|
||||
if not os.path.exists(file_path):
|
||||
raise FileNotFoundError(
|
||||
f"Markdown file {file_path} not found."
|
||||
)
|
||||
|
||||
try:
|
||||
with open(file_path, "r", encoding="utf-8") as file:
|
||||
content = file.read()
|
||||
|
||||
# Parse YAML frontmatter (Claude Code sub-agent format)
|
||||
yaml_result = self.parse_yaml_frontmatter(content)
|
||||
frontmatter = yaml_result["frontmatter"]
|
||||
remaining_content = yaml_result["content"]
|
||||
|
||||
if not frontmatter:
|
||||
raise ValueError(
|
||||
f"No YAML frontmatter found in {file_path}. File must use Claude Code sub-agent format with YAML frontmatter."
|
||||
)
|
||||
|
||||
# Use YAML frontmatter data
|
||||
config_data = {
|
||||
"name": frontmatter.get("name", Path(file_path).stem),
|
||||
"description": frontmatter.get(
|
||||
"description", "Agent loaded from markdown"
|
||||
),
|
||||
"model_name": frontmatter.get("model_name")
|
||||
or frontmatter.get("model", DEFAULT_MODEL),
|
||||
"temperature": frontmatter.get("temperature", 0.1),
|
||||
"max_loops": frontmatter.get("max_loops", 1),
|
||||
"mcp_url": frontmatter.get("mcp_url"),
|
||||
"system_prompt": remaining_content.strip(),
|
||||
"streaming_on": frontmatter.get(
|
||||
"streaming_on", False
|
||||
),
|
||||
}
|
||||
|
||||
# Use default model if not specified
|
||||
if not config_data["model_name"]:
|
||||
config_data["model_name"] = DEFAULT_MODEL
|
||||
|
||||
logger.info(
|
||||
f"Successfully parsed markdown file: {file_path}"
|
||||
)
|
||||
return MarkdownAgentConfig(**config_data)
|
||||
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Error parsing markdown file {file_path}: {str(e)}"
|
||||
)
|
||||
raise ValueError(
|
||||
f"Error parsing markdown file {file_path}: {str(e)}"
|
||||
)
|
||||
|
||||
def load_agent_from_markdown(
|
||||
self, file_path: str, **kwargs
|
||||
) -> "Agent":
|
||||
"""
|
||||
Load a single agent from a markdown file.
|
||||
|
||||
Args:
|
||||
file_path: Path to markdown file
|
||||
**kwargs: Additional arguments to override default configuration
|
||||
|
||||
Returns:
|
||||
Configured Agent instance
|
||||
"""
|
||||
config = self.parse_markdown_file(file_path)
|
||||
|
||||
# Override with any provided kwargs
|
||||
config_dict = config.model_dump()
|
||||
config_dict.update(kwargs)
|
||||
|
||||
# Map config fields to Agent parameters, handling special cases
|
||||
field_mapping = {
|
||||
"name": "agent_name", # name -> agent_name
|
||||
"description": None, # not used by Agent
|
||||
"mcp_url": None, # not used by Agent
|
||||
}
|
||||
|
||||
agent_fields = {}
|
||||
for config_key, config_value in config_dict.items():
|
||||
# Handle special field mappings
|
||||
if config_key in field_mapping:
|
||||
agent_key = field_mapping[config_key]
|
||||
if agent_key: # Only include if mapped to something
|
||||
agent_fields[agent_key] = config_value
|
||||
else:
|
||||
# Direct mapping for most fields
|
||||
agent_fields[config_key] = config_value
|
||||
|
||||
try:
|
||||
# Lazy import to avoid circular dependency
|
||||
from swarms.structs.agent import Agent
|
||||
|
||||
logger.info(
|
||||
f"Creating agent '{config.name}' from {file_path}"
|
||||
)
|
||||
agent = Agent(**agent_fields)
|
||||
logger.info(
|
||||
f"Successfully created agent '{config.name}' from {file_path}"
|
||||
)
|
||||
return agent
|
||||
except Exception as e:
|
||||
import traceback
|
||||
|
||||
logger.error(
|
||||
f"Error creating agent from {file_path}: {str(e)}"
|
||||
)
|
||||
logger.error(f"Traceback: {traceback.format_exc()}")
|
||||
raise ValueError(
|
||||
f"Error creating agent from {file_path}: {str(e)}"
|
||||
)
|
||||
|
||||
def load_agents_from_markdown(
|
||||
self,
|
||||
file_paths: Union[str, List[str]],
|
||||
concurrent: bool = True,
|
||||
max_workers: Optional[int] = None,
|
||||
max_file_size_mb: float = 10.0,
|
||||
**kwargs,
|
||||
) -> List["Agent"]:
|
||||
"""
|
||||
Load multiple agents from markdown files with optional concurrent processing.
|
||||
|
||||
Args:
|
||||
file_paths: Single file path, directory path, or list of file paths
|
||||
concurrent: Whether to use concurrent processing for multiple files
|
||||
max_workers: Maximum number of worker threads (defaults to CPU count)
|
||||
max_file_size_mb: Maximum file size in MB to prevent memory issues
|
||||
**kwargs: Additional arguments to override default configuration
|
||||
|
||||
Returns:
|
||||
List of configured Agent instances
|
||||
"""
|
||||
agents = []
|
||||
paths_to_process = []
|
||||
|
||||
# Handle different input types
|
||||
if isinstance(file_paths, str):
|
||||
if os.path.isdir(file_paths):
|
||||
# Directory - find all .md files
|
||||
md_files = list(Path(file_paths).glob("*.md"))
|
||||
paths_to_process = [str(f) for f in md_files]
|
||||
elif os.path.isfile(file_paths):
|
||||
# Single file
|
||||
paths_to_process = [file_paths]
|
||||
else:
|
||||
raise FileNotFoundError(
|
||||
f"Path {file_paths} not found."
|
||||
)
|
||||
elif isinstance(file_paths, list):
|
||||
paths_to_process = file_paths
|
||||
else:
|
||||
raise ValueError(
|
||||
"file_paths must be a string or list of strings"
|
||||
)
|
||||
|
||||
# Validate file sizes to prevent memory issues
|
||||
for file_path in paths_to_process:
|
||||
try:
|
||||
file_size_mb = os.path.getsize(file_path) / (
|
||||
1024 * 1024
|
||||
)
|
||||
if file_size_mb > max_file_size_mb:
|
||||
logger.warning(
|
||||
f"Skipping {file_path}: size {file_size_mb:.2f}MB exceeds limit {max_file_size_mb}MB"
|
||||
)
|
||||
paths_to_process.remove(file_path)
|
||||
except OSError:
|
||||
logger.warning(
|
||||
f"Could not check size of {file_path}, skipping validation"
|
||||
)
|
||||
|
||||
# Adjust max_workers for I/O-bound operations
|
||||
if max_workers is None and concurrent:
|
||||
# For I/O-bound: use more threads than CPU count, but cap it
|
||||
max_workers = min(
|
||||
20, len(paths_to_process), os.cpu_count() * 2
|
||||
)
|
||||
|
||||
# Use concurrent processing for multiple files if enabled
|
||||
if concurrent and len(paths_to_process) > 1:
|
||||
logger.info(
|
||||
f"Loading {len(paths_to_process)} agents concurrently with {max_workers} workers..."
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(
|
||||
max_workers=max_workers
|
||||
) as executor:
|
||||
# Submit all tasks
|
||||
future_to_path = {
|
||||
executor.submit(
|
||||
self.load_agent_from_markdown,
|
||||
file_path,
|
||||
**kwargs,
|
||||
): file_path
|
||||
for file_path in paths_to_process
|
||||
}
|
||||
|
||||
# Collect results as they complete with timeout
|
||||
for future in as_completed(
|
||||
future_to_path, timeout=300
|
||||
): # 5 minute timeout
|
||||
file_path = future_to_path[future]
|
||||
try:
|
||||
agent = future.result(
|
||||
timeout=60
|
||||
) # 1 minute per agent
|
||||
agents.append(agent)
|
||||
logger.info(
|
||||
f"Successfully loaded agent from {file_path}"
|
||||
)
|
||||
except TimeoutError:
|
||||
logger.error(f"Timeout loading {file_path}")
|
||||
continue
|
||||
except Exception as e:
|
||||
logger.error(
|
||||
f"Failed to load {file_path}: {str(e)}"
|
||||
)
|
||||
continue
|
||||
else:
|
||||
# Sequential processing for single file or when concurrent is disabled
|
||||
logger.info(
|
||||
f"Loading {len(paths_to_process)} agents sequentially..."
|
||||
)
|
||||
for file_path in paths_to_process:
|
||||
try:
|
||||
agent = self.load_agent_from_markdown(
|
||||
file_path, **kwargs
|
||||
)
|
||||
agents.append(agent)
|
||||
except Exception as e:
|
||||
logger.warning(
|
||||
f"Skipping {file_path} due to error: {str(e)}"
|
||||
)
|
||||
continue
|
||||
|
||||
logger.info(
|
||||
f"Successfully loaded {len(agents)} agents from markdown files"
|
||||
)
|
||||
return agents
|
||||
|
||||
def load_single_agent(self, file_path: str, **kwargs) -> "Agent":
|
||||
"""
|
||||
Convenience method for loading a single agent.
|
||||
Uses Claude Code sub-agent YAML frontmatter format.
|
||||
|
||||
Args:
|
||||
file_path: Path to markdown file with YAML frontmatter
|
||||
**kwargs: Additional configuration overrides
|
||||
|
||||
Returns:
|
||||
Configured Agent instance
|
||||
"""
|
||||
return self.load_agent_from_markdown(file_path, **kwargs)
|
||||
|
||||
def load_multiple_agents(
|
||||
self, file_paths: Union[str, List[str]], **kwargs
|
||||
) -> List["Agent"]:
|
||||
"""
|
||||
Convenience method for loading multiple agents.
|
||||
Uses Claude Code sub-agent YAML frontmatter format.
|
||||
|
||||
Args:
|
||||
file_paths: Directory path or list of file paths with YAML frontmatter
|
||||
**kwargs: Additional configuration overrides
|
||||
|
||||
Returns:
|
||||
List of configured Agent instances
|
||||
"""
|
||||
return self.load_agents_from_markdown(file_paths, **kwargs)
|
||||
|
||||
|
||||
# Convenience functions
|
||||
def load_agent_from_markdown(file_path: str, **kwargs) -> "Agent":
|
||||
"""
|
||||
Load a single agent from a markdown file with Claude Code YAML frontmatter format.
|
||||
|
||||
Args:
|
||||
file_path: Path to markdown file with YAML frontmatter
|
||||
**kwargs: Additional configuration overrides
|
||||
|
||||
Returns:
|
||||
Configured Agent instance
|
||||
"""
|
||||
loader = AgentLoader()
|
||||
return loader.load_single_agent(file_path, **kwargs)
|
||||
|
||||
|
||||
def load_agents_from_markdown(
|
||||
file_paths: Union[str, List[str]],
|
||||
concurrent: bool = True,
|
||||
max_file_size_mb: float = 10.0,
|
||||
**kwargs,
|
||||
) -> List["Agent"]:
|
||||
"""
|
||||
Load multiple agents from markdown files with Claude Code YAML frontmatter format.
|
||||
|
||||
Args:
|
||||
file_paths: Directory path or list of file paths with YAML frontmatter
|
||||
concurrent: Whether to use concurrent processing for multiple files
|
||||
max_file_size_mb: Maximum file size in MB to prevent memory issues
|
||||
**kwargs: Additional configuration overrides
|
||||
|
||||
Returns:
|
||||
List of configured Agent instances
|
||||
"""
|
||||
loader = AgentLoader()
|
||||
return loader.load_agents_from_markdown(
|
||||
file_paths,
|
||||
concurrent=concurrent,
|
||||
max_file_size_mb=max_file_size_mb,
|
||||
**kwargs,
|
||||
)
|
Loading…
Reference in new issue