[FEAT][ToolStorage]

pull/552/head
Kye Gomez 5 months ago
parent cd5c74b6c7
commit 790378cc53

1
.gitignore vendored

@ -15,6 +15,7 @@ Financial-Analysis-Agent_state.json
artifacts_five artifacts_five
errors errors
chroma chroma
agent_workspace
Accounting Assistant_state.json Accounting Assistant_state.json
Unit Testing Agent_state.json Unit Testing Agent_state.json
Devin_state.json Devin_state.json

@ -168,7 +168,8 @@ nav:
- Overview: "swarms/tools/main.md" - Overview: "swarms/tools/main.md"
- What are tools?: "swarms/tools/build_tool.md" - What are tools?: "swarms/tools/build_tool.md"
- ToolAgent: "swarms/agents/tool_agent.md" - ToolAgent: "swarms/agents/tool_agent.md"
- Tool Decorator: "swarms/tools/decorator.md" # - Tool Decorator: "swarms/tools/decorator.md"
- Tool Storage: "swarms/tools/tool_storage.md"
- RAG or Long Term Memory: - RAG or Long Term Memory:
- Long Term Memory with RAG: "swarms/memory/diy_memory.md" - Long Term Memory with RAG: "swarms/memory/diy_memory.md"
- Artifacts: - Artifacts:

@ -0,0 +1,204 @@
# ToolStorage
The `ToolStorage` module provides a structured and efficient way to manage and utilize various tool functions. It is designed to store tool functions, manage settings, and ensure smooth registration and retrieval of tools. This module is particularly useful in applications that require dynamic management of a collection of functions, such as plugin systems, modular software, or any application where functions need to be registered and called dynamically.
## Class: ToolStorage
The `ToolStorage` class is the core component of the module. It provides functionalities to add, retrieve, and list tool functions as well as manage settings.
### Attributes
| Attribute | Type | Description |
|------------|--------------------|-----------------------------------------------------------------------|
| `verbose` | `bool` | A flag to enable verbose logging. |
| `tools` | `List[Callable]` | A list of tool functions. |
| `_tools` | `Dict[str, Callable]` | A dictionary that stores the tools, where the key is the tool name and the value is the tool function. |
| `_settings`| `Dict[str, Any]` | A dictionary that stores the settings, where the key is the setting name and the value is the setting value. |
### Methods
#### `__init__`
Initializes the `ToolStorage` instance.
| Parameter | Type | Default | Description |
|------------|-------------------|---------|------------------------------------------------------------|
| `verbose` | `bool` | `None` | A flag to enable verbose logging. |
| `tools` | `List[Callable]` | `None` | A list of tool functions to initialize the storage with. |
| `*args` | `tuple` | `None` | Additional positional arguments. |
| `**kwargs` | `dict` | `None` | Additional keyword arguments. |
#### `add_tool`
Adds a tool to the storage.
| Parameter | Type | Description |
|-----------|----------|------------------------------|
| `func` | `Callable` | The tool function to be added. |
**Raises:**
- `ValueError`: If a tool with the same name already exists.
#### `get_tool`
Retrieves a tool by its name.
| Parameter | Type | Description |
|-----------|--------|-------------------------------|
| `name` | `str` | The name of the tool to retrieve. |
**Returns:**
- `Callable`: The tool function.
**Raises:**
- `ValueError`: If no tool with the given name is found.
#### `set_setting`
Sets a setting in the storage.
| Parameter | Type | Description |
|-----------|--------|--------------------------|
| `key` | `str` | The key for the setting. |
| `value` | `Any` | The value for the setting. |
#### `get_setting`
Gets a setting from the storage.
| Parameter | Type | Description |
|-----------|--------|--------------------------|
| `key` | `str` | The key for the setting. |
**Returns:**
- `Any`: The value of the setting.
**Raises:**
- `KeyError`: If the setting is not found.
#### `list_tools`
Lists all registered tools.
**Returns:**
- `List[str]`: A list of tool names.
## Decorator: tool_registry
The `tool_registry` decorator registers a function as a tool in the storage.
| Parameter | Type | Description |
|-----------|----------------|----------------------------------|
| `storage` | `ToolStorage` | The storage instance to register the tool in. |
**Returns:**
- `Callable`: The decorator function.
## Usage Examples
### Full Example
```python
from swarms import ToolStorage, tool_registry
storage = ToolStorage()
# Example usage
@tool_registry(storage)
def example_tool(x: int, y: int) -> int:
"""
Example tool function that adds two numbers.
Args:
x (int): The first number.
y (int): The second number.
Returns:
int: The sum of the two numbers.
"""
return x + y
# Query all the tools and get the example tool
print(storage.list_tools()) # Should print ['example_tool']
# print(storage.get_tool('example_tool')) # Should print <function example_tool at 0x...>
# Find the tool by names and call it
print(storage.get_tool("example_tool")) # Should print 5
# Test the storage and querying
if __name__ == "__main__":
print(storage.list_tools()) # Should print ['example_tool']
print(storage.get_tool("example_tool")) # Should print 5
storage.set_setting("example_setting", 42)
print(storage.get_setting("example_setting")) # Should print 42
```
### Basic Usage
#### Example 1: Initializing ToolStorage and Adding a Tool
```python
from swarms.tools.tool_registry import ToolStorage, tool_registry
# Initialize ToolStorage
storage = ToolStorage()
# Define a tool function
@tool_registry(storage)
def add_numbers(x: int, y: int) -> int:
return x + y
# List tools
print(storage.list_tools()) # Output: ['add_numbers']
# Retrieve and use the tool
add_tool = storage.get_tool('add_numbers')
print(add_tool(5, 3)) # Output: 8
```
### Advanced Usage
#### Example 2: Managing Settings
```python
# Set a setting
storage.set_setting('max_retries', 5)
# Get a setting
max_retries = storage.get_setting('max_retries')
print(max_retries) # Output: 5
```
### Error Handling
#### Example 3: Handling Errors in Tool Retrieval
```python
try:
non_existent_tool = storage.get_tool('non_existent')
except ValueError as e:
print(e) # Output: No tool found with name: non_existent
```
#### Example 4: Handling Duplicate Tool Addition
```python
try:
@tool_registry(storage)
def add_numbers(x: int, y: int) -> int:
return x + y
except ValueError as e:
print(e) # Output: Tool with name add_numbers already exists.
```
## Conclusion
The `ToolStorage` module provides a robust solution for managing tool functions and settings. Its design allows for easy registration, retrieval, and management of tools, making it a valuable asset in various applications requiring dynamic function handling. The inclusion of detailed logging ensures that the operations are transparent and any issues can be quickly identified and resolved.

@ -17,35 +17,36 @@ agent = Agent(
agent_name="Financial-Analysis-Agent", agent_name="Financial-Analysis-Agent",
system_prompt=FINANCIAL_AGENT_SYS_PROMPT, system_prompt=FINANCIAL_AGENT_SYS_PROMPT,
llm=model, llm=model,
max_loops=2, max_loops=1,
autosave=True, autosave=True,
# dynamic_temperature_enabled=True, # dynamic_temperature_enabled=True,
dashboard=False, dashboard=False,
verbose=True, code_interpreter=True,
streaming_on=True, # verbose=True,
# interactive=True, # Set to False to disable interactive mode # streaming_on=True,
dynamic_temperature_enabled=True, # # interactive=True, # Set to False to disable interactive mode
saved_state_path="finance_agent.json", # dynamic_temperature_enabled=True,
# tools=[#Add your functions here# ], # saved_state_path="finance_agent.json",
# stopping_token="Stop!", # # tools=[#Add your functions here# ],
# interactive=True, # # stopping_token="Stop!",
# docs_folder="docs", # Enter your folder name # # interactive=True,
# pdf_path="docs/finance_agent.pdf", # # docs_folder="docs", # Enter your folder name
# sop="Calculate the profit for a company.", # # pdf_path="docs/finance_agent.pdf",
# sop_list=["Calculate the profit for a company."], # # sop="Calculate the profit for a company.",
user_name="swarms_corp", # # sop_list=["Calculate the profit for a company."],
# # docs= # user_name="swarms_corp",
# # docs_folder="docs", # # # docs=
retry_attempts=3, # # # docs_folder="docs",
# context_length=1000, # retry_attempts=3,
# tool_schema = dict # # context_length=1000,
context_length=200000, # # tool_schema = dict
# tool_schema= # context_length=200000,
# # tool_schema=
# tools # tools
# agent_ops_on=True, # agent_ops_on=True,
) )
agent.run( agent.run(
"What are the components of a startups stock incentive equity plan" "Generate the python code to fetch financial news using the OpenAI API. only return the python code"
) )

@ -0,0 +1,35 @@
from swarms.tools.tool_registry import ToolStorage, tool_registry
storage = ToolStorage()
# Example usage
@tool_registry(storage)
def example_tool(x: int, y: int) -> int:
"""
Example tool function that adds two numbers.
Args:
x (int): The first number.
y (int): The second number.
Returns:
int: The sum of the two numbers.
"""
return x + y
# Query all the tools and get the example tool
print(storage.list_tools()) # Should print ['example_tool']
# print(storage.get_tool('example_tool')) # Should print <function example_tool at 0x...>
# Find the tool by names and call it
print(storage.get_tool("example_tool")) # Should print 5
# Test the storage and querying
if __name__ == "__main__":
print(storage.list_tools()) # Should print ['example_tool']
print(storage.get_tool("example_tool")) # Should print 5
storage.set_setting("example_setting", 42)
print(storage.get_setting("example_setting")) # Should print 42

@ -45,6 +45,7 @@ class Artifact(BaseModel):
if ext.lower() not in [ if ext.lower() not in [
".py", ".py",
".csv", ".csv",
".tsv",
".txt", ".txt",
".json", ".json",
".xml", ".xml",

@ -42,6 +42,8 @@ from swarms.tools.tool_parse_exec import parse_and_execute_json
from swarms.utils.data_to_text import data_to_text from swarms.utils.data_to_text import data_to_text
from swarms.utils.parse_code import extract_code_from_markdown from swarms.utils.parse_code import extract_code_from_markdown
from swarms.utils.pdf_to_text import pdf_to_text from swarms.utils.pdf_to_text import pdf_to_text
from swarms.tools.prebuilt.code_executor import CodeExecutor
from swarms.models.popular_llms import OpenAIChat
# Utils # Utils
@ -180,7 +182,10 @@ class Agent(BaseStructure):
def __init__( def __init__(
self, self,
id: Optional[str] = agent_id, id: Optional[str] = agent_id,
llm: Optional[Any] = None, llm: Optional[Any] = OpenAIChat(
model_name="gpt-4o",
# max_tokens = 4000,
),
template: Optional[str] = None, template: Optional[str] = None,
max_loops: Optional[int] = 1, max_loops: Optional[int] = 1,
stopping_condition: Optional[Callable[[str], bool]] = None, stopping_condition: Optional[Callable[[str], bool]] = None,
@ -267,6 +272,7 @@ class Agent(BaseStructure):
frequency_penalty: float = 0.0, frequency_penalty: float = 0.0,
presence_penalty: float = 0.0, presence_penalty: float = 0.0,
temperature: float = 0.1, temperature: float = 0.1,
workspace_dir: str = "agent_workspace",
*args, *args,
**kwargs, **kwargs,
): ):
@ -358,6 +364,7 @@ class Agent(BaseStructure):
self.frequency_penalty = frequency_penalty self.frequency_penalty = frequency_penalty
self.presence_penalty = presence_penalty self.presence_penalty = presence_penalty
self.temperature = temperature self.temperature = temperature
self.workspace_dir = workspace_dir
# Name # Name
self.name = agent_name self.name = agent_name
@ -477,6 +484,17 @@ class Agent(BaseStructure):
if agent_ops_on is True: if agent_ops_on is True:
self.activate_agentops() self.activate_agentops()
# Code Executor
self.code_executor = CodeExecutor(
max_output_length=1000,
artifacts_directory=self.workspace_dir,
)
# Artifact
# self.artifact = Artifact(
# )
def set_system_prompt(self, system_prompt: str): def set_system_prompt(self, system_prompt: str):
"""Set the system prompt""" """Set the system prompt"""
self.system_prompt = system_prompt self.system_prompt = system_prompt
@ -795,8 +813,16 @@ class Agent(BaseStructure):
# self.parse_function_call_and_execute(response) # self.parse_function_call_and_execute(response)
self.parse_and_execute_tools(response) self.parse_and_execute_tools(response)
# if self.code_interpreter is not False: if self.code_interpreter is True:
# self.code_interpreter_execution(response) # Parse the code and execute
code = extract_code_from_markdown(response)
output = self.code_executor.execute(code)
# Add to memory
self.short_memory.add(
role=self.agent_name, content=output
)
if self.evaluator: if self.evaluator:
evaluated_response = self.evaluator(response) evaluated_response = self.evaluator(response)
@ -919,6 +945,32 @@ class Agent(BaseStructure):
logger.error(f"Error calling agent: {error}") logger.error(f"Error calling agent: {error}")
raise error raise error
def ingest_docs(self, docs: List[str]):
"""Ingest the documents"""
for doc in docs:
if doc.endswith(".pdf"):
text = pdf_to_text(doc)
self.short_memory.add(role=self.agent_name, content=text)
else:
with open(doc, "r") as f:
text = f.read()
self.short_memory.add(
role=self.agent_name, content=text
)
def get_docs_from_doc_folders(self):
"""Get the documents from the document folders"""
for doc in os.listdir(self.docs_folder):
if doc.endswith(".pdf"):
text = pdf_to_text(doc)
self.short_memory.add(role=self.agent_name, content=text)
else:
with open(doc, "r") as f:
text = f.read()
self.short_memory.add(
role=self.agent_name, content=text
)
def parse_and_execute_tools(self, response: str, *args, **kwargs): def parse_and_execute_tools(self, response: str, *args, **kwargs):
# Extract json from markdown # Extract json from markdown
# response = extract_code_from_markdown(response) # response = extract_code_from_markdown(response)

@ -1,11 +0,0 @@
"""
A loop is a sequence of instructions that is continually repeated until a certain condition is met. In the context of agent-based systems, a loop can be used to control the behavior of an agent, such as how many times it interacts with other agents or how long it runs for.
- Code Example:
loop = Loop(
PlanGeneratorAgent(),
ExecutionAgent(),
max_loops=10,
)
"""

@ -30,6 +30,7 @@ from swarms.tools.cohere_func_call_schema import (
CohereFuncSchema, CohereFuncSchema,
ParameterDefinition, ParameterDefinition,
) )
from swarms.tools.tool_registry import ToolStorage, tool_registry
__all__ = [ __all__ = [
"BaseTool", "BaseTool",
@ -53,4 +54,6 @@ __all__ = [
"tool_find_by_name", "tool_find_by_name",
"CohereFuncSchema", "CohereFuncSchema",
"ParameterDefinition", "ParameterDefinition",
"ToolStorage",
"tool_registry",
] ]

@ -0,0 +1,69 @@
from typing import List
from swarms.artifacts.main_artifact import Artifact, FileVersion
def artifact_save(
file_path: str,
file_type: str,
contents: str = "",
versions: List[FileVersion] = [],
edit_count: int = 0,
):
"""
Saves an artifact with the given file path, file type, contents, versions, and edit count.
Args:
file_path (str): The path of the file.
file_type (str): The type of the file.
contents (str, optional): The contents of the file. Defaults to an empty string.
versions (List[FileVersion], optional): The list of file versions. Defaults to an empty list.
edit_count (int, optional): The number of times the file has been edited. Defaults to 0.
Returns:
Artifact: The saved artifact.
"""
out = Artifact(
file_path=file_path,
file_type=file_type,
contents=contents,
versions=versions,
edit_count=edit_count,
)
out.save()
return out
def edit_artifact(
file_path: str,
file_type: str,
contents: str = "",
versions: List[FileVersion] = [],
edit_count: int = 0,
):
"""
Edits an artifact with the given file path, file type, contents, versions, and edit count.
Args:
file_path (str): The path of the file.
file_type (str): The type of the file.
contents (str, optional): The contents of the file. Defaults to an empty string.
versions (List[FileVersion], optional): The list of file versions. Defaults to an empty list.
edit_count (int, optional): The number of times the file has been edited. Defaults to 0.
Returns:
Artifact: The edited artifact.
"""
out = Artifact(
file_path=file_path,
file_type=file_type,
contents=contents,
versions=versions,
edit_count=edit_count,
)
out.edit(contents)
return out

@ -0,0 +1,132 @@
import os
import subprocess
from loguru import logger
import black
from swarms.models.tiktoken_wrapper import TikTokenizer
class CodeExecutor:
"""
A class to execute Python code and return the output as a string.
The class also logs the input and output using loguru and stores the outputs
in a folder called 'artifacts'.
Methods:
execute(code: str) -> str:
Executes the given Python code and returns the output.
"""
def __init__(
self,
max_output_length: int = 1000,
artifacts_directory: str = "artifacts",
) -> None:
"""
Initializes the CodeExecutor class and sets up the logging.
"""
self.max_output_length = max_output_length
self.artifacts_dir = artifacts_directory
os.makedirs(self.artifacts_dir, exist_ok=True)
self.setup_logging()
self.tokenizer = TikTokenizer()
def setup_logging(self) -> None:
"""
Sets up the loguru logger with colorful output.
"""
logger.add(
os.path.join(self.artifacts_dir, "code_execution.log"),
format="{time} {level} {message}",
level="DEBUG",
)
logger.info("Logger initialized and artifacts directory set up.")
def format_code(self, code: str) -> str:
"""
Formats the given Python code using black.
Args:
code (str): The Python code to format.
Returns:
str: The formatted Python code.
Raises:
ValueError: If the code cannot be formatted.
"""
try:
formatted_code = black.format_str(code, mode=black.FileMode())
return formatted_code
except black.InvalidInput as e:
logger.error(f"Error formatting code: {e}")
raise ValueError(f"Error formatting code: {e}") from e
def execute(self, code: str) -> str:
"""
Executes the given Python code and returns the output.
Args:
code (str): The Python code to execute.
Returns:
str: The output of the executed code.
Raises:
RuntimeError: If there is an error during the execution of the code.
"""
try:
formatted_code = self.format_code(code)
logger.info(f"Executing code:\n{formatted_code}")
completed_process = subprocess.run(
["python", "-c", formatted_code],
capture_output=True,
text=True,
check=True,
)
output = completed_process.stdout
logger.info(f"Code output:\n{output}")
token_count = self.tokenizer.count_tokens(output)
print(token_count)
if (
self.max_output_length
and token_count > self.max_output_length
):
logger.warning(
f"Output length exceeds {self.max_output_length} characters. Truncating output."
)
output = output[: self.max_output_length] + "..."
return output
except subprocess.CalledProcessError as e:
logger.error(f"Error executing code: {e.stderr}")
raise RuntimeError(f"Error executing code: {e.stderr}") from e
# # Example usage:
# if __name__ == "__main__":
# executor = CodeExecutor(max_output_length=300)
# code = """
# import requests
# from typing import Any
# def fetch_financial_news(api_key: str, query: str, num_articles: int) -> Any:
# try:
# url = f"https://newsapi.org/v2/everything?q={query}&apiKey={api_key}"
# response = requests.get(url)
# response.raise_for_status()
# return response.json()
# except requests.RequestException as e:
# print(f"Request Error: {e}")
# raise
# except ValueError as e:
# print(f"Value Error: {e}")
# raise
# api_key = ""
# result = fetch_financial_news(api_key, query="Nvidia news", num_articles=5)
# print(result)
# """
# result = executor.execute(code)
# print(result)

@ -0,0 +1,148 @@
from typing import Callable, List, Dict, Any
from loguru import logger
class ToolStorage:
"""
A class that represents a storage for tools.
Attributes:
verbose (bool): A flag to enable verbose logging.
tools (List[Callable]): A list of tool functions.
_tools (Dict[str, Callable]): A dictionary that stores the tools, where the key is the tool name and the value is the tool function.
_settings (Dict[str, Any]): A dictionary that stores the settings, where the key is the setting name and the value is the setting value.
"""
def __init__(
self,
verbose: bool = None,
tools: List[Callable] = None,
*args,
**kwargs,
) -> None:
self.verbose = verbose
self.tools = tools
self._tools: Dict[str, Callable] = {}
self._settings: Dict[str, Any] = {}
def add_tool(self, func: Callable) -> None:
"""
Adds a tool to the storage.
Args:
func (Callable): The tool function to be added.
Raises:
ValueError: If a tool with the same name already exists.
"""
try:
logger.info(f"Adding tool: {func.__name__}")
if func.__name__ in self._tools:
raise ValueError(
f"Tool with name {func.__name__} already exists."
)
self._tools[func.__name__] = func
logger.info(f"Added tool: {func.__name__}")
except ValueError as e:
logger.error(e)
raise
def get_tool(self, name: str) -> Callable:
"""
Retrieves a tool by its name.
Args:
name (str): The name of the tool to retrieve.
Returns:
Callable: The tool function.
Raises:
ValueError: If no tool with the given name is found.
"""
try:
logger.info(f"Getting tool: {name}")
if name not in self._tools:
raise ValueError(f"No tool found with name: {name}")
return self._tools[name]
except ValueError as e:
logger.error(e)
raise
def set_setting(self, key: str, value: Any) -> None:
"""
Sets a setting in the storage.
Args:
key (str): The key for the setting.
value (Any): The value for the setting.
"""
self._settings[key] = value
logger.info(f"Setting {key} set to {value}")
def get_setting(self, key: str) -> Any:
"""
Gets a setting from the storage.
Args:
key (str): The key for the setting.
Returns:
Any: The value of the setting.
Raises:
KeyError: If the setting is not found.
"""
try:
return self._settings[key]
except KeyError as e:
logger.error(f"Setting {key} not found error: {e}")
raise
def list_tools(self) -> List[str]:
"""
Lists all registered tools.
Returns:
List[str]: A list of tool names.
"""
return list(self._tools.keys())
# Decorator
def tool_registry(storage: ToolStorage) -> Callable:
"""
A decorator that registers a function as a tool in the storage.
Args:
storage (ToolStorage): The storage instance to register the tool in.
Returns:
Callable: The decorator function.
"""
def decorator(func: Callable) -> Callable:
logger.info(f"Registering tool: {func.__name__}")
storage.add_tool(func)
def wrapper(*args, **kwargs):
try:
result = func(*args, **kwargs)
logger.info(f"Tool {func.__name__} executed successfully")
return result
except Exception as e:
logger.error(f"Error executing tool {func.__name__}: {e}")
raise
logger.info(f"Registered tool: {func.__name__}")
return wrapper
return decorator
# # Test the storage and querying
# if __name__ == "__main__":
# print(storage.list_tools()) # Should print ['example_tool']
# # print(use_example_tool(2, 3)) # Should print 5
# storage.set_setting("example_setting", 42)
# print(storage.get_setting("example_setting")) # Should print 42

@ -1,7 +1,7 @@
import re import re
def extract_code_from_markdown(markdown_content: str): def extract_code_from_markdown(markdown_content: str) -> str:
""" """
Extracts code blocks from a Markdown string and returns them as a single string. Extracts code blocks from a Markdown string and returns them as a single string.
@ -13,14 +13,22 @@ def extract_code_from_markdown(markdown_content: str):
""" """
# Regular expression for fenced code blocks with optional language specifier # Regular expression for fenced code blocks with optional language specifier
pattern = r"```(?:\w+\n)?(.*?)```" pattern = r"```(?:\w+\n)?(.*?)```"
matches = re.findall(pattern, markdown_content, re.DOTALL)
# Find all matches of the pattern
matches = re.finditer(pattern, markdown_content, re.DOTALL)
# Extract the content inside the backticks
code_blocks = [match.group(1).strip() for match in matches]
# Concatenate all code blocks separated by newlines # Concatenate all code blocks separated by newlines
return "\n".join(code.strip() for code in matches) return "\n".join(code_blocks)
# example = """```json # example = """
# { "type": "function", "function": { "name": "fetch_financial_news", "parameters": { "query": "Nvidia news", "num_articles": 5 } } } # hello im an agent
# ```""" # ```bash
# pip install swarms
# ```
# """
# print(extract_code_from_markdown(example)) # Output: { "type": "function", "function": { "name": "fetch_financial_news", "parameters": { "query": "Nvidia news", "num_articles": 5 } } } # print(extract_code_from_markdown(example)) # Output: { "type": "function", "function": { "name": "fetch_financial_news", "parameters": { "query": "Nvidia news", "num_articles": 5 } } }

Loading…
Cancel
Save