evelynmitchell
12 months ago
committed by
GitHub
commit
6c4e5cc2db
@ -0,0 +1,102 @@
|
||||
# Swarms Data Room
|
||||
|
||||
## Table of Contents
|
||||
|
||||
**Introduction**
|
||||
|
||||
- Overview of the Company
|
||||
|
||||
- Vision and Mission Statement
|
||||
|
||||
- Executive Summary
|
||||
|
||||
**Corporate Documents**
|
||||
|
||||
- Articles of Incorporation
|
||||
|
||||
- Bylaws
|
||||
|
||||
- Shareholder Agreements
|
||||
|
||||
- Board Meeting Minutes
|
||||
|
||||
- Company Structure and Org Chart
|
||||
|
||||
**Financial Information**
|
||||
|
||||
- Historical Financial Statements
|
||||
|
||||
- Income Statements
|
||||
|
||||
- Balance Sheets
|
||||
|
||||
- Cash Flow Statements
|
||||
|
||||
- Financial Projections and Forecasts
|
||||
|
||||
- Cap Table
|
||||
|
||||
- Funding History and Use of Funds
|
||||
|
||||
**Products and Services**
|
||||
|
||||
- Detailed Descriptions of Products/Services
|
||||
|
||||
- Product Development Roadmap
|
||||
|
||||
- User Manuals and Technical Specifications
|
||||
|
||||
- Case Studies and Use Cases
|
||||
|
||||
|
||||
## **Introdution**
|
||||
Swarms provides automation-as-a-service through swarms of autonomous agents that work together as a team. We enable our customers to build, deploy, and scale production-grade multi-agent applications to automate real-world tasks.
|
||||
|
||||
|
||||
### **Vision**
|
||||
Our vision for 2024 is to provide the most reliable infrastructure for deploying autonomous agents into the real world through the Swarm Cloud, our premier cloud platform for the scalable deployment of Multi-Modal Autonomous Agents. The platform focuses on delivering maximum value to users by only taking a small fee when utilizing the agents for the hosted compute power needed to host the agents.
|
||||
|
||||
### **Executive Summary**
|
||||
The Swarm Corporation aims to enable AI models to automate complex workflows and operations, not just singular low-value tasks. We believe collaboration between multiple agents can overcome limitations of individual agents for reasoning, planning, etc. This will allow automation of processes in mission-critical industries like security, logistics, and manufacturing where AI adoption is currently low.
|
||||
|
||||
We provide an open source framework to deploy production-grade multi-modal agents in just a few lines of code. This builds our user base, recruits talent, gets customer feedback to improve products, gains awareness and trust.
|
||||
|
||||
Our business model focuses on customer satisfaction, openness, integration with other tools/platforms, and production-grade reliability.
|
||||
|
||||
Go-to-market strategy is to get the framework to product-market fit with over 50K weekly recurring users, then secure high-value contracts in target industries. Long-term monetization via microtransactions, usage-based pricing, subscriptions.
|
||||
|
||||
The team has thousands of hours building and optimizing autonomous agents. Leadership includes AI engineers, product experts, open source contributors and community builders.
|
||||
|
||||
Key milestones: get 80K framework users in January 2024, start contracts in target verticals, introduce commercial products in 2025 with various pricing models.
|
||||
|
||||
|
||||
### **The Swarm Corporation Memo**
|
||||
To learn more about our mission, vision, plans for GTM, and much more please refer to the [Swarm Memo here](https://docs.google.com/document/d/1hS_nv_lFjCqLfnJBoF6ULY9roTbSgSuCkvXvSUSc7Lo/edit?usp=sharing)
|
||||
|
||||
|
||||
## **Product**
|
||||
Swarms is an open source framework for developers in python to enable seamless, reliable, and scalable multi-agent orchestration through modularity, customization, and precision.
|
||||
|
||||
[Here is the official Swarms Github Page:](https://github.com/kyegomez/swarms)
|
||||
|
||||
### Product Growth Metrics
|
||||
|
||||
- Total Downloads of all time: [](https://github.com/kyegomez/swarms/issues)
|
||||
|
||||
- Click here for Downloads this month: [](https://pepy.tech/project/swarms)
|
||||
|
||||
- Total Downloads this week: [](https://github.com/kyegomez/swarms/issues)
|
||||
|
||||
- Click here for Forks which represent the number of times a user has copied the entire codebase for optimization, contribution, or usage. [](https://github.com/kyegomez/swarms/network)
|
||||
|
||||
- Stars are the number of people that have liked our project, click here for more: [](https://github.com/kyegomez/swarms/stargazers)
|
||||
|
||||
- Various Project Statistics such as watchers, number of contributors, date repository was created and much more. [CLICK HERE](https://libraries.io/github/kyegomez/swarms)
|
||||
|
||||
- Contribution Based Statistics such as number of contributors, number of lines of code changed, and much more [HERE](https://github.com/kyegomez/swarms/graphs/contributors)
|
||||
|
||||
- [Github Community insights](https://github.com/kyegomez/swarms/graphs/community)
|
||||
|
||||
- [Github Traffic Metrics](https://github.com/kyegomez/swarms/graphs/traffic)
|
||||
|
||||
- Issues with the framework or Github Issues: [](https://github.com/kyegomez/swarms/issues)
|
||||
@ -0,0 +1,124 @@
|
||||
# swarms.agents
|
||||
|
||||
## 1. Introduction
|
||||
|
||||
`AbstractAgent` is an abstract class that serves as a foundation for implementing AI agents. An agent is an entity that can communicate with other agents and perform actions. The `AbstractAgent` class allows for customization in the implementation of the `receive` method, enabling different agents to define unique actions for receiving and processing messages.
|
||||
|
||||
`AbstractAgent` provides capabilities for managing tools and accessing memory, and has methods for running, chatting, and stepping through communication with other agents.
|
||||
|
||||
## 2. Class Definition
|
||||
|
||||
```python
|
||||
class AbstractAgent:
|
||||
"""An abstract class for AI agent.
|
||||
|
||||
An agent can communicate with other agents and perform actions.
|
||||
Different agents can differ in what actions they perform in the `receive` method.
|
||||
|
||||
Agents are full and completed:
|
||||
|
||||
Agents = llm + tools + memory
|
||||
"""
|
||||
|
||||
def __init__(self, name: str):
|
||||
"""
|
||||
Args:
|
||||
name (str): name of the agent.
|
||||
"""
|
||||
self._name = name
|
||||
|
||||
@property
|
||||
def name(self):
|
||||
"""Get the name of the agent."""
|
||||
return self._name
|
||||
|
||||
def tools(self, tools):
|
||||
"""init tools"""
|
||||
|
||||
def memory(self, memory_store):
|
||||
"""init memory"""
|
||||
pass
|
||||
|
||||
def reset(self):
|
||||
"""(Abstract method) Reset the agent."""
|
||||
|
||||
def run(self, task: str):
|
||||
"""Run the agent once"""
|
||||
|
||||
def _arun(self, taks: str):
|
||||
"""Run Async run"""
|
||||
|
||||
def chat(self, messages: List[Dict]):
|
||||
"""Chat with the agent"""
|
||||
|
||||
def _achat(self, messages: List[Dict]):
|
||||
"""Asynchronous Chat"""
|
||||
|
||||
def step(self, message: str):
|
||||
"""Step through the agent"""
|
||||
|
||||
def _astep(self, message: str):
|
||||
"""Asynchronous step"""
|
||||
```
|
||||
|
||||
## 3. Functionality and Usage
|
||||
|
||||
The `AbstractAgent` class represents a generic AI agent and provides a set of methods to interact with it.
|
||||
|
||||
To create an instance of an agent, the `name` of the agent should be specified.
|
||||
|
||||
### Core Methods
|
||||
|
||||
#### 1. `reset`
|
||||
|
||||
The `reset` method allows the agent to be reset to its initial state.
|
||||
|
||||
```python
|
||||
agent.reset()
|
||||
```
|
||||
|
||||
#### 2. `run`
|
||||
|
||||
The `run` method allows the agent to perform a specific task.
|
||||
|
||||
```python
|
||||
agent.run('some_task')
|
||||
```
|
||||
|
||||
#### 3. `chat`
|
||||
|
||||
The `chat` method enables communication with the agent through a series of messages.
|
||||
|
||||
```python
|
||||
messages = [{'id': 1, 'text': 'Hello, agent!'}, {'id': 2, 'text': 'How are you?'}]
|
||||
agent.chat(messages)
|
||||
```
|
||||
|
||||
#### 4. `step`
|
||||
|
||||
The `step` method allows the agent to process a single message.
|
||||
|
||||
```python
|
||||
agent.step('Hello, agent!')
|
||||
```
|
||||
|
||||
### Asynchronous Methods
|
||||
|
||||
The class also provides asynchronous variants of the core methods.
|
||||
|
||||
### Additional Functionality
|
||||
|
||||
Additional functionalities for agent initialization and management of tools and memory are also provided.
|
||||
|
||||
```python
|
||||
agent.tools(some_tools)
|
||||
agent.memory(some_memory_store)
|
||||
```
|
||||
|
||||
## 4. Additional Information and Tips
|
||||
|
||||
When implementing a new agent using the `AbstractAgent` class, ensure that the `receive` method is overridden to define the specific behavior of the agent upon receiving messages.
|
||||
|
||||
## 5. References and Resources
|
||||
|
||||
For further exploration and understanding of AI agents and agent communication, refer to the relevant literature and research on this topic.
|
||||
@ -0,0 +1,120 @@
|
||||
# The Module/Class Name: Message
|
||||
|
||||
In the swarms.agents framework, the class `Message` is used to represent a message with timestamp and optional metadata.
|
||||
|
||||
## Overview and Introduction
|
||||
|
||||
The `Message` class is a fundamental component that enables the representation of messages within an agent system. Messages contain essential information such as the sender, content, timestamp, and optional metadata.
|
||||
|
||||
## Class Definition
|
||||
|
||||
### Constructor: `__init__`
|
||||
|
||||
The constructor of the `Message` class takes three parameters:
|
||||
|
||||
1. `sender` (str): The sender of the message.
|
||||
2. `content` (str): The content of the message.
|
||||
3. `metadata` (dict or None): Optional metadata associated with the message.
|
||||
|
||||
### Methods
|
||||
|
||||
1. `__repr__(self)`: Returns a string representation of the `Message` object, including the timestamp, sender, and content.
|
||||
|
||||
```python
|
||||
class Message:
|
||||
"""
|
||||
Represents a message with timestamp and optional metadata.
|
||||
|
||||
Usage
|
||||
--------------
|
||||
mes = Message(
|
||||
sender = "Kye",
|
||||
content = "message"
|
||||
)
|
||||
|
||||
print(mes)
|
||||
"""
|
||||
|
||||
def __init__(self, sender, content, metadata=None):
|
||||
self.timestamp = datetime.datetime.now()
|
||||
self.sender = sender
|
||||
self.content = content
|
||||
self.metadata = metadata or {}
|
||||
|
||||
def __repr__(self):
|
||||
"""
|
||||
__repr__ represents the string representation of the Message object.
|
||||
|
||||
Returns:
|
||||
(str) A string containing the timestamp, sender, and content of the message.
|
||||
"""
|
||||
return f"{self.timestamp} - {self.sender}: {self.content}"
|
||||
```
|
||||
|
||||
## Functionality and Usage
|
||||
|
||||
The `Message` class represents a message in the agent system. Upon initialization, the `timestamp` is set to the current date and time, and the `metadata` is set to an empty dictionary if no metadata is provided.
|
||||
|
||||
### Usage Example 1
|
||||
|
||||
Creating a `Message` object and displaying its string representation.
|
||||
|
||||
```python
|
||||
mes = Message(
|
||||
sender = "Kye",
|
||||
content = "Hello! How are you?"
|
||||
)
|
||||
|
||||
print(mes)
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
2023-09-20 13:45:00 - Kye: Hello! How are you?
|
||||
```
|
||||
|
||||
### Usage Example 2
|
||||
|
||||
Creating a `Message` object with metadata.
|
||||
|
||||
```python
|
||||
metadata = {"priority": "high", "category": "urgent"}
|
||||
mes_with_metadata = Message(
|
||||
sender = "Alice",
|
||||
content = "Important update",
|
||||
metadata = metadata
|
||||
)
|
||||
|
||||
print(mes_with_metadata)
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
2023-09-20 13:46:00 - Alice: Important update
|
||||
```
|
||||
|
||||
### Usage Example 3
|
||||
|
||||
Creating a `Message` object without providing metadata.
|
||||
|
||||
```python
|
||||
mes_no_metadata = Message(
|
||||
sender = "Bob",
|
||||
content = "Reminder: Meeting at 2PM"
|
||||
)
|
||||
|
||||
print(mes_no_metadata)
|
||||
```
|
||||
|
||||
Output:
|
||||
```
|
||||
2023-09-20 13:47:00 - Bob: Reminder: Meeting at 2PM
|
||||
```
|
||||
|
||||
## Additional Information and Tips
|
||||
|
||||
When creating a new `Message` object, ensure that the required parameters `sender` and `content` are provided. The `timestamp` will automatically be assigned the current date and time. Optional `metadata` can be included to provide additional context or information associated with the message.
|
||||
|
||||
## References and Resources
|
||||
|
||||
For further information on the `Message` class and its usage, refer to the official swarms.agents documentation and relevant tutorials related to message handling and communication within the agent system.
|
||||
@ -0,0 +1,79 @@
|
||||
# Module/Class Name: OmniModalAgent
|
||||
|
||||
The `OmniModalAgent` class is a module that operates based on the Language Model (LLM) aka Language Understanding Model, Plans, Tasks, and Tools. It is designed to be a multi-modal chatbot which uses various AI-based capabilities for fulfilling user requests.
|
||||
|
||||
It has the following architecture:
|
||||
1. Language Model (LLM).
|
||||
2. Chat Planner - Plans
|
||||
3. Task Executor - Tasks
|
||||
4. Tools - Tools
|
||||
|
||||
|
||||
|
||||
---
|
||||
|
||||
### Usage
|
||||
from swarms import OmniModalAgent, OpenAIChat
|
||||
|
||||
llm = OpenAIChat()
|
||||
agent = OmniModalAgent(llm)
|
||||
response = agent.run("Hello, how are you? Create an image of how your are doing!")
|
||||
|
||||
---
|
||||
|
||||
---
|
||||
|
||||
### Initialization
|
||||
|
||||
The constructor of `OmniModalAgent` class takes two main parameters:
|
||||
- `llm`: A `BaseLanguageModel` that represents the language model
|
||||
- `tools`: A List of `BaseTool` instances that are used by the agent for fulfilling different requests.
|
||||
|
||||
```python
|
||||
def __init__(
|
||||
self,
|
||||
llm: BaseLanguageModel,
|
||||
# tools: List[BaseTool]
|
||||
):
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Methods
|
||||
|
||||
The class has two main methods:
|
||||
1. `run`: This method takes an input string and executes various plans and tasks using the provided tools. Ultimately, it generates a response based on the user's input and returns it.
|
||||
- Parameters:
|
||||
- `input`: A string representing the user's input text.
|
||||
- Returns:
|
||||
- A string representing the response.
|
||||
|
||||
Usage:
|
||||
```python
|
||||
response = agent.run("Hello, how are you? Create an image of how your are doing!")
|
||||
```
|
||||
|
||||
2. `chat`: This method is used to simulate a chat dialog with the agent. It can take user's messages and return the response (or stream the response word-by-word if required).
|
||||
- Parameters:
|
||||
- `msg` (optional): A string representing the message to send to the agent.
|
||||
- `streaming` (optional): A boolean specifying whether to stream the response.
|
||||
- Returns:
|
||||
- A string representing the response from the agent.
|
||||
|
||||
Usage:
|
||||
```python
|
||||
response = agent.chat("Hello")
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Streaming Response
|
||||
|
||||
The class provides a method `_stream_response` that can be used to get the response token by token (i.e. word by word). It yields individual tokens from the response.
|
||||
|
||||
Usage:
|
||||
```python
|
||||
for token in _stream_response(response):
|
||||
print(token)
|
||||
```
|
||||
|
||||
@ -0,0 +1,113 @@
|
||||
# ToolAgent Documentation
|
||||
|
||||
|
||||
### Overview and Introduction
|
||||
|
||||
The `ToolAgent` class represents an intelligent agent capable of performing a specific task using a pre-trained model and tokenizer. It leverages the Transformer models of the Hugging Face `transformers` library to generate outputs that adhere to a specific JSON schema. This provides developers with a flexible tool for creating bots, text generators, and conversational AI agents. The `ToolAgent` operates based on a JSON schema provided by you, the user. Using the schema, the agent applies the provided model and tokenizer to generate structured text data that matches the specified format.
|
||||
|
||||
The primary objective of the `ToolAgent` class is to amplify the efficiency of developers and AI practitioners by simplifying the process of generating meaningful outputs that navigate the complexities of the model and tokenizer.
|
||||
|
||||
### Class Definition
|
||||
|
||||
The `ToolAgent` class has the following definition:
|
||||
|
||||
```python
|
||||
class ToolAgent(AbstractLLM):
|
||||
def __init__(
|
||||
self,
|
||||
name: str,
|
||||
description: str,
|
||||
model: Any,
|
||||
tokenizer: Any,
|
||||
json_schema: Any,
|
||||
*args,
|
||||
**kwargs,
|
||||
)
|
||||
def run(self, task: str, *args, **kwargs)
|
||||
def __call__(self, task: str, *args, **kwargs)
|
||||
```
|
||||
|
||||
### Arguments
|
||||
|
||||
The `ToolAgent` class takes the following arguments:
|
||||
|
||||
| Argument | Type | Description |
|
||||
| --- | --- | --- |
|
||||
| name | str | The name of the tool agent.
|
||||
| description | str | A description of the tool agent.
|
||||
| model | Any | The model used by the tool agent (e.g., `transformers.AutoModelForCausalLM`).
|
||||
| tokenizer | Any | The tokenizer used by the tool agent (e.g., `transformers.AutoTokenizer`).
|
||||
| json_schema | Any | The JSON schema used by the tool agent.
|
||||
| *args | - | Variable-length arguments.
|
||||
| **kwargs | - | Keyword arguments.
|
||||
|
||||
### Methods
|
||||
|
||||
`ToolAgent` exposes the following methods:
|
||||
|
||||
#### `run(self, task: str, *args, **kwargs) -> Any`
|
||||
|
||||
- Description: Runs the tool agent for a specific task.
|
||||
- Parameters:
|
||||
- `task` (str): The task to be performed by the tool agent.
|
||||
- `*args`: Variable-length argument list.
|
||||
- `**kwargs`: Arbitrary keyword arguments.
|
||||
- Returns: The output of the tool agent.
|
||||
- Raises: Exception if an error occurs during the execution of the tool agent.
|
||||
|
||||
|
||||
#### `__call__(self, task: str, *args, **kwargs) -> Any`
|
||||
|
||||
- Description: Calls the tool agent to perform a specific task.
|
||||
- Parameters:
|
||||
- `task` (str): The task to be performed by the tool agent.
|
||||
- `*args`: Variable-length argument list.
|
||||
- `**kwargs`: Arbitrary keyword arguments.
|
||||
- Returns: The output of the tool agent.
|
||||
|
||||
### Usage Example
|
||||
|
||||
```python
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
from swarms import ToolAgent
|
||||
|
||||
# Creating a model and tokenizer
|
||||
model = AutoModelForCausalLM.from_pretrained("databricks/dolly-v2-12b")
|
||||
tokenizer = AutoTokenizer.from_pretrained("databricks/dolly-v2-12b")
|
||||
|
||||
# Defining a JSON schema
|
||||
json_schema = {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"name": {"type": "string"},
|
||||
"age": {"type": "number"},
|
||||
"is_student": {"type": "boolean"},
|
||||
"courses": {
|
||||
"type": "array",
|
||||
"items": {"type": "string"}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Defining a task
|
||||
task = "Generate a person's information based on the following schema:"
|
||||
|
||||
# Creating the ToolAgent instance
|
||||
agent = ToolAgent(model=model, tokenizer=tokenizer, json_schema=json_schema)
|
||||
|
||||
# Running the tool agent
|
||||
generated_data = agent.run(task)
|
||||
|
||||
# Accessing and printing the generated data
|
||||
print(generated_data)
|
||||
```
|
||||
|
||||
### Additional Information and Tips
|
||||
|
||||
When using the `ToolAgent`, it is important to ensure compatibility between the provided model, tokenizer, and the JSON schema. Additionally, any errors encountered during the execution of the tool agent are propagated as exceptions. Handling such exceptions appropriately can improve the robustness of the tool agent usage.
|
||||
|
||||
### References and Resources
|
||||
|
||||
For further exploration and understanding of the underlying Transformer-based models and tokenizers, refer to the Hugging Face `transformers` library documentation and examples. Additionally, for JSON schema modeling, you can refer to the official JSON Schema specification and examples.
|
||||
|
||||
This documentation provides a comprehensive guide on using the `ToolAgent` class from `swarms` library, and it is recommended to refer back to this document when utilizing the `ToolAgent` for developing your custom conversational agents or text generation tools.
|
||||
@ -0,0 +1,78 @@
|
||||
# WorkerClass Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The Worker class represents an autonomous agent that can perform tasks through function calls or by running a chat. It can be used to create applications that demand effective user interactions like search engines, human-like conversational bots, or digital assistants.
|
||||
|
||||
The `Worker` class is part of the `swarms.agents` codebase. This module is largely used in Natural Language Processing (NLP) projects where the agent undertakes conversations and other language-specific operations.
|
||||
|
||||
## Class Definition
|
||||
|
||||
The class `Worker` has the following arguments:
|
||||
|
||||
| Argument | Type | Default Value | Description |
|
||||
|-----------------------|---------------|----------------------------------|----------------------------------------------------|
|
||||
| name | str | "Worker" | Name of the agent. |
|
||||
| role | str | "Worker in a swarm" | Role of the agent. |
|
||||
| external_tools | list | None | List of external tools available to the agent. |
|
||||
| human_in_the_loop | bool | False | Determines whether human interaction is required. |
|
||||
| temperature | float | 0.5 | Temperature for the autonomous agent. |
|
||||
| llm | None | None | Language model. |
|
||||
| openai_api_key | str | None | OpenAI API key. |
|
||||
| tools | List[Any] | None | List of tools available to the agent. |
|
||||
| embedding_size | int | 1536 | Size of the word embeddings. |
|
||||
| search_kwargs | dict | {"k": 8} | Search parameters. |
|
||||
| args | Multiple | | Additional arguments that can be passed. |
|
||||
| kwargs | Multiple | | Additional keyword arguments that can be passed. |
|
||||
## Usage
|
||||
|
||||
#### Example 1: Creating and Running an Agent
|
||||
|
||||
```python
|
||||
from swarms import Worker
|
||||
|
||||
worker = Worker(
|
||||
name="My Worker",
|
||||
role="Worker",
|
||||
external_tools=[MyTool1(), MyTool2()],
|
||||
human_in_the_loop=False,
|
||||
temperature=0.5,
|
||||
llm=some_language_model,
|
||||
openai_api_key="my_key"
|
||||
)
|
||||
worker.run("What's the weather in Miami?")
|
||||
```
|
||||
|
||||
#### Example 2: Receiving and Sending Messages
|
||||
|
||||
```python
|
||||
worker.receieve("User", "Hello there!")
|
||||
worker.receieve("User", "Can you tell me something about history?")
|
||||
worker.send()
|
||||
```
|
||||
|
||||
#### Example 3: Setting up Tools
|
||||
|
||||
```python
|
||||
external_tools = [MyTool1(), MyTool2()]
|
||||
worker = Worker(
|
||||
name="My Worker",
|
||||
role="Worker",
|
||||
external_tools=external_tools,
|
||||
human_in_the_loop=False,
|
||||
temperature=0.5,
|
||||
)
|
||||
```
|
||||
|
||||
## Additional Information and Tips
|
||||
|
||||
- The class allows the setting up of tools for the worker to operate effectively. It provides setup facilities for essential computing infrastructure, such as the agent's memory and language model.
|
||||
- By setting the `human_in_the_loop` parameter to True, interactions with the worker can be made more user-centric.
|
||||
- The `openai_api_key` argument can be provided for leveraging the OpenAI infrastructure and services.
|
||||
- A qualified language model can be passed as an instance of the `llm` object, which can be useful when integrating with state-of-the-art text generation engines.
|
||||
|
||||
## References and Resources
|
||||
|
||||
- [OpenAI APIs](https://openai.com)
|
||||
- [Models and Languages at HuggingFace](https://huggingface.co/models)
|
||||
- [Deep Learning and Language Modeling at the Allen Institute for AI](https://allenai.org)
|
||||
@ -0,0 +1,16 @@
|
||||
from swarms import QwenVLMultiModal
|
||||
|
||||
# Instantiate the QwenVLMultiModal model
|
||||
model = QwenVLMultiModal(
|
||||
model_name="Qwen/Qwen-VL-Chat",
|
||||
device="cuda",
|
||||
quantize=True,
|
||||
)
|
||||
|
||||
# Run the model
|
||||
response = model(
|
||||
"Hello, how are you?", "https://example.com/image.jpg"
|
||||
)
|
||||
|
||||
# Print the response
|
||||
print(response)
|
||||
@ -0,0 +1,38 @@
|
||||
# Example
|
||||
|
||||
import os
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from swarms import Agent, OpenAIChat
|
||||
from swarms.structs.company import Company
|
||||
|
||||
load_dotenv()
|
||||
|
||||
llm = OpenAIChat(
|
||||
openai_api_key=os.getenv("OPENAI_API_KEY"), max_tokens=4000
|
||||
)
|
||||
|
||||
ceo = Agent(llm=llm, ai_name="CEO")
|
||||
dev = Agent(llm=llm, ai_name="Developer")
|
||||
va = Agent(llm=llm, ai_name="VA")
|
||||
|
||||
# Create a company
|
||||
company = Company(
|
||||
org_chart=[[dev, va]],
|
||||
shared_instructions="Do your best",
|
||||
ceo=ceo,
|
||||
)
|
||||
|
||||
# Add agents to the company
|
||||
hr = Agent(llm=llm, name="HR")
|
||||
company.add(hr)
|
||||
|
||||
# Get an agent from the company
|
||||
hr = company.get("CEO")
|
||||
|
||||
# Remove an agent from the company
|
||||
company.remove(hr)
|
||||
|
||||
# Run the company
|
||||
company.run()
|
||||
@ -1,13 +1,35 @@
|
||||
from swarms.agents.message import Message
|
||||
from swarms.agents.base import AbstractAgent
|
||||
from swarms.agents.tool_agent import ToolAgent
|
||||
from swarms.agents.simple_agent import SimpleAgent
|
||||
from swarms.agents.omni_modal_agent import OmniModalAgent
|
||||
from swarms.agents.simple_agent import SimpleAgent
|
||||
from swarms.agents.stopping_conditions import (
|
||||
check_cancelled,
|
||||
check_complete,
|
||||
check_done,
|
||||
check_end,
|
||||
check_error,
|
||||
check_exit,
|
||||
check_failure,
|
||||
check_finished,
|
||||
check_stopped,
|
||||
check_success,
|
||||
)
|
||||
from swarms.agents.tool_agent import ToolAgent
|
||||
from swarms.agents.worker_agent import Worker
|
||||
|
||||
__all__ = [
|
||||
"Message",
|
||||
"AbstractAgent",
|
||||
"ToolAgent",
|
||||
"SimpleAgent",
|
||||
"OmniModalAgent",
|
||||
"check_done",
|
||||
"check_finished",
|
||||
"check_complete",
|
||||
"check_success",
|
||||
"check_failure",
|
||||
"check_error",
|
||||
"check_stopped",
|
||||
"check_cancelled",
|
||||
"check_exit",
|
||||
"check_end",
|
||||
"Worker",
|
||||
]
|
||||
|
||||
@ -0,0 +1,38 @@
|
||||
def check_done(s):
|
||||
return "<DONE>" in s
|
||||
|
||||
|
||||
def check_finished(s):
|
||||
return "finished" in s
|
||||
|
||||
|
||||
def check_complete(s):
|
||||
return "complete" in s
|
||||
|
||||
|
||||
def check_success(s):
|
||||
return "success" in s
|
||||
|
||||
|
||||
def check_failure(s):
|
||||
return "failure" in s
|
||||
|
||||
|
||||
def check_error(s):
|
||||
return "error" in s
|
||||
|
||||
|
||||
def check_stopped(s):
|
||||
return "stopped" in s
|
||||
|
||||
|
||||
def check_cancelled(s):
|
||||
return "cancelled" in s
|
||||
|
||||
|
||||
def check_exit(s):
|
||||
return "exit" in s
|
||||
|
||||
|
||||
def check_end(s):
|
||||
return "end" in s
|
||||
@ -1,82 +1,82 @@
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer, pipeline
|
||||
import requests
|
||||
from PIL import Image
|
||||
from transformers import AutoProcessor, LlavaForConditionalGeneration
|
||||
from typing import Tuple, Union
|
||||
from io import BytesIO
|
||||
from swarms.models.base_multimodal_model import BaseMultiModalModel
|
||||
|
||||
|
||||
class MultiModalLlava:
|
||||
class LavaMultiModal(BaseMultiModalModel):
|
||||
"""
|
||||
LLava Model
|
||||
A class to handle multi-modal inputs (text and image) using the Llava model for conditional generation.
|
||||
|
||||
Attributes:
|
||||
model_name (str): The name or path of the pre-trained model.
|
||||
max_length (int): The maximum length of the generated sequence.
|
||||
|
||||
Args:
|
||||
model_name_or_path: The model name or path to the model
|
||||
revision: The revision of the model to use
|
||||
device: The device to run the model on
|
||||
max_new_tokens: The maximum number of tokens to generate
|
||||
do_sample: Whether or not to use sampling
|
||||
temperature: The temperature of the sampling
|
||||
top_p: The top p value for sampling
|
||||
top_k: The top k value for sampling
|
||||
repetition_penalty: The repetition penalty for sampling
|
||||
device_map: The device map to use
|
||||
model_name (str): The name of the pre-trained model.
|
||||
max_length (int): The maximum length of the generated sequence.
|
||||
*args: Additional positional arguments.
|
||||
**kwargs: Additional keyword arguments.
|
||||
|
||||
Methods:
|
||||
__call__: Call the model
|
||||
chat: Interactive chat in terminal
|
||||
Examples:
|
||||
>>> model = LavaMultiModal()
|
||||
>>> model.run("A cat", "https://example.com/cat.jpg")
|
||||
|
||||
Example:
|
||||
>>> from swarms.models.llava import LlavaModel
|
||||
>>> model = LlavaModel(device="cpu")
|
||||
>>> model("Hello, I am a robot.")
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name_or_path="TheBloke/llava-v1.5-13B-GPTQ",
|
||||
revision="main",
|
||||
device="cuda",
|
||||
max_new_tokens=512,
|
||||
do_sample=True,
|
||||
temperature=0.7,
|
||||
top_p=0.95,
|
||||
top_k=40,
|
||||
repetition_penalty=1.1,
|
||||
device_map: str = "auto",
|
||||
):
|
||||
self.device = device
|
||||
self.model = AutoModelForCausalLM.from_pretrained(
|
||||
model_name_or_path,
|
||||
device_map=device_map,
|
||||
trust_remote_code=False,
|
||||
revision=revision,
|
||||
).to(self.device)
|
||||
model_name: str = "llava-hf/llava-1.5-7b-hf",
|
||||
max_length: int = 30,
|
||||
*args,
|
||||
**kwargs,
|
||||
) -> None:
|
||||
super().__init__(*args, **kwargs)
|
||||
self.model_name = model_name
|
||||
self.max_length = max_length
|
||||
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(
|
||||
model_name_or_path, use_fast=True
|
||||
)
|
||||
self.pipe = pipeline(
|
||||
"text-generation",
|
||||
model=self.model,
|
||||
tokenizer=self.tokenizer,
|
||||
max_new_tokens=max_new_tokens,
|
||||
do_sample=do_sample,
|
||||
temperature=temperature,
|
||||
top_p=top_p,
|
||||
top_k=top_k,
|
||||
repetition_penalty=repetition_penalty,
|
||||
device=0 if self.device == "cuda" else -1,
|
||||
self.model = LlavaForConditionalGeneration.from_pretrained(
|
||||
model_name, *args, **kwargs
|
||||
)
|
||||
self.processor = AutoProcessor.from_pretrained(model_name)
|
||||
|
||||
def __call__(self, prompt):
|
||||
"""Call the model"""
|
||||
return self.pipe(prompt)[0]["generated_text"]
|
||||
def run(
|
||||
self, text: str, img: str, *args, **kwargs
|
||||
) -> Union[str, Tuple[None, str]]:
|
||||
"""
|
||||
Processes the input text and image, and generates a response.
|
||||
|
||||
def chat(self):
|
||||
"""Interactive chat in terminal"""
|
||||
print(
|
||||
"Starting chat with LlavaModel. Type 'exit' to end the"
|
||||
" session."
|
||||
)
|
||||
while True:
|
||||
user_input = input("You: ")
|
||||
if user_input.lower() == "exit":
|
||||
break
|
||||
response = self(user_input)
|
||||
print(f"Model: {response}")
|
||||
Args:
|
||||
text (str): The input text for the model.
|
||||
img (str): The URL of the image to process.
|
||||
max_length (int): The maximum length of the generated sequence.
|
||||
|
||||
Returns:
|
||||
Union[str, Tuple[None, str]]: The generated response string or a tuple (None, error message) in case of an error.
|
||||
"""
|
||||
try:
|
||||
response = requests.get(img, stream=True)
|
||||
response.raise_for_status()
|
||||
image = Image.open(BytesIO(response.content))
|
||||
|
||||
inputs = self.processor(
|
||||
text=text, images=image, return_tensors="pt"
|
||||
)
|
||||
|
||||
# Generate
|
||||
generate_ids = self.model.generate(
|
||||
**inputs, max_length=self.max_length, **kwargs
|
||||
)
|
||||
return self.processor.batch_decode(
|
||||
generate_ids,
|
||||
skip_special_tokens=True,
|
||||
clean_up_tokenization_spaces=False,
|
||||
*args,
|
||||
)[0]
|
||||
|
||||
except requests.RequestException as e:
|
||||
return None, f"Error fetching image: {str(e)}"
|
||||
except Exception as e:
|
||||
return None, f"Error during model processing: {str(e)}"
|
||||
|
||||
@ -0,0 +1,144 @@
|
||||
import os
|
||||
from dataclasses import dataclass
|
||||
from typing import Tuple
|
||||
|
||||
import numpy as np
|
||||
import requests
|
||||
import torch
|
||||
import torch.nn.functional as F
|
||||
from skimage import transform
|
||||
from torch import Tensor
|
||||
|
||||
|
||||
def sam_model_registry():
|
||||
pass
|
||||
|
||||
|
||||
@dataclass
|
||||
class MedicalSAM:
|
||||
"""
|
||||
MedicalSAM class for performing semantic segmentation on medical images using the SAM model.
|
||||
|
||||
Attributes:
|
||||
model_path (str): The file path to the model weights.
|
||||
device (str): The device to run the model on (default is "cuda:0").
|
||||
model_weights_url (str): The URL to download the model weights from.
|
||||
|
||||
Methods:
|
||||
__post_init__(): Initializes the MedicalSAM object.
|
||||
download_model_weights(model_path: str): Downloads the model weights from the specified URL and saves them to the given file path.
|
||||
preprocess(img): Preprocesses the input image.
|
||||
run(img, box): Runs the semantic segmentation on the input image within the specified bounding box.
|
||||
|
||||
"""
|
||||
|
||||
model_path: str
|
||||
device: str = "cuda:0"
|
||||
model_weights_url: str = "https://dl.fbaipublicfiles.com/segment_anything/sam_vit_b_01ec64.pth"
|
||||
|
||||
def __post_init__(self):
|
||||
if not os.path.exists(self.model_path):
|
||||
self.download_model_weights(self.model_path)
|
||||
|
||||
self.model = sam_model_registry["vit_b"](
|
||||
checkpoint=self.model_path
|
||||
)
|
||||
self.model = self.model.to(self.device)
|
||||
self.model.eval()
|
||||
|
||||
def download_model_weights(self, model_path: str):
|
||||
"""
|
||||
Downloads the model weights from the specified URL and saves them to the given file path.
|
||||
|
||||
Args:
|
||||
model_path (str): The file path where the model weights will be saved.
|
||||
|
||||
Raises:
|
||||
Exception: If the model weights fail to download.
|
||||
"""
|
||||
response = requests.get(self.model_weights_url, stream=True)
|
||||
if response.status_code == 200:
|
||||
with open(model_path, "wb") as f:
|
||||
f.write(response.content)
|
||||
else:
|
||||
raise Exception("Failed to download model weights.")
|
||||
|
||||
def preprocess(self, img: np.ndarray) -> Tuple[Tensor, int, int]:
|
||||
"""
|
||||
Preprocesses the input image.
|
||||
|
||||
Args:
|
||||
img: The input image.
|
||||
|
||||
Returns:
|
||||
img_tensor: The preprocessed image tensor.
|
||||
H: The original height of the image.
|
||||
W: The original width of the image.
|
||||
"""
|
||||
if len(img.shape) == 2:
|
||||
img = np.repeat(img[:, :, None], 3, axis=-1)
|
||||
H, W, _ = img.shape
|
||||
img = transform.resize(
|
||||
img,
|
||||
(1024, 1024),
|
||||
order=3,
|
||||
preserve_range=True,
|
||||
anti_aliasing=True,
|
||||
).astype(np.uint8)
|
||||
img = img - img.min() / np.clip(
|
||||
img.max() - img.min(), a_min=1e-8, a_max=None
|
||||
)
|
||||
img = torch.tensor(img).float().permute(2, 0, 1).unsqueeze(0)
|
||||
return img, H, W
|
||||
|
||||
@torch.no_grad()
|
||||
def run(self, img: np.ndarray, box: np.ndarray) -> np.ndarray:
|
||||
"""
|
||||
Runs the semantic segmentation on the input image within the specified bounding box.
|
||||
|
||||
Args:
|
||||
img: The input image.
|
||||
box: The bounding box coordinates (x1, y1, x2, y2).
|
||||
|
||||
Returns:
|
||||
medsam_seg: The segmented image.
|
||||
"""
|
||||
img_tensor, H, W = self.preprocess(img)
|
||||
img_tensor = img_tensor.to(self.device)
|
||||
box_1024 = box / np.array([W, H, W, H]) * 1024
|
||||
img = self.model.image_encoder(img_tensor)
|
||||
|
||||
box_torch = torch.as_tensor(
|
||||
box_1024, dtype=torch.float, device=img_tensor.device
|
||||
)
|
||||
|
||||
if len(box_torch.shape) == 2:
|
||||
box_torch = box_torch[:, None, :]
|
||||
|
||||
sparse_embeddings, dense_embeddings = (
|
||||
self.model.prompt_encoder(
|
||||
points=None,
|
||||
boxes=box_torch,
|
||||
masks=None,
|
||||
)
|
||||
)
|
||||
|
||||
low_res_logits, _ = self.model.mask_decoder(
|
||||
image_embeddings=img,
|
||||
image_pe=self.model.prompt_encoder.get_dense_pe(),
|
||||
sparse_prompt_embeddings=sparse_embeddings,
|
||||
dense_prompt_embeddings=dense_embeddings,
|
||||
multimask_output=False,
|
||||
)
|
||||
|
||||
low_res_pred = torch.sigmoid(low_res_logits)
|
||||
low_res_pred = F.interpolate(
|
||||
low_res_pred,
|
||||
size=(H, W),
|
||||
mode="bilinear",
|
||||
align_corners=False,
|
||||
)
|
||||
low_res_pred = low_res_pred.squeeze().cpu().numpy()
|
||||
medsam_seg = (low_res_pred > 0.5).astype(np.uint8)
|
||||
|
||||
return medsam_seg
|
||||
@ -0,0 +1,108 @@
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Optional, Tuple
|
||||
|
||||
from PIL import Image
|
||||
from transformers import AutoModelForCausalLM, AutoTokenizer
|
||||
|
||||
from swarms.models.base_multimodal_model import BaseMultiModalModel
|
||||
|
||||
|
||||
@dataclass
|
||||
class QwenVLMultiModal(BaseMultiModalModel):
|
||||
"""
|
||||
QwenVLMultiModal is a class that represents a multi-modal model for Qwen chatbot.
|
||||
It inherits from the BaseMultiModalModel class.
|
||||
|
||||
Examples:
|
||||
>>> model = QwenVLMultiModal()
|
||||
>>> model.run("Hello, how are you?", "https://example.com/image.jpg")
|
||||
|
||||
"""
|
||||
|
||||
model_name: str = "Qwen/Qwen-VL-Chat"
|
||||
device: str = "cuda"
|
||||
args: tuple = field(default_factory=tuple)
|
||||
kwargs: dict = field(default_factory=dict)
|
||||
quantize: bool = False
|
||||
|
||||
def __post_init__(self):
|
||||
"""
|
||||
Initializes the QwenVLMultiModal object.
|
||||
It initializes the tokenizer and the model for the Qwen chatbot.
|
||||
"""
|
||||
|
||||
if self.quantize:
|
||||
self.model_name = "Qwen/Qwen-VL-Chat-Int4"
|
||||
|
||||
self.tokenizer = AutoTokenizer.from_pretrained(
|
||||
self.model_name, trust_remote_code=True
|
||||
)
|
||||
self.model = AutoModelForCausalLM.from_pretrained(
|
||||
self.model_name,
|
||||
device_map=self.device,
|
||||
trust_remote_code=True,
|
||||
).eval()
|
||||
|
||||
def run(
|
||||
self, text: str, img: str, *args, **kwargs
|
||||
) -> Tuple[Optional[str], Optional[Image.Image]]:
|
||||
"""
|
||||
Runs the Qwen chatbot model on the given text and image inputs.
|
||||
|
||||
Args:
|
||||
text (str): The input text for the chatbot.
|
||||
img (str): The input image for the chatbot.
|
||||
*args: Additional positional arguments.
|
||||
**kwargs: Additional keyword arguments.
|
||||
|
||||
Returns:
|
||||
Tuple[Optional[str], Optional[Image.Image]]: A tuple containing the response generated by the chatbot
|
||||
and the image associated with the response (if any).
|
||||
"""
|
||||
try:
|
||||
query = self.tokenizer.from_list_format(
|
||||
[
|
||||
{"image": img, "text": text},
|
||||
]
|
||||
)
|
||||
|
||||
inputs = self.tokenizer(query, return_tensors="pt")
|
||||
inputs = inputs.to(self.model.device)
|
||||
pred = self.model.generate(**inputs)
|
||||
response = self.tokenizer.decode(
|
||||
pred.cpu()[0], skip_special_tokens=False
|
||||
)
|
||||
return response
|
||||
except Exception as error:
|
||||
print(f"[ERROR]: [QwenVLMultiModal]: {error}")
|
||||
|
||||
def chat(
|
||||
self, text: str, img: str, *args, **kwargs
|
||||
) -> tuple[str, list]:
|
||||
"""
|
||||
Chat with the model using text and image inputs.
|
||||
|
||||
Args:
|
||||
text (str): The text input for the chat.
|
||||
img (str): The image input for the chat.
|
||||
*args: Additional positional arguments.
|
||||
**kwargs: Additional keyword arguments.
|
||||
|
||||
Returns:
|
||||
tuple[str, list]: A tuple containing the response and chat history.
|
||||
|
||||
Raises:
|
||||
Exception: If an error occurs during the chat.
|
||||
|
||||
"""
|
||||
try:
|
||||
response, history = self.model.chat(
|
||||
self.tokenizer,
|
||||
query=f"<img>{img}</img>这是什么",
|
||||
history=None,
|
||||
)
|
||||
return response, history
|
||||
except Exception as e:
|
||||
raise Exception(
|
||||
"An error occurred during the chat."
|
||||
) from e
|
||||
@ -0,0 +1,94 @@
|
||||
from io import BytesIO
|
||||
|
||||
import requests
|
||||
import torch
|
||||
from PIL import Image
|
||||
from transformers import (
|
||||
AutoProcessor,
|
||||
VipLlavaForConditionalGeneration,
|
||||
)
|
||||
|
||||
from swarms.models.base_multimodal_model import BaseMultiModalModel
|
||||
|
||||
|
||||
class VipLlavaMultiModal(BaseMultiModalModel):
|
||||
"""
|
||||
A multi-modal model for VIP-LLAVA.
|
||||
|
||||
Args:
|
||||
model_name (str): The name or path of the pre-trained model.
|
||||
max_new_tokens (int): The maximum number of new tokens to generate.
|
||||
device_map (str): The device mapping for the model.
|
||||
torch_dtype: The torch data type for the model.
|
||||
*args: Additional positional arguments.
|
||||
**kwargs: Additional keyword arguments.
|
||||
"""
|
||||
|
||||
def __init__(
|
||||
self,
|
||||
model_name: str = "llava-hf/vip-llava-7b-hf",
|
||||
max_new_tokens: int = 500,
|
||||
device_map: str = "auto",
|
||||
torch_dtype=torch.float16,
|
||||
*args,
|
||||
**kwargs,
|
||||
):
|
||||
super().__init__(*args, **kwargs)
|
||||
self.model_name = model_name
|
||||
self.max_new_tokens = max_new_tokens
|
||||
self.device_map = device_map
|
||||
self.torch_dtype = torch_dtype
|
||||
|
||||
self.model = VipLlavaForConditionalGeneration.from_pretrained(
|
||||
model_name,
|
||||
device_map=device_map,
|
||||
torch_dtype=torch_dtype,
|
||||
*args,
|
||||
**kwargs,
|
||||
)
|
||||
self.processor = AutoProcessor.from_pretrained(
|
||||
model_name, *args, **kwargs
|
||||
)
|
||||
|
||||
def run(self, text: str, img: str, *args, **kwargs):
|
||||
"""
|
||||
Run the VIP-LLAVA model.
|
||||
|
||||
Args:
|
||||
text (str): The input text.
|
||||
img (str): The URL of the input image.
|
||||
*args: Additional positional arguments.
|
||||
**kwargs: Additional keyword arguments.
|
||||
|
||||
Returns:
|
||||
str: The generated output text.
|
||||
tuple: A tuple containing None and the error message if an error occurs.
|
||||
"""
|
||||
try:
|
||||
response = requests.get(img, stream=True)
|
||||
response.raise_for_status()
|
||||
image = Image.open(BytesIO(response.content))
|
||||
|
||||
inputs = self.processor(
|
||||
text=text,
|
||||
images=image,
|
||||
return_tensors="pt",
|
||||
*args,
|
||||
**kwargs,
|
||||
).to(0, self.torch_dtype)
|
||||
|
||||
# Generate
|
||||
generate_ids = self.model.generate(
|
||||
**inputs, max_new_tokens=self.max_new_tokens, **kwargs
|
||||
)
|
||||
|
||||
return self.processor.decode(
|
||||
generate_ids[0][len(inputs["input_ids"][0]) :],
|
||||
skip_special_tokens=True,
|
||||
)
|
||||
|
||||
except requests.RequestException as error:
|
||||
return None, f"Error fetching image: {error}"
|
||||
|
||||
except Exception as error:
|
||||
return None, f"Error during model inference: {error}"
|
||||
@ -0,0 +1,103 @@
|
||||
import asyncio
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Any, Callable, List, Optional
|
||||
from swarms.structs.task import Task
|
||||
from swarms.utils.logger import logger
|
||||
|
||||
|
||||
@dataclass
|
||||
class AsyncWorkflow:
|
||||
"""
|
||||
Represents an asynchronous workflow to run tasks.
|
||||
|
||||
Attributes:
|
||||
name (str): The name of the workflow.
|
||||
description (str): The description of the workflow.
|
||||
max_loops (int): The maximum number of loops to run the workflow.
|
||||
autosave (bool): Flag indicating whether to autosave the results.
|
||||
dashboard (bool): Flag indicating whether to display a dashboard.
|
||||
task_pool (List[Any]): The list of tasks in the workflow.
|
||||
results (List[Any]): The list of results from running the tasks.
|
||||
loop (Optional[asyncio.AbstractEventLoop]): The event loop to use.
|
||||
stopping_condition (Optional[Callable]): The stopping condition for the workflow.
|
||||
|
||||
Methods:
|
||||
add(tasks: List[Any]) -> None:
|
||||
Add tasks to the workflow.
|
||||
|
||||
delete(task: Task = None, tasks: List[Task] = None) -> None:
|
||||
Delete a task from the workflow.
|
||||
|
||||
run() -> List[Any]:
|
||||
Run the workflow and return the results.
|
||||
"""
|
||||
|
||||
name: str = "Async Workflow"
|
||||
description: str = "A workflow to run asynchronous tasks"
|
||||
max_loops: int = 1
|
||||
autosave: bool = True
|
||||
dashboard: bool = False
|
||||
task_pool: List[Any] = field(default_factory=list)
|
||||
results: List[Any] = field(default_factory=list)
|
||||
loop: Optional[asyncio.AbstractEventLoop] = None
|
||||
stopping_condition: Optional[Callable] = None
|
||||
|
||||
async def add(self, task: Any, tasks: List[Any]):
|
||||
"""Add tasks to the workflow"""
|
||||
try:
|
||||
if tasks:
|
||||
for task in tasks:
|
||||
self.task_pool.extend(tasks)
|
||||
elif task:
|
||||
self.task_pool.append(task)
|
||||
|
||||
else:
|
||||
if task and tasks:
|
||||
# Add the task and tasks to the task pool
|
||||
self.task_pool.append(task)
|
||||
self.task_pool.extend(tasks)
|
||||
else:
|
||||
raise ValueError(
|
||||
"Either task or tasks must be provided"
|
||||
)
|
||||
|
||||
except Exception as error:
|
||||
logger.error(f"[ERROR][AsyncWorkflow] {error}")
|
||||
|
||||
async def delete(
|
||||
self, task: Any = None, tasks: List[Task] = None
|
||||
):
|
||||
"""Delete a task from the workflow"""
|
||||
try:
|
||||
if task:
|
||||
self.task_pool.remove(task)
|
||||
elif tasks:
|
||||
for task in tasks:
|
||||
self.task_pool.remove(task)
|
||||
except Exception as error:
|
||||
logger.error(f"[ERROR][AsyncWorkflow] {error}")
|
||||
|
||||
async def run(self):
|
||||
"""Run the workflow"""
|
||||
if self.loop is None:
|
||||
self.loop = asyncio.get_event_loop()
|
||||
for i in range(self.max_loops):
|
||||
logger.info(
|
||||
f"[INFO][AsyncWorkflow] Loop {i + 1}/{self.max_loops}"
|
||||
)
|
||||
futures = [
|
||||
asyncio.ensure_future(task.execute())
|
||||
for task in self.task_pool
|
||||
]
|
||||
self.results = await asyncio.gather(*futures)
|
||||
# if self.autosave:
|
||||
# self.save()
|
||||
# if self.dashboard:
|
||||
# self.display()
|
||||
|
||||
# Add a stopping condition to stop the workflow, if provided but stopping_condition takes in a parameter s for string
|
||||
if self.stopping_condition:
|
||||
if self.stopping_condition(self.results):
|
||||
break
|
||||
|
||||
return self.results
|
||||
@ -0,0 +1,175 @@
|
||||
from dataclasses import dataclass, field
|
||||
from typing import Dict, List, Optional, Union
|
||||
|
||||
from swarms.structs.agent import Agent
|
||||
from swarms.utils.logger import logger
|
||||
from swarms.structs.conversation import Conversation
|
||||
|
||||
|
||||
@dataclass
|
||||
class Company:
|
||||
"""
|
||||
Represents a company with a hierarchical organizational structure.
|
||||
"""
|
||||
|
||||
org_chart: List[List[Agent]]
|
||||
shared_instructions: str = None
|
||||
ceo: Optional[Agent] = None
|
||||
agents: List[Agent] = field(default_factory=list)
|
||||
agent_interactions: Dict[str, List[str]] = field(
|
||||
default_factory=dict
|
||||
)
|
||||
history: Conversation = field(default_factory=Conversation)
|
||||
|
||||
def __post_init__(self):
|
||||
self._parse_org_chart(self.org_chart)
|
||||
|
||||
def add(self, agent: Agent) -> None:
|
||||
"""
|
||||
Adds an agent to the company.
|
||||
|
||||
Args:
|
||||
agent (Agent): The agent to be added.
|
||||
|
||||
Raises:
|
||||
ValueError: If an agent with the same ID already exists in the company.
|
||||
"""
|
||||
try:
|
||||
if any(
|
||||
existing_agent.id == agent.id
|
||||
for existing_agent in self.agents
|
||||
):
|
||||
raise ValueError(
|
||||
f"Agent with id {agent.id} already exists in the"
|
||||
" company."
|
||||
)
|
||||
self.agents.append(agent)
|
||||
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
f"[ERROR][CLASS: Company][METHOD: add] {error}"
|
||||
)
|
||||
raise error
|
||||
|
||||
def get(self, agent_name: str) -> Agent:
|
||||
"""
|
||||
Retrieves an agent from the company by name.
|
||||
|
||||
Args:
|
||||
agent_name (str): The name of the agent to retrieve.
|
||||
|
||||
Returns:
|
||||
Agent: The retrieved agent.
|
||||
|
||||
Raises:
|
||||
ValueError: If an agent with the specified name does not exist in the company.
|
||||
"""
|
||||
try:
|
||||
for agent in self.agents:
|
||||
if agent.name == agent_name:
|
||||
return agent
|
||||
raise ValueError(
|
||||
f"Agent with name {agent_name} does not exist in the"
|
||||
" company."
|
||||
)
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
f"[ERROR][CLASS: Company][METHOD: get] {error}"
|
||||
)
|
||||
raise error
|
||||
|
||||
def remove(self, agent: Agent) -> None:
|
||||
"""
|
||||
Removes an agent from the company.
|
||||
|
||||
Args:
|
||||
agent (Agent): The agent to be removed.
|
||||
"""
|
||||
try:
|
||||
self.agents.remove(agent)
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
f"[ERROR][CLASS: Company][METHOD: remove] {error}"
|
||||
)
|
||||
raise error
|
||||
|
||||
def _parse_org_chart(
|
||||
self, org_chart: Union[List[Agent], List[List[Agent]]]
|
||||
) -> None:
|
||||
"""
|
||||
Parses the organization chart and adds agents to the company.
|
||||
|
||||
Args:
|
||||
org_chart (Union[List[Agent], List[List[Agent]]]): The organization chart
|
||||
representing the hierarchy of agents.
|
||||
|
||||
Raises:
|
||||
ValueError: If more than one CEO is found in the org chart or if an invalid
|
||||
agent is encountered.
|
||||
"""
|
||||
try:
|
||||
for node in org_chart:
|
||||
if isinstance(node, Agent):
|
||||
if self.ceo:
|
||||
raise ValueError("1 CEO is only allowed")
|
||||
self.ceo = node
|
||||
self.add(node)
|
||||
|
||||
elif isinstance(node, list):
|
||||
for agent in node:
|
||||
if not isinstance(agent, Agent):
|
||||
raise ValueError(
|
||||
"Invalid agent in org chart"
|
||||
)
|
||||
self.add(agent)
|
||||
|
||||
for i, agent in enumerate(node):
|
||||
if i == len(node) - 1:
|
||||
continue
|
||||
|
||||
for other_agent in node[i + 1]:
|
||||
self.__init_task(agent, other_agent)
|
||||
except Exception as error:
|
||||
logger.error(
|
||||
"[ERROR][CLASS: Company][METHOD: _parse_org_chart]"
|
||||
f" {error}"
|
||||
)
|
||||
raise error
|
||||
|
||||
def _init_interaction(
|
||||
self,
|
||||
agent1: Agent,
|
||||
agent2: Agent,
|
||||
) -> None:
|
||||
"""
|
||||
Initializes the interaction between two agents.
|
||||
|
||||
Args:
|
||||
agent1 (Agent): The first agent involved in the interaction.
|
||||
agent2 (Agent): The second agent involved in the interaction.
|
||||
|
||||
Returns:
|
||||
None
|
||||
"""
|
||||
if agent1.ai_name not in self.agents_interactions:
|
||||
self.agents_interactions[agent1.ai_name] = []
|
||||
self.agents_interactions[agent1.ai_name].append(
|
||||
agent2.ai_name
|
||||
)
|
||||
|
||||
def run(self):
|
||||
"""
|
||||
Run the company
|
||||
"""
|
||||
for (
|
||||
agent_name,
|
||||
interaction_agents,
|
||||
) in self.agents_interactions.items():
|
||||
agent = self.get(agent_name)
|
||||
for interaction_agent in interaction_agents:
|
||||
task_description = (
|
||||
f"Task for {agent_name} to interact with"
|
||||
f" {interaction_agent}"
|
||||
)
|
||||
print(f"{task_description} is being executed")
|
||||
agent.run(task_description)
|
||||
@ -0,0 +1,58 @@
|
||||
import logging
|
||||
import pymongo
|
||||
import platform
|
||||
import datetime
|
||||
|
||||
|
||||
class Telemetry:
|
||||
def __init__(self, db_url, db_name):
|
||||
self.logger = self.setup_logging()
|
||||
self.db = self.setup_db(db_url, db_name)
|
||||
|
||||
def setup_logging(self):
|
||||
logger = logging.getLogger("telemetry")
|
||||
logger.setLevel(logging.DEBUG)
|
||||
handler = logging.StreamHandler()
|
||||
handler.setFormatter(
|
||||
logging.Formatter(
|
||||
"%(asctime)s - %(name)s - %(levelname)s - %(message)s"
|
||||
)
|
||||
)
|
||||
logger.addHandler(handler)
|
||||
return logger
|
||||
|
||||
def setup_db(self, db_url, db_name):
|
||||
client = pymongo.MongoClient(db_url)
|
||||
return client[db_name]
|
||||
|
||||
def capture_device_data(self):
|
||||
data = {
|
||||
"system": platform.system(),
|
||||
"node": platform.node(),
|
||||
"release": platform.release(),
|
||||
"version": platform.version(),
|
||||
"machine": platform.machine(),
|
||||
"processor": platform.processor(),
|
||||
"time": datetime.datetime.now(),
|
||||
}
|
||||
return data
|
||||
|
||||
def send_to_db(self, collection_name, data):
|
||||
collection = self.db[collection_name]
|
||||
collection.insert_one(data)
|
||||
|
||||
def log_and_capture(self, message, level, collection_name):
|
||||
if level == "info":
|
||||
self.logger.info(message)
|
||||
elif level == "error":
|
||||
self.logger.error(message)
|
||||
data = self.capture_device_data()
|
||||
data["log"] = message
|
||||
self.send_to_db(collection_name, data)
|
||||
|
||||
def log_import(self, module_name):
|
||||
self.logger.info(f"Importing module {module_name}")
|
||||
module = __import__(module_name, fromlist=["*"])
|
||||
for k in dir(module):
|
||||
if not k.startswith("__"):
|
||||
self.logger.info(f"Imported {k} from {module_name}")
|
||||
@ -0,0 +1,22 @@
|
||||
import requests
|
||||
|
||||
|
||||
def download_weights_from_url(
|
||||
url: str, save_path: str = "models/weights.pth"
|
||||
):
|
||||
"""
|
||||
Downloads model weights from the given URL and saves them to the specified path.
|
||||
|
||||
Args:
|
||||
url (str): The URL from which to download the model weights.
|
||||
save_path (str, optional): The path where the downloaded weights should be saved.
|
||||
Defaults to "models/weights.pth".
|
||||
"""
|
||||
response = requests.get(url, stream=True)
|
||||
response.raise_for_status()
|
||||
|
||||
with open(save_path, "wb") as f:
|
||||
for chunk in response.iter_content(chunk_size=8192):
|
||||
f.write(chunk)
|
||||
|
||||
print(f"Model weights downloaded and saved to {save_path}")
|
||||
@ -0,0 +1,33 @@
|
||||
import inspect
|
||||
|
||||
|
||||
def get_cls_init_params(cls) -> str:
|
||||
"""
|
||||
Get the initialization parameters of a class.
|
||||
|
||||
Args:
|
||||
cls: The class to retrieve the initialization parameters from.
|
||||
|
||||
Returns:
|
||||
str: A string representation of the initialization parameters.
|
||||
|
||||
"""
|
||||
init_signature = inspect.signature(cls.__init__)
|
||||
params = init_signature.parameters
|
||||
params_str_list = []
|
||||
|
||||
for name, param in params.items():
|
||||
if name == "self":
|
||||
continue
|
||||
if name == "kwargs":
|
||||
value = "Any keyword arguments"
|
||||
elif hasattr(cls, name):
|
||||
value = getattr(cls, name)
|
||||
else:
|
||||
value = cls.__dict__.get(name, "Unknown")
|
||||
|
||||
params_str_list.append(
|
||||
f" {name.capitalize().replace('_', ' ')}: {value}"
|
||||
)
|
||||
|
||||
return "\n".join(params_str_list)
|
||||
@ -0,0 +1,21 @@
|
||||
import re
|
||||
|
||||
|
||||
def get_file_extension(s):
|
||||
"""
|
||||
Get the file extension from a given string.
|
||||
|
||||
Args:
|
||||
s (str): The input string.
|
||||
|
||||
Returns:
|
||||
str or None: The file extension if found, or None if not found.
|
||||
|
||||
Raises:
|
||||
ValueError: If the input is not a string.
|
||||
"""
|
||||
if not isinstance(s, str):
|
||||
raise ValueError("Input must be a string")
|
||||
|
||||
match = re.search(r"\.(pdf|csv|txt|docx|xlsx)$", s, re.IGNORECASE)
|
||||
return match.group()[1:] if match else None
|
||||
@ -0,0 +1,46 @@
|
||||
import os
|
||||
|
||||
|
||||
def parse_log_file(filename: str):
|
||||
"""
|
||||
Parse a log file and return a list of log entries.
|
||||
|
||||
Each log entry is a dictionary with keys for the timestamp, name, level, and message.
|
||||
|
||||
Args:
|
||||
filename (str): The name of the log file.
|
||||
|
||||
Returns:
|
||||
list: A list of log entries.
|
||||
|
||||
Raises:
|
||||
FileNotFoundError: If the log file does not exist.
|
||||
ValueError: If a log entry does not have the correct format.
|
||||
"""
|
||||
# Check if the file exists
|
||||
if not os.path.exists(filename):
|
||||
raise FileNotFoundError(
|
||||
f"The file {filename} does not exist."
|
||||
)
|
||||
|
||||
log_entries = []
|
||||
|
||||
with open(filename, "r") as file:
|
||||
for line in file:
|
||||
parts = line.split(" - ")
|
||||
# Check if the log entry has the correct format
|
||||
if len(parts) != 4:
|
||||
raise ValueError(
|
||||
f"The log entry '{line}' does not have the"
|
||||
" correct format."
|
||||
)
|
||||
timestamp, name, level, message = parts
|
||||
log_entry = {
|
||||
"timestamp": timestamp,
|
||||
"name": name,
|
||||
"level": level,
|
||||
"message": message.rstrip("\n"),
|
||||
}
|
||||
log_entries.append(log_entry)
|
||||
|
||||
return log_entries
|
||||
@ -0,0 +1,60 @@
|
||||
from unittest.mock import Mock, patch
|
||||
from swarms.models.qwen import QwenVLMultiModal
|
||||
|
||||
|
||||
def test_post_init():
|
||||
with patch(
|
||||
"swarms.models.qwen.AutoTokenizer.from_pretrained"
|
||||
) as mock_tokenizer, patch(
|
||||
"swarms.models.qwen.AutoModelForCausalLM.from_pretrained"
|
||||
) as mock_model:
|
||||
mock_tokenizer.return_value = Mock()
|
||||
mock_model.return_value = Mock()
|
||||
|
||||
model = QwenVLMultiModal()
|
||||
mock_tokenizer.assert_called_once_with(
|
||||
model.model_name, trust_remote_code=True
|
||||
)
|
||||
mock_model.assert_called_once_with(
|
||||
model.model_name,
|
||||
device_map=model.device,
|
||||
trust_remote_code=True,
|
||||
)
|
||||
|
||||
|
||||
def test_run():
|
||||
with patch(
|
||||
"swarms.models.qwen.AutoTokenizer.from_list_format"
|
||||
) as mock_format, patch(
|
||||
"swarms.models.qwen.AutoTokenizer.__call__"
|
||||
) as mock_call, patch(
|
||||
"swarms.models.qwen.AutoModelForCausalLM.generate"
|
||||
) as mock_generate, patch(
|
||||
"swarms.models.qwen.AutoTokenizer.decode"
|
||||
) as mock_decode:
|
||||
mock_format.return_value = Mock()
|
||||
mock_call.return_value = Mock()
|
||||
mock_generate.return_value = Mock()
|
||||
mock_decode.return_value = "response"
|
||||
|
||||
model = QwenVLMultiModal()
|
||||
response = model.run(
|
||||
"Hello, how are you?", "https://example.com/image.jpg"
|
||||
)
|
||||
|
||||
assert response == "response"
|
||||
|
||||
|
||||
def test_chat():
|
||||
with patch(
|
||||
"swarms.models.qwen.AutoModelForCausalLM.chat"
|
||||
) as mock_chat:
|
||||
mock_chat.return_value = ("response", ["history"])
|
||||
|
||||
model = QwenVLMultiModal()
|
||||
response, history = model.chat(
|
||||
"Hello, how are you?", "https://example.com/image.jpg"
|
||||
)
|
||||
|
||||
assert response == "response"
|
||||
assert history == ["history"]
|
||||
@ -0,0 +1,70 @@
|
||||
import pytest
|
||||
from swarms.structs.agent import Agent
|
||||
from swarms.structs.company import Company
|
||||
from swarms import OpenAIChat
|
||||
|
||||
# Mock OpenAIChat instance
|
||||
llm = OpenAIChat(openai_api_key="test_key", max_tokens=4000)
|
||||
|
||||
# Mock Agents
|
||||
ceo = Agent(llm=llm, name="CEO")
|
||||
dev = Agent(llm=llm, name="Developer")
|
||||
va = Agent(llm=llm, name="VA")
|
||||
hr = Agent(llm=llm, name="HR")
|
||||
shared_instructions = "Listen to your boss"
|
||||
|
||||
|
||||
def test_add_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
company.add(hr)
|
||||
assert hr in company.agents
|
||||
|
||||
|
||||
def test_get_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
company.add(hr)
|
||||
assert company.get("HR") == hr
|
||||
|
||||
|
||||
def test_remove_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
company.add(hr)
|
||||
company.remove(hr)
|
||||
assert hr not in company.agents
|
||||
|
||||
|
||||
def test_add_existing_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
company.add(hr)
|
||||
with pytest.raises(ValueError):
|
||||
company.add(hr)
|
||||
|
||||
|
||||
def test_get_nonexistent_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
company.get("Nonexistent")
|
||||
|
||||
|
||||
def test_remove_nonexistent_agent():
|
||||
company = Company(
|
||||
org_chart=[[ceo, [dev, va]]],
|
||||
shared_instructions=shared_instructions,
|
||||
)
|
||||
with pytest.raises(ValueError):
|
||||
company.remove(hr)
|
||||
Loading…
Reference in new issue