[FEATS][File processing utils] [docs]

docs/diy_your_own_agent.md

@@ -0,0 +1,349 @@
+# Create your own agent with `Agent` class
+
+In the rapidly evolving world of artificial intelligence (AI), the demand for specialized and highly customized agents is on the rise. Whether it's for task automation, decision support systems, or intelligent virtual assistants, the ability to create tailored agents can unlock new possibilities and efficiencies across various domains. Enter the Agent class, a powerful and flexible tool designed by Anthropic that empowers AI agents to build their own custom agents, tailored to their specific needs.
+
+This comprehensive guide will explore the process of inheriting from the Agent class, enabling agents to create their own custom agent classes. By leveraging the rich features and extensibility of the Agent class, agents can imbue their offspring agents with unique capabilities, specialized toolsets, and tailored decision-making processes.
+
+## Understanding the Agent Class
+
+Before we dive into the intricacies of creating custom agent classes, let's revisit the foundational elements of the Agent class itself. The Agent class is a versatile and feature-rich class designed to streamline the process of building and managing AI agents. It acts as a backbone, connecting language models (LLMs) with various tools, long-term memory, and a wide range of customization options.
+
+### Key Features of the Agent Class
+
+The Agent class offers a plethora of features that can be inherited and extended by custom agent classes. Here are some of the key features that make the Agent class a powerful foundation:
+
+1\. **Language Model Integration**: The Agent class supports seamless integration with popular language models such as LangChain, HuggingFace Transformers, and Autogen, allowing custom agent classes to leverage the power of state-of-the-art language models.
+
+2\. **Tool Integration**: One of the standout features of the Agent class is its ability to integrate with various tools. Custom agent classes can inherit this capability and incorporate specialized tools tailored to their specific use cases.
+
+3\. **Long-Term Memory**: The Agent class provides built-in support for long-term memory, enabling custom agent classes to retain and access information from previous interactions, essential for maintaining context and learning from past experiences.
+
+4\. **Customizable Prompts and Standard Operating Procedures (SOPs)**: The Agent class allows you to define custom prompts and Standard Operating Procedures (SOPs) that guide an agent's behavior and decision-making process. Custom agent classes can inherit and extend these prompts and SOPs to align with their unique objectives and requirements.
+
+5\. **Interactive and Dashboard Modes**: The Agent class supports interactive and dashboard modes, enabling real-time monitoring and interaction with agents. Custom agent classes can inherit these modes, facilitating efficient development, debugging, and user interaction.
+
+6\. **Autosave and State Management**: With the Agent class, agents can easily save and load their state, including configuration, memory, and history. Custom agent classes can inherit this capability, ensuring seamless task continuation and enabling efficient collaboration among team members.
+
+7\. **Response Filtering**: The Agent class provides built-in response filtering capabilities, allowing agents to filter out or replace specific words or phrases in their responses. Custom agent classes can inherit and extend this feature to ensure compliance with content moderation policies or specific guidelines.
+
+8\. **Code Execution and Multimodal Support**: The Agent class supports code execution and multimodal input/output, enabling agents to process and generate code, as well as handle various data formats such as images, audio, and video. Custom agent classes can inherit and specialize these capabilities for their unique use cases.
+
+9\. **Extensibility and Customization**: The Agent class is designed to be highly extensible and customizable, allowing agents to tailor its behavior, add custom functionality, and integrate with external libraries and APIs. Custom agent classes can leverage this extensibility to introduce specialized features and capabilities.
+
+### Creating a Custom Agent Class
+
+Now that we have a solid understanding of the Agent class and its features, let's dive into the process of creating a custom agent class by inheriting from the Agent class. Throughout this process, we'll explore how agents can leverage and extend the existing functionality, while introducing specialized features and capabilities tailored to their unique requirements.
+
+#### Step 1: Inherit from the Agent Class
+
+The first step in creating a custom agent class is to inherit from the Agent class. This will provide your custom agent class with the foundational features and capabilities of the Agent class, which can then be extended and customized as needed.
+
+```python
+
+from swarms import Agent
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Add custom initialization logic here
+
+```
+
+In the example above, we define a new class `MyCustomAgent` that inherits from the `Agent` class. Within the `__init__` method, we call the parent class's `__init__` method using `super().__init__(*args, **kwargs)`, which ensures that the parent class's initialization logic is executed. You can then add any custom initialization logic specific to your custom agent class.
+
+#### Step 2: Customize the Agent's Behavior
+
+One of the key advantages of inheriting from the Agent class is the ability to customize the agent's behavior according to your specific requirements. This can be achieved by overriding or extending the existing methods, or by introducing new methods altogether.
+
+```python
+from swarms import Agent
+
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom initialization logic
+
+    def custom_method(self, *args, **kwargs):
+
+        # Implement custom logic here
+
+        pass
+
+    def run(self, task, *args, **kwargs):
+
+        # Customize the run method
+
+        response = super().run(task, *args, **kwargs)
+
+        # Additional custom logic
+
+        return response
+
+```
+
+In the example above, we introduce a new `custom_method` that can encapsulate any specialized logic or functionality specific to your custom agent class. Additionally, we override the `run` method, which is responsible for executing the agent's main task loop. Within the overridden `run` method, you can call the parent class's `run` method using `super().run(task, *args, **kwargs)` and then introduce any additional custom logic before or after the parent method's execution.
+
+#### Step 3: Integrate Custom Tools
+
+One of the powerful features of the Agent class is the ability to integrate with various tools. Custom agent classes can inherit this capability and incorporate specialized tools tailored to their unique use cases.
+
+```python
+
+from swarms.tools import BaseTool
+from swarms import Agent
+
+
+class CustomTool(BaseTool):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom tool initialization logic
+
+    def run(self, *args, **kwargs):
+
+        # Custom tool logic
+
+        return result
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom initialization logic
+
+        self.tools = [CustomTool()]
+
+    def run(self, task, *args, **kwargs):
+
+        # Customize the run method
+
+        response = super().run(task, *args, **kwargs)
+
+        # Utilize custom tools
+
+        for tool in self.tools:
+
+            result = tool.run(*args, **kwargs)
+
+            # Process tool result
+
+        return response
+
+```
+
+In the example above, we define a new `CustomTool` class that inherits from the `BaseTool` class provided by the Agent class framework. Within the `CustomTool` class, you can implement the specialized logic and functionality required by your custom tool.
+
+Next, within the `MyCustomAgent` class, we initialize an instance of the `CustomTool` and store it in the `self.tools` list. This list can then be utilized within the overridden `run` method, where you can execute each tool and process its results as needed.
+
+#### Step 4: Extend Memory Management
+
+The Agent class provides built-in support for long-term memory, allowing agents to retain and access information from previous interactions. Custom agent classes can inherit and extend this capability by introducing specialized memory management techniques.
+
+```python
+
+from swarms.memory import AbstractVectorDatabase
+from swarms import Agent
+
+
+class CustomMemory(AbstractVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom memory initialization logic
+
+    def query(self, *args, **kwargs):
+
+        # Custom memory query logic
+
+        return result
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom initialization logic
+
+        self.long_term_memory = CustomMemory()
+
+    def run(self, task, *args, **kwargs):
+
+        # Customize the run method
+
+        response = super().run(task, *args, **kwargs)
+
+        # Utilize custom memory
+
+        memory_result = self.long_term_memory.query(*args, **kwargs)
+
+        # Process memory result
+
+        return response
+
+```
+
+In the example above, we define a new `CustomMemory` class that inherits from the `AbstractVectorDatabase` class provided by the Agent class framework. Within the `CustomMemory` class, you can implement specialized memory management logic, such as custom indexing, retrieval, and storage mechanisms.
+
+Next, within the `MyCustomAgent` class, we initialize an instance of the `CustomMemory` class and assign it to the `self.long_term_memory` attribute. This custom memory instance can then be utilized within the overridden `run` method, where you can query the memory and process the results as needed.
+
+Step 5: Introduce Custom Prompts and Standard Operating Procedures (SOPs)
+
+The Agent class allows you to define custom prompts and Standard Operating Procedures (SOPs) that guide an agent's behavior and decision-making process. Custom agent classes can inherit and extend these prompts and SOPs to align with their unique objectives and requirements.
+
+```python
+from swarms import Agent
+
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom initialization logic
+
+        self.custom_sop = "Custom SOP for MyCustomAgent..."
+
+        self.custom_prompt = "Custom prompt for MyCustomAgent..."
+
+    def run(self, task, *args, **kwargs):
+
+        # Customize the run method
+
+        response = super().run(task, *args, **kwargs)
+
+        # Utilize custom prompts and SOPs
+
+        custom_prompt = self.construct_dynamic_prompt(self.custom_prompt)
+
+        custom_sop = self.construct_dynamic_sop(self.custom_sop)
+
+        # Process custom prompts and SOPs
+
+        return response
+
+    def construct_dynamic_prompt(self, prompt):
+
+        # Custom prompt construction logic
+
+        return prompt
+
+    def construct_dynamic_sop(self, sop):
+
+        # Custom SOP construction logic
+
+        return sop
+
+```
+
+In the example above, we define two new attributes within the `MyCustomAgent` class: `custom_sop` and `custom_prompt`. These attributes can be used to store custom prompts and SOPs specific to your custom agent class.
+
+Within the overridden `run` method, you can utilize these custom prompts and SOPs by calling the `construct_dynamic_prompt` and `construct_dynamic_sop` methods, which can be defined within the `MyCustomAgent` class to implement specialized prompt and SOP construction logic.
+
+#### Step 6: Introduce Custom Response Handling
+
+The Agent class provides built-in response filtering capabilities, allowing agents to filter out or replace specific words or phrases in their responses. Custom agent classes can inherit and extend this feature to ensure compliance with content moderation policies or specific guidelines.
+
+```python
+from swarms import Agent
+
+
+class MyCustomAgent(Agent):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Custom initialization logic
+
+        self.response_filters = ["filter_word_1", "filter_word_2"]
+
+    def run(self, task, *args, **kwargs):
+
+        # Customize the run method
+
+        response = super().run(task, *args, **kwargs)
+
+        # Apply custom response filtering
+
+        filtered_response = self.apply_response_filters(response)
+
+        return filtered_response
+
+    def apply_response_filters(self, response):
+
+        # Custom response filtering logic
+
+        for word in self.response_filters:
+
+            response = response.replace(word, "[FILTERED]")
+
+        return response
+
+```
+
+In the example above, we define a new attribute `response_filters` within the `MyCustomAgent` class, which is a list of words or phrases that should be filtered out or replaced in the agent's responses.
+
+Within the overridden `run` method, we call the `apply_response_filters` method, which can be defined within the `MyCustomAgent` class to implement specialized response filtering logic. In the example, we iterate over the `response_filters` list and replace each filtered word or phrase with a placeholder string (`"[FILTERED]"`).
+
+### Advanced Customization and Integration
+
+The Agent class and its inherited custom agent classes can be further extended and customized to suit specific requirements and integrate with external libraries, APIs, and services. Here are some advanced customization and integration examples:
+
+1\. **Multimodal Input/Output Integration**: Custom agent classes can leverage the multimodal input/output capabilities of the Agent class and introduce specialized handling for various data formats such as images, audio, and video.
+
+2\. **Code Execution and Integration**: The Agent class supports code execution, enabling agents to run and evaluate code snippets. Custom agent classes can inherit and extend this capability, introducing specialized code execution environments, sandboxing mechanisms, or integration with external code repositories or platforms.
+
+3\. **External API and Service Integration**: Custom agent classes can integrate with external APIs and services, enabling agents to leverage specialized data sources, computational resources, or domain-specific services.
+
+4\. **Performance Optimization**: Depending on the use case and requirements, custom agent classes can introduce performance optimizations, such as adjusting loop intervals, retry attempts, or enabling parallel execution for certain tasks.
+
+5\. **Logging and Monitoring**: Custom agent classes can introduce specialized logging and monitoring mechanisms, enabling agents to track their performance, identify potential issues, and generate detailed reports or dashboards.
+
+6\. **Security and Privacy Enhancements**: Custom agent classes can implement security and privacy enhancements, such as data encryption, access control mechanisms, or compliance with industry-specific regulations and standards.
+
+7\. **Distributed Execution and Scaling**: Custom agent classes can be designed to support distributed execution and scaling, enabling agents to leverage cloud computing resources or distributed computing frameworks for handling large-scale tasks or high-concurrency workloads.
+
+By leveraging these advanced customization and integration capabilities, agents can create highly specialized and sophisticated custom agent classes tailored to their unique requirements and use cases.
+
+### Best Practices and Considerations
+
+While building custom agent classes by inheriting from the Agent class offers immense flexibility and power, it's essential to follow best practices and consider potential challenges and considerations:
+
+1\. **Maintainability and Documentation**: As custom agent classes become more complex, it's crucial to prioritize maintainability and thorough documentation. Clear and concise code, comprehensive comments, and up-to-date documentation can significantly improve the long-term sustainability and collaboration efforts surrounding custom agent classes.
+
+2\. **Testing and Validation**: Custom agent classes should undergo rigorous testing and validation to ensure their correctness, reliability, and adherence to expected behaviors. Establish a robust testing framework and continuously validate the agent's performance, particularly after introducing new features or integrations.
+
+3\. **Security and Privacy Considerations**: When building custom agent classes, it's essential to consider security and privacy implications, especially if the agents will handle sensitive data or interact with critical systems. Implement appropriate security measures, such as access controls, data encryption, and secure communication protocols, to protect against potential vulnerabilities and ensure compliance with relevant regulations and standards.
+
+4\. **Scalability and Performance Monitoring**: As custom agent classes are deployed and adopted, it's important to monitor their scalability and performance characteristics. Identify potential bottlenecks, resource constraints, or performance degradation, and implement appropriate optimization strategies or scaling mechanisms to ensure efficient and reliable operation.
+
+5\. **Collaboration and Knowledge Sharing**: Building custom agent classes often involves collaboration among teams and stakeholders. Foster an environment of knowledge sharing, code reviews, and open communication to ensure that everyone involved understands the agent's capabilities, limitations, and intended use cases.
+
+6\. **Ethical Considerations**: As AI agents become more advanced and autonomous, it's crucial to consider the ethical implications of their actions and decisions. Implement appropriate safeguards, oversight mechanisms, and ethical guidelines to ensure that custom agent classes operate in a responsible and transparent manner, aligning with ethical principles and societal values.
+
+7\. **Continuous Learning and Adaptation**: The field of AI is rapidly evolving, with new techniques, tools, and best practices emerging regularly. Stay up-to-date with the latest developments and be prepared to adapt and refine your custom agent classes as new advancements become available.
+
+By following these best practices and considering potential challenges, agents can create robust, reliable, and ethical custom agent classes that meet their specific requirements while adhering to industry standards and best practices.
+
+# Conclusion
+
+In this comprehensive guide, we have explored the process of creating custom agent classes by inheriting from the powerful Agent class. We have covered the key features of the Agent class, walked through the step-by-step process of inheriting and extending its functionality, and discussed advanced customization and integration techniques.
+
+Building custom agent classes empowers AI agents to create tailored and specialized agents capable of tackling unique challenges and addressing specific domain requirements. By leveraging the rich features and extensibility of the Agent class, agents can imbue their offspring agents with unique capabilities, specialized toolsets, and tailored decision-making processes.
+
+Remember, the journey of building custom agent classes is an iterative and collaborative process that requires continuous learning, adaptation, and refinement. Embrace the

docs/swarms/memory/diy_memory.md

@@ -0,0 +1,436 @@
+
+
+# Building Custom Vector Memory Databases with the AbstractVectorDatabase Class
+
+In the age of large language models (LLMs) and AI-powered applications, efficient memory management has become a crucial component. Vector databases, which store and retrieve data in high-dimensional vector spaces, have emerged as powerful tools for handling the vast amounts of data generated and consumed by AI systems. However, integrating vector databases into your applications can be a daunting task, requiring in-depth knowledge of their underlying architectures and APIs.
+
+Enter the `AbstractVectorDatabase` class, a powerful abstraction layer designed to simplify the process of creating and integrating custom vector memory databases into your AI applications. By inheriting from this class, developers can build tailored vector database solutions that seamlessly integrate with their existing systems, enabling efficient storage, retrieval, and manipulation of high-dimensional data.
+
+In this comprehensive guide, we'll explore the `AbstractVectorDatabase` class in detail, covering its core functionality and diving deep into the process of creating custom vector memory databases using popular solutions like PostgreSQL, Pinecone, Chroma, FAISS, and more. Whether you're a seasoned AI developer or just starting to explore the world of vector databases, this guide will provide you with the knowledge and tools necessary to build robust, scalable, and efficient memory solutions for your AI applications.
+
+## Understanding the AbstractVectorDatabase Class
+
+Before we dive into the implementation details, let's take a closer look at the `AbstractVectorDatabase` class and its core functionality.
+
+The `AbstractVectorDatabase` class is an abstract base class that defines the interface for interacting with a vector database. It serves as a blueprint for creating concrete implementations of vector databases, ensuring a consistent and standardized approach to database operations across different systems.
+
+The class provides a set of abstract methods that define the essential functionality required for working with vector databases, such as connecting to the database, executing queries, and performing CRUD (Create, Read, Update, Delete) operations.
+
+Here's a breakdown of the abstract methods defined in the `AbstractVectorDatabase` class:
+
+1\. `connect()`: This method establishes a connection to the vector database.
+
+2\. `close()`: This method closes the connection to the vector database.
+
+3\. `query(query: str)`: This method executes a given query on the vector database.
+
+4\. `fetch_all()`: This method retrieves all rows from the result set of a query.
+
+5\. `fetch_one()`: This method retrieves a single row from the result set of a query.
+
+6\. `add(doc: str)`: This method adds a new record to the vector database.
+
+7\. `get(query: str)`: This method retrieves a record from the vector database based on a given query.
+
+8\. `update(doc)`: This method updates a record in the vector database.
+
+9\. `delete(message)`: This method deletes a record from the vector database.
+
+By inheriting from the `AbstractVectorDatabase` class and implementing these abstract methods, developers can create concrete vector database implementations tailored to their specific needs and requirements.
+
+## Creating a Custom Vector Memory Database
+
+Now that we have a solid understanding of the `AbstractVectorDatabase` class, let's dive into the process of creating a custom vector memory database by inheriting from this class. Throughout this guide, we'll explore various vector database solutions, including PostgreSQL, Pinecone, Chroma, FAISS, and more, showcasing how to integrate them seamlessly into your AI applications.
+
+### Step 1: Inherit from the AbstractVectorDatabase Class
+
+The first step in creating a custom vector memory database is to inherit from the `AbstractVectorDatabase` class. This will provide your custom implementation with the foundational structure and interface defined by the abstract class.
+
+```python
+
+from abc import ABC, abstractmethod
+from swarms import AbstractVectorDatabase
+
+class MyCustomVectorDatabase(AbstractVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        # Custom initialization logic
+
+        pass
+
+```
+
+In the example above, we define a new class `MyCustomVectorDatabase` that inherits from the `AbstractVectorDatabase` class. Within the `__init__` method, you can add any custom initialization logic specific to your vector database implementation.
+
+### Step 2: Implement the Abstract Methods
+
+The next step is to implement the abstract methods defined in the `AbstractVectorDatabase` class. These methods provide the core functionality for interacting with your vector database, such as connecting, querying, and performing CRUD operations.
+
+```python
+from swarms import AbstractVectorDatabase
+
+
+class MyCustomVectorDatabase(AbstractVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        # Custom initialization logic
+
+        pass
+
+    def connect(self):
+
+        # Implementation for connecting to the vector database
+
+        pass
+
+    def close(self):
+
+        # Implementation for closing the vector database connection
+
+        pass
+
+    def query(self, query: str):
+
+        # Implementation for executing a query on the vector database
+
+        pass
+
+    def fetch_all(self):
+
+        # Implementation for fetching all rows from the result set
+
+        pass
+
+    def fetch_one(self):
+
+        # Implementation for fetching a single row from the result set
+
+        pass
+
+    def add(self, doc: str):
+
+        # Implementation for adding a new record to the vector database
+
+        pass
+
+    def get(self, query: str):
+
+        # Implementation for retrieving a record from the vector database
+
+        pass
+
+    def update(self, doc):
+
+        # Implementation for updating a record in the vector database
+
+        pass
+
+    def delete(self, message):
+
+        # Implementation for deleting a record from the vector database
+
+        pass
+
+```
+
+In this example, we define placeholders for each of the abstract methods within the `MyCustomVectorDatabase` class. These placeholders will be replaced with the actual implementation logic specific to your chosen vector database solution.
+
+### Step 3: Choose and Integrate Your Vector Database Solution
+
+With the foundational structure in place, it's time to choose a specific vector database solution and integrate it into your custom implementation. In this guide, we'll explore several popular vector database solutions, including PostgreSQL, Pinecone, Chroma, FAISS, and more, providing examples and guidance on how to integrate them seamlessly.
+
+### PostgreSQL Integration
+
+PostgreSQL is a powerful open-source relational database management system that supports vector data types and operations, making it a viable choice for building custom vector memory databases.
+
+```python
+
+import psycopg2
+from swarms import AbstractVectorDatabase
+
+class PostgreSQLVectorDatabase(MyCustomVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # PostgreSQL connection details
+
+        self.conn = psycopg2.connect(
+
+            host="localhost",
+
+            database="vector_db",
+
+            user="postgres",
+
+            password="your_password"
+
+        )
+
+        self.cur = self.conn.cursor()
+
+    def connect(self):
+
+        # PostgreSQL connection logic
+
+        pass
+
+    def close(self):
+
+        # Close PostgreSQL connection
+
+        self.cur.close()
+
+        self.conn.close()
+
+    def query(self, query: str):
+
+        # Execute PostgreSQL query
+
+        self.cur.execute(query)
+
+    def fetch_all(self):
+
+        # Fetch all rows from PostgreSQL result set
+
+        return self.cur.fetchall()
+
+    # Implement other abstract methods
+
+```
+
+In this example, we define a `PostgreSQLVectorDatabase` class that inherits from `MyCustomVectorDatabase`. Within the `__init__` method, we establish a connection to a PostgreSQL database using the `psycopg2` library. We then implement the `connect()`, `close()`, `query()`, and `fetch_all()` methods specific to PostgreSQL.
+
+### Pinecone Integration
+
+Pinecone is a managed vector database service that provides efficient storage, retrieval, and manipulation of high-dimensional vector data.
+
+```python
+
+import pinecone
+from swarms import AbstractVectorDatabase
+
+
+class PineconeVectorDatabase(MyCustomVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Pinecone initialization
+
+        pinecone.init(api_key="your_api_key", environment="your_environment")
+
+        self.index = pinecone.Index("your_index_name")
+
+    def connect(self):
+
+        # Pinecone connection logic
+
+        pass
+
+    def close(self):
+
+        # Close Pinecone connection
+
+        pass
+
+    def query(self, query: str):
+
+        # Execute Pinecone query
+
+        results = self.index.query(query)
+
+        return results
+
+    def add(self, doc: str):
+
+        # Add document to Pinecone index
+
+        self.index.upsert([("id", doc)])
+
+    # Implement other abstract methods
+
+```
+
+In this example, we define a `PineconeVectorDatabase` class that inherits from `MyCustomVectorDatabase`. Within the `__init__` method, we initialize the Pinecone client and create an index. We then implement the `query()` and `add()` methods specific to the Pinecone API.
+
+### Chroma Integration
+
+Chroma is an open-source vector database library that provides efficient storage, retrieval, and manipulation of vector data using various backends, including DuckDB, Chromadb, and more.
+
+```python
+
+from chromadb.client import Client
+from swarms import AbstractVectorDatabase
+
+class ChromaVectorDatabase(MyCustomVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # Chroma initialization
+
+        self.client = Client()
+
+        self.collection = self.client.get_or_create_collection("vector_collection")
+
+    def connect(self):
+
+        # Chroma connection logic
+
+        pass
+
+    def close(self):
+
+        # Close Chroma connection
+
+        pass
+
+    def query(self, query: str):
+
+        # Execute Chroma query
+
+        results = self.collection.query(query)
+
+        return results
+
+    def add(self, doc: str):
+
+        # Add document to Chroma collection
+
+        self.collection.add(doc)
+
+    # Implement other abstract methods
+
+```
+
+In this example, we define a `ChromaVectorDatabase` class that inherits from `MyCustomVectorDatabase`. Within the `__init__` method, we create a Chroma client and get or create a collection. We then implement the `query()` and `add()` methods specific to the Chroma API.
+
+### FAISS Integration
+
+FAISS (Facebook AI Similarity Search) is a library for efficient similarity search and clustering of dense vectors, developed by Meta AI.
+
+```python
+
+import faiss
+
+class FAISSVectorDatabase(MyCustomVectorDatabase):
+
+    def __init__(self, *args, **kwargs):
+
+        super().__init__(*args, **kwargs)
+
+        # FAISS initialization
+
+        self.index = faiss.IndexFlatL2(64)  # Assuming 64-dimensional vectors
+
+        self.index_path = "faiss_index.index"
+
+    def connect(self):
+
+        # FAISS connection logic
+
+        self.index = faiss.read_index(self.index_path)
+
+    def close(self):
+
+        # Close FAISS connection
+
+        faiss.write_index(self.index, self.index_path)
+
+    def query(self, query: str):
+
+        # Execute FAISS query
+
+        query_vector = # Convert query to vector
+
+        distances, indices = self.index.search(query_vector, k=10)
+
+        return [(self.index.reconstruct(i), d) for i, d in zip(indices, distances)]
+
+    def add(self, doc: str):
+
+        # Add document to FAISS index
+
+        doc_vector = # Convert doc to vector
+
+        self.index.add(doc_vector)
+
+    # Implement other abstract methods
+
+```
+
+In this example, we define a `FAISSVectorDatabase` class that inherits from `MyCustomVectorDatabase`. Within the `__init__` method, we create a FAISS index and set the index path. We then implement the `connect()`, `close()`, `query()`, and `add()` methods specific to the FAISS library, assuming 64-dimensional vectors for simplicity.
+
+These examples provide a starting point for integrating various vector database solutions into your custom implementation. Each solution has its own strengths, weaknesses, and trade-offs, so it's essential to carefully evaluate your requirements and choose the solution that best fits your needs.
+
+### Step 4: Add Custom Functionality and Optimizations
+
+Once you've integrated your chosen vector database solution, you can further extend and optimize your custom implementation by adding custom functionality and performance optimizations.
+
+#### Custom Functionality:
+
+- **Indexing Strategies**: Implement custom indexing strategies to optimize search performance and memory usage.
+
+- **Data Preprocessing**: Add data preprocessing logic to handle different data formats, perform embedding, and prepare data for storage in the vector database.
+
+- **Query Optimization**: Introduce query optimization techniques, such as query caching, result filtering, or query rewriting, to improve query performance.
+
+- **Data Partitioning**: Implement data partitioning strategies to distribute data across multiple nodes or shards for better scalability and performance.
+
+- **Metadata Management**: Introduce metadata management capabilities to store and retrieve additional information associated with the vector data.
+
+Performance Optimizations:
+
+- **Caching**: Implement caching mechanisms to reduce redundant computations and improve response times.
+
+- **Asynchronous Operations**: Utilize asynchronous programming techniques to improve concurrency and responsiveness.
+
+- **Multithreading and Parallelization**: Leverage multithreading and parallelization to distribute computationally intensive tasks across multiple cores or processors.
+
+- **Load Balancing**: Implement load balancing strategies to distribute workloads evenly across multiple nodes or instances.
+
+- **Monitoring and Profiling**: Introduce monitoring and profiling tools to identify performance bottlenecks and optimize critical sections of your code.
+
+By adding custom functionality and performance optimizations, you can tailor your custom vector memory database to meet the specific requirements of your AI applications, ensuring efficient and scalable data management.
+
+### Best Practices and Considerations
+
+Building custom vector memory databases is a powerful but complex endeavor. To ensure the success and longevity of your implementation, it's essential to follow best practices and consider potential challenges and considerations.
+
+1\. **Scalability and Performance Testing**: Vector databases can quickly grow in size and complexity as your AI applications handle increasing amounts of data. Thoroughly test your implementation for scalability and performance under various load conditions, and optimize accordingly.
+
+2\. **Data Quality and Integrity**: Ensure that the data stored in your vector database is accurate, consistent, and free from duplicates or errors. Implement data validation and cleansing mechanisms to maintain data quality and integrity.
+
+3\. **Security and Access Control**: Vector databases may store sensitive or proprietary data. Implement robust security measures, such as encryption, access controls, and auditing mechanisms, to protect your data from unauthorized access or breaches.
+
+4\. **Distributed Architectures**: As your data and workloads grow, consider implementing distributed architectures to distribute the storage and computational load across multiple nodes or clusters. This can improve scalability, fault tolerance, and overall performance.
+
+5\. **Data Versioning and Backup**: Implement data versioning and backup strategies to ensure data integrity and enable recovery in case of errors or system failures.
+
+6\. **Documentation and Maintainability**: Well-documented code and comprehensive documentation are essential for ensuring the long-term maintainability and extensibility of your custom vector memory database implementation.
+
+7\. **Continuous Integration and Deployment**: Adopt continuous integration and deployment practices to streamline the development, testing, and deployment processes, ensuring that changes are thoroughly tested and deployed efficiently.
+
+8\. **Compliance and Regulatory Requirements**: Depending on your industry and use case, ensure that your custom vector memory database implementation complies with relevant regulations and standards, such as data privacy laws or industry-specific guidelines.
+
+9\. **Community Engagement and Collaboration**: Stay engaged with the vector database community, participate in discussions, and collaborate with other developers to share knowledge, best practices, and insights.
+
+By following these best practices and considering potential challenges, you can build robust, scalable, and efficient custom vector memory databases that meet the demanding requirements of modern AI applications.
+
+# Conclusion
+
+In this comprehensive guide, we've explored the `AbstractVectorDatabase` class and its role in simplifying the process of creating custom vector memory databases. We've covered the core functionality of the class, walked through the step-by-step process of inheriting and extending its functionality, and provided examples of integrating popular vector database solutions like PostgreSQL, Pinecone, Chroma, and FAISS.
+
+Building custom vector memory databases empowers developers to create tailored and efficient data management solutions that seamlessly integrate with their AI applications. By leveraging the power of vector databases, you can unlock new possibilities in data storage, retrieval, and manipulation, enabling your AI systems to handle vast amounts of high-dimensional data with ease.
+
+Remember, the journey of building custom vector memory databases is an iterative and collaborative process that requires continuous learning, adaptation, and refinement. Embrace the challenges, stay up-to-date with the latest developments in vector databases and AI, and continuously strive to optimize and enhance your implementations.
+
+As you embark on this journey, keep in mind the importance of scalability, performance, data quality, security, and compliance. Foster an environment of collaboration, knowledge sharing, and community engagement to ensure that your custom vector memory databases are robust, reliable, and capable of meeting the ever-evolving demands of the AI landscape.
+
+So, dive in, leverage the power of the `AbstractVectorDatabase` class, and create the custom vector memory databases that will drive the future of AI-powered applications.

docs/why_swarms.md

@@ -1,6 +1,6 @@
 # Why Swarms?
 
-The need for multiple agents to work together in artificial intelligence (AI) and particularly in the context of Large Language Models (LLMs) stems from several inherent limitations and challenges in handling complex, dynamic, and multifaceted tasks with single-agent systems. Collaborating with multiple agents offers a pathway to enhance computational efficiency, cognitive diversity, and problem-solving capabilities. This section delves into the rationale behind employing multi-agent systems and strategizes on overcoming the associated expenses, such as API bills and hosting costs.
+The need for multiple agents to work together in artificial intelligence (AI) and particularly in the context of Large Language Models (LLMs) stems from several inherent limitations and challenges in handling complex, dynamic, and multifaceted tasks with single-agent systems. Collaborating with multiple agents offers a pathway to enhance reliability, computational efficiency, cognitive diversity, and problem-solving capabilities. This section delves into the rationale behind employing multi-agent systems and strategizes on overcoming the associated expenses, such as API bills and hosting costs.
 
 ### Why Multiple Agents Are Necessary
 

mkdocs.yml

@@ -59,6 +59,8 @@ nav:
     - Overview: "index.md"
     - Contributing: "contributing.md"
     - Limitations of Individual Agents: "limits_of_individual_agents.md"
+    - Why Swarms: "why_swarms.md"
+    - DIY Build Your Own Agent: "diy_your_own_agent.md"
 - Swarms Framework:
   - Overview: "swarms/index.md" 
   - swarms.agents:
@@ -129,6 +131,7 @@ nav:
       - AnthropicTokenizer: "swarms/tokenizers/anthropictokenizer.md"
       - OpenaiTokenizer: "swarms/tokenizers/openaitokenizer.md"
   - swarms.memory:
+    - Building Custom Vector Memory Databases with the AbstractVectorDatabase Class: "swarms/memory/diy_memory.md"
     - Vector Databases:
       - Weaviate: "swarms/memory/weaviate.md"
       - PineconeDB: "swarms/memory/pinecone.md"

playground/demos/autoswarm/autoswarm.py

@@ -7,17 +7,33 @@ import swarms.prompts.autoswarm as sdsp
 # Load environment variables and initialize the OpenAI Chat model
 load_dotenv()
 api_key = os.getenv("OPENAI_API_KEY")
-llm = OpenAIChat(model_name = "gpt-4", openai_api_key=api_key)
+llm = OpenAIChat(model_name="gpt-4", openai_api_key=api_key)
 
 user_idea = "screenplay writing team"
 
-role_identification_agent = Agent(llm=llm, sop=sdsp.AGENT_ROLE_IDENTIFICATION_AGENT_PROMPT, max_loops=1)
-agent_configuration_agent = Agent(llm=llm, sop=sdsp.AGENT_CONFIGURATION_AGENT_PROMPT, max_loops=1)
-swarm_assembly_agent = Agent(llm=llm, sop=sdsp.SWARM_ASSEMBLY_AGENT_PROMPT, max_loops=1)
-testing_optimization_agent = Agent(llm=llm, sop=sdsp.TESTING_OPTIMIZATION_AGENT_PROMPT, max_loops=1)
+role_identification_agent = Agent(
+    llm=llm,
+    sop=sdsp.AGENT_ROLE_IDENTIFICATION_AGENT_PROMPT,
+    max_loops=1,
+)
+agent_configuration_agent = Agent(
+    llm=llm, sop=sdsp.AGENT_CONFIGURATION_AGENT_PROMPT, max_loops=1
+)
+swarm_assembly_agent = Agent(
+    llm=llm, sop=sdsp.SWARM_ASSEMBLY_AGENT_PROMPT, max_loops=1
+)
+testing_optimization_agent = Agent(
+    llm=llm, sop=sdsp.TESTING_OPTIMIZATION_AGENT_PROMPT, max_loops=1
+)
 
 # Process the user idea through each agent
 role_identification_output = role_identification_agent.run(user_idea)
-agent_configuration_output = agent_configuration_agent.run(role_identification_output)
-swarm_assembly_output = swarm_assembly_agent.run(agent_configuration_output)
-testing_optimization_output = testing_optimization_agent.run(swarm_assembly_output)
+agent_configuration_output = agent_configuration_agent.run(
+    role_identification_output
+)
+swarm_assembly_output = swarm_assembly_agent.run(
+    agent_configuration_output
+)
+testing_optimization_output = testing_optimization_agent.run(
+    swarm_assembly_output
+)

swarms/agents/base.py

@@ -17,8 +17,9 @@ class AbstractAgent:
     def __init__(
         self,
         name: str,
-        # tools: List[Tool],
-        # memory: Memory
+        *args,
+        **kwargs
+        
     ):
         """
         Args:

swarms/memory/action_subtask.py

@@ -1,8 +1,6 @@
-from dataclasses import dataclass
+from pydantic import BaseModel
 
-
-@dataclass
-class ActionSubtaskEntry:
+class ActionSubtaskEntry(BaseModel):
     """Used to store ActionSubtask data to preserve TaskMemory pointers and context in the form of thought and action.
 
     Attributes:

swarms/models/moondream_mm.py

@@ -0,0 +1,63 @@
+from PIL import Image
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+from swarms.models.base_multimodal_model import BaseMultiModalModel
+
+
+class MoonDream(BaseMultiModalModel):
+    """
+    MoonDream is a multi-modal model that combines text and image inputs to generate descriptive answers for images.
+
+    Args:
+        model_name (str): The name or path of the pre-trained model to be used.
+        revision (str): The specific revision of the pre-trained model to be used.
+
+    Attributes:
+        model_name (str): The name or path of the pre-trained model.
+        revision (str): The specific revision of the pre-trained model.
+        model (AutoModelForCausalLM): The pre-trained model for generating answers.
+        tokenizer (AutoTokenizer): The tokenizer for processing text inputs.
+
+    """
+
+    def __init__(
+        self,
+        model_name: str = "vikhyatk/moondream2",
+        revision: str = "2024-03-04",
+        system_prompt: str = None,
+        *args,
+        **kwargs,
+    ):
+        super().__init__()
+        self.model_name = model_name
+        self.revision = revision
+        self.system_prompt = system_prompt
+
+        self.model = AutoModelForCausalLM.from_pretrained(
+            model_name,
+            trust_remote_code=True,
+            revision=revision,
+            *args,
+            **kwargs,
+        )
+        self.tokenizer = AutoTokenizer.from_pretrained(
+            model_name, revision=revision
+        )
+
+    def run(self, task: str, img: str):
+        """
+        Runs the MoonDream model to generate a descriptive answer for the given image.
+
+        Args:
+            task (str): The task or question related to the image.
+            img (str): The path or URL of the image file.
+
+        Returns:
+            str: The descriptive answer generated by the MoonDream model.
+
+        """
+        image = Image.open(img)
+        enc_image = self.model.encode_image(image)
+        return self.model.answer_question(
+            enc_image, f"{self.system_propmpt} {task}", self.tokenizer
+        )

swarms/prompts/code_spawner.py

@@ -56,9 +56,7 @@ Note: The expectation is that the refactored code will be structured and tagged
 """
 
 
-
 # Push the final codebase to a GitHub repository, managing code changes and revisions
 GITHUB_PUSH_PROMPT = """
 Push the final codebase to a GitHub repository. Manage code changes and maintain a history of revisions using version control integration. Here are the final changes: {changes}
 """
-

swarms/structs/__init__.py

@@ -1,5 +1,5 @@
 from swarms.structs.agent import Agent
-from swarms.structs.agent_base import AgentJob
+from swarms.structs.agent_job import AgentJob
 from swarms.structs.autoscaler import AutoScaler
 from swarms.structs.base import BaseStructure
 from swarms.structs.base_swarm import AbstractSwarm

swarms/structs/base_multiagent_structure.py

@@ -1,197 +0,0 @@
-import json
-from typing import List, Optional, Sequence
-
-import yaml
-
-from swarms.structs.agent import Agent
-from swarms.structs.conversation import Conversation
-from swarms.utils.logger import logger
-
-
-class BaseMultiAgentStructure:
-    """
-    Base class for a multi-agent structure.
-
-    Args:
-        agents (List[Agent], optional): List of agents in the structure. Defaults to None.
-        callbacks (Optional[Sequence[callable]], optional): List of callbacks for the structure. Defaults to None.
-        autosave (bool, optional): Flag indicating whether to enable autosave. Defaults to False.
-        logging (bool, optional): Flag indicating whether to enable logging. Defaults to False.
-        *args: Variable length argument list.
-        **kwargs: Arbitrary keyword arguments.
-
-    Attributes:
-        agents (List[Agent]): List of agents in the structure.
-        callbacks (Optional[Sequence[callable]]): List of callbacks for the structure.
-        autosave (bool): Flag indicating whether autosave is enabled.
-        logging (bool): Flag indicating whether logging is enabled.
-        conversation (Conversation): Conversation object for the structure.
-
-    Methods:
-        metadata(): Get the metadata of the multi-agent structure.
-        save_to_json(filename: str): Save the current state of the multi-agent structure to a JSON file.
-        load_from_json(filename: str): Load the state of the multi-agent structure from a JSON file.
-    """
-
-    def __init__(
-        self,
-        agents: List[Agent] = None,
-        callbacks: Optional[Sequence[callable]] = None,
-        autosave: bool = False,
-        logging: bool = False,
-        return_metadata: bool = False,
-        metadata_filename: str = "multiagent_structure_metadata.json",
-        *args,
-        **kwargs,
-    ):
-        self.agents = agents
-        self.callbacks = callbacks
-        self.autosave = autosave
-        self.logging = logging
-        self.return_metadata = return_metadata
-        self.metadata_filename = metadata_filename
-        self.conversation = Conversation(
-            time_enabled=True, *args, **kwargs
-        )
-        if self.logging:
-            self.logging.basicConfig(
-                level=logging.INFO,
-                format="%(asctime)s - %(levelname)s - %(message)s",
-            )
-
-        # Handle the case where the agents are not provided
-        # Handle agents
-        for agent in self.agents:
-            if not isinstance(agent, Agent):
-                raise TypeError("Agents must be of type Agent.")
-
-        if self.agents is None:
-            self.agents = []
-
-        # Handle the case where the callbacks are not provided
-        if self.callbacks is None:
-            self.callbacks = []
-
-        # Handle the case where the autosave is not provided
-        if self.autosave is None:
-            self.autosave = False
-
-        # Handle the case where the logging is not provided
-        if self.logging is None:
-            self.logging = False
-
-        # Handle callbacks
-        if callbacks is not None:
-            for callback in self.callbacks:
-                if not callable(callback):
-                    raise TypeError("Callback must be callable.")
-
-        # Handle autosave
-        if autosave:
-            self.save_to_json(metadata_filename)
-
-    def metadata(self):
-        """
-        Get the metadata of the multi-agent structure.
-
-        Returns:
-            dict: The metadata of the multi-agent structure.
-        """
-        return {
-            "agents": self.agents,
-            "callbacks": self.callbacks,
-            "autosave": self.autosave,
-            "logging": self.logging,
-            "conversation": self.conversation,
-        }
-
-    def save_to_json(self, filename: str):
-        """
-        Save the current state of the multi-agent structure to a JSON file.
-
-        Args:
-            filename (str): The name of the file to save the multi-agent structure to.
-
-        Returns:
-            None
-        """
-        try:
-            with open(filename, "w") as f:
-                json.dump(self.__dict__, f)
-        except Exception as e:
-            logger.error(e)
-
-    def load_from_json(self, filename: str):
-        """
-        Load the state of the multi-agent structure from a JSON file.
-
-        Args:
-            filename (str): The name of the file to load the multi-agent structure from.
-
-        Returns:
-            None
-        """
-        try:
-            with open(filename) as f:
-                self.__dict__ = json.load(f)
-        except Exception as e:
-            logger.error(e)
-
-    def save_to_yaml(self, filename: str):
-        """
-        Save the current state of the multi-agent structure to a YAML file.
-
-        Args:
-            filename (str): The name of the file to save the multi-agent structure to.
-
-        Returns:
-            None
-        """
-        try:
-            with open(filename, "w") as f:
-                yaml.dump(self.__dict__, f)
-        except Exception as e:
-            logger.error(e)
-
-    def load_from_yaml(self, filename: str):
-        """
-        Load the state of the multi-agent structure from a YAML file.
-
-        Args:
-            filename (str): The name of the file to load the multi-agent structure from.
-
-        Returns:
-            None
-        """
-        try:
-            with open(filename) as f:
-                self.__dict__ = yaml.load(f)
-        except Exception as e:
-            logger.error(e)
-
-    def __repr__(self):
-        return f"{self.__class__.__name__}({self.__dict__})"
-
-    def __str__(self):
-        return f"{self.__class__.__name__}({self.__dict__})"
-
-    def __len__(self):
-        return len(self.agents)
-
-    def __getitem__(self, index):
-        return self.agents[index]
-
-    def __setitem__(self, index, value):
-        self.agents[index] = value
-
-    def __delitem__(self, index):
-        del self.agents[index]
-
-    def __iter__(self):
-        return iter(self.agents)
-
-    def __reversed__(self):
-        return reversed(self.agents)
-
-    def __contains__(self, value):
-        return value in self.agents

swarms/structs/base_swarm.py

@@ -1,10 +1,12 @@
+import yaml
+import json
 import asyncio
 from abc import ABC, abstractmethod
 from concurrent.futures import ThreadPoolExecutor, as_completed
-from typing import Any, Callable, Dict, List, Optional
-
+from typing import Any, Callable, Dict, List, Optional, Sequence
+from swarms.utils.loguru_logger import logger
 from swarms.structs.agent import Agent
-
+from swarms.structs.conversation import Conversation
 
 class AbstractSwarm(ABC):
     """
@@ -54,10 +56,63 @@ class AbstractSwarm(ABC):
     """
 
     # @abstractmethod
-    def __init__(self, agents: List[Agent], max_loops: int = 200):
+    def __init__(
+        self, 
+        agents: List[Agent], 
+        max_loops: int = 200,
+        callbacks: Optional[Sequence[callable]] = None,
+        autosave: bool = False,
+        logging: bool = False,
+        return_metadata: bool = False,
+        metadata_filename: str = "multiagent_structure_metadata.json",
+        stopping_function: Optional[Callable] = None,
+        stopping_condition: Optional[str] = "stop",
+        stopping_condition_args: Optional[Dict] = None,
+        *args,
+        **kwargs,
+    ):
         """Initialize the swarm with agents"""
         self.agents = agents
         self.max_loops = max_loops
+        self.callbacks = callbacks
+        self.autosave = autosave
+        self.logging = logging
+        self.return_metadata = return_metadata
+        self.metadata_filename = metadata_filename
+        self.conversation = Conversation(
+            time_enabled=True, *args, **kwargs
+        )
+
+        # Handle the case where the agents are not provided
+        # Handle agents
+        for agent in self.agents:
+            if not isinstance(agent, Agent):
+                raise TypeError("Agents must be of type Agent.")
+
+        if self.agents is None:
+            self.agents = []
+
+        # Handle the case where the callbacks are not provided
+        if self.callbacks is None:
+            self.callbacks = []
+
+        # Handle the case where the autosave is not provided
+        if self.autosave is None:
+            self.autosave = False
+
+        # Handle the case where the logging is not provided
+        if self.logging is None:
+            self.logging = False
+
+        # Handle callbacks
+        if callbacks is not None:
+            for callback in self.callbacks:
+                if not callable(callback):
+                    raise TypeError("Callback must be callable.")
+
+        # Handle autosave
+        if autosave:
+            self.save_to_json(metadata_filename)
 
     # @abstractmethod
     def communicate(self):
@@ -85,6 +140,8 @@ class AbstractSwarm(ABC):
 
     def step(self):
         """Step the swarm"""
+        
+        
 
     # @abstractmethod
     def add_agent(self, agent: "Agent"):
@@ -363,17 +420,17 @@ class AbstractSwarm(ABC):
         """Remove an llm from the god mode"""
         self.agents.remove(agent)
 
-    def add_agent(self, agent: Agent = None, *args, **kwargs):
-        """Add an agent to the swarm
+    # def add_agent(self, agent: Agent = None, *args, **kwargs):
+    #     """Add an agent to the swarm
 
-        Args:
-            agent (Agent, optional): _description_. Defaults to None.
+    #     Args:
+    #         agent (Agent, optional): _description_. Defaults to None.
 
-        Returns:
-            _type_: _description_
-        """
-        self.agents.append(agent)
-        return agent
+    #     Returns:
+    #         _type_: _description_
+    #     """
+    #     self.agents.append(agent)
+    #     return agent
 
     def run_all(self, task: str = None, *args, **kwargs):
         """Run all agents
@@ -460,3 +517,109 @@ class AbstractSwarm(ABC):
         Args:
             from (Agent | SwarmManagerBase): Instance of Agent or SwarmManagerBase representing the source of the relationship.
         """
+
+    def metadata(self):
+        """
+        Get the metadata of the multi-agent structure.
+
+        Returns:
+            dict: The metadata of the multi-agent structure.
+        """
+        return {
+            "agents": self.agents,
+            "callbacks": self.callbacks,
+            "autosave": self.autosave,
+            "logging": self.logging,
+            "conversation": self.conversation,
+        }
+
+    def save_to_json(self, filename: str):
+        """
+        Save the current state of the multi-agent structure to a JSON file.
+
+        Args:
+            filename (str): The name of the file to save the multi-agent structure to.
+
+        Returns:
+            None
+        """
+        try:
+            with open(filename, "w") as f:
+                json.dump(self.__dict__, f)
+        except Exception as e:
+            logger.error(e)
+
+    def load_from_json(self, filename: str):
+        """
+        Load the state of the multi-agent structure from a JSON file.
+
+        Args:
+            filename (str): The name of the file to load the multi-agent structure from.
+
+        Returns:
+            None
+        """
+        try:
+            with open(filename) as f:
+                self.__dict__ = json.load(f)
+        except Exception as e:
+            logger.error(e)
+
+    def save_to_yaml(self, filename: str):
+        """
+        Save the current state of the multi-agent structure to a YAML file.
+
+        Args:
+            filename (str): The name of the file to save the multi-agent structure to.
+
+        Returns:
+            None
+        """
+        try:
+            with open(filename, "w") as f:
+                yaml.dump(self.__dict__, f)
+        except Exception as e:
+            logger.error(e)
+
+    def load_from_yaml(self, filename: str):
+        """
+        Load the state of the multi-agent structure from a YAML file.
+
+        Args:
+            filename (str): The name of the file to load the multi-agent structure from.
+
+        Returns:
+            None
+        """
+        try:
+            with open(filename) as f:
+                self.__dict__ = yaml.load(f)
+        except Exception as e:
+            logger.error(e)
+
+    def __repr__(self):
+        return f"{self.__class__.__name__}({self.__dict__})"
+
+    def __str__(self):
+        return f"{self.__class__.__name__}({self.__dict__})"
+
+    def __len__(self):
+        return len(self.agents)
+
+    def __getitem__(self, index):
+        return self.agents[index]
+
+    def __setitem__(self, index, value):
+        self.agents[index] = value
+
+    def __delitem__(self, index):
+        del self.agents[index]
+
+    def __iter__(self):
+        return iter(self.agents)
+
+    def __reversed__(self):
+        return reversed(self.agents)
+
+    def __contains__(self, value):
+        return value in self.agents

swarms/structs/sermon_swarm.py

@@ -0,0 +1,65 @@
+from typing import Union, Sequence, List, Callable
+from swarms.structs.agent import Agent
+
+
+class SermonSwarm:
+    """
+    Represents a swarm of agents that communicate through sermons.
+
+    Args:
+        priest (Agent): The priest agent responsible for generating sermons.
+        agents (Sequence[Agent]): The list of agents in the swarm.
+        max_loops (int, optional): The maximum number of loops to run the agents. Defaults to 5.
+        stop_condition (Union[str, List[str]], optional): The condition(s) that can stop the agents. 
+            Defaults to "stop".
+        stop_function (Union[None, Callable], optional): The function to apply to the sermons before 
+            checking the stop condition. Defaults to None.
+    """
+
+    def __init__(
+        self,
+        priest: Agent,
+        agents: Sequence[Agent],
+        max_loops: int = 5,
+        stop_condition: Union[str, List[str]] = "stop",
+        stop_function: Union[None, Callable] = None,
+        *args,
+        **kwargs
+    ):
+        super().__init__(*args, **kwargs)
+        self.priest = priest
+        self.agents = agents
+        self.max_loops = max_loops
+        self.stop_condition = stop_condition
+        self.stop_function = stop_function
+        
+    def run(self, task: str, *args, **kwargs):
+        """
+        Runs the swarm by generating sermons from the priest and executing the task on each agent.
+
+        Args:
+            task (str): The task to be executed by the agents.
+            *args: Additional positional arguments for the task.
+            **kwargs: Additional keyword arguments for the task.
+        """
+        sermon = self.priest(task, *args, **kwargs)
+        
+        # Add the sermon to the memory of all agents
+        for agent in self.agents:
+            agent.add_message_to_memory(sermon)
+            
+        # Then run the agents
+        for _ in range(self.max_loops):
+            for agent in self.agents:
+                preach = agent.run(task, *args, **kwargs)
+                
+                if self.stop_function:
+                    preach = self.stop_function(preach)
+                
+                if self.stop_condition in preach:
+                    if self.stop_condition is True:
+                        break
+                    
+                    elif self.stop_condition in preach:
+                        break
+                    

swarms/utils/__init__.py

@@ -21,6 +21,8 @@ from swarms.utils.file_processing import (
     parse_tagged_output,
     sanitize_file_path,
     zip_workspace,
+    create_file_in_folder,
+    zip_folders,
 )
 from swarms.utils.find_img_path import find_image_path
 from swarms.utils.json_output_parser import JsonOutputParser
@@ -100,4 +102,6 @@ __all__ = [
     "csv_to_dataframe",
     "dataframe_to_strings",
     "execute_concurrently",
+    "create_file_in_folder",
+    "zip_folders",
 ]

swarms/utils/file_processing.py

@@ -1,10 +1,8 @@
-import logging
-import tempfile
-import shutil
+import json
 import os
 import re
-from swarms.utils.parse_code import extract_code_from_markdown
-import json
+import shutil
+import tempfile
 
 
 def zip_workspace(workspace_path: str, output_filename: str):
@@ -33,33 +31,6 @@ def sanitize_file_path(file_path: str):
     return sanitized_path
 
 
-def parse_tagged_output(output, workspace_path: str):
-    """
-    Parses tagged output and saves files to the workspace directory.
-    Adds logging for each step of the process.
-    """
-    pattern = r"<!--START_FILE_PATH-->(.*?)<!--END_FILE_PATH-->(.*?)<!--START_CONTENT-->(.*?)<!--END_CONTENT-->"
-    files = re.findall(pattern, output, re.DOTALL)
-    if not files:
-        logging.error("No files found in the output to parse.")
-        return
-
-    for file_path, _, content in files:
-        sanitized_path = sanitize_file_path(file_path)
-        content = extract_code_from_markdown(
-            content
-        )  # Remove code block markers
-        full_path = os.path.join(workspace_path, sanitized_path)
-        try:
-            os.makedirs(os.path.dirname(full_path), exist_ok=True)
-            with open(full_path, "w") as file:
-                file.write(content.strip())
-            logging.info(f"File saved: {full_path}")
-        except Exception as e:
-            logging.error(
-                f"Failed to save file {sanitized_path}: {e}"
-            )
-
 
 def load_json(json_string: str):
     """
@@ -90,3 +61,55 @@ def create_file(
     with open(file_path, "w") as file:
         file.write(content)
     return file_path
+
+
+def create_file_in_folder(
+    folder_path: str, file_name: str, content: str
+):
+    """
+    Creates a file in the specified folder with the given file name and content.
+
+    Args:
+        folder_path (str): The path of the folder where the file will be created.
+        file_name (str): The name of the file to be created.
+        content (str): The content to be written to the file.
+
+    Returns:
+        str: The path of the created file.
+    """
+    if not os.path.exists(folder_path):
+        os.makedirs(folder_path)
+
+    # Create the file in the folder
+    file_path = os.path.join(folder_path, file_name)
+    with open(file_path, "w") as file:
+        file.write(content)
+
+    return file_path
+
+
+def zip_folders(folder1_path, folder2_path, zip_file_path):
+    """
+    Zip two folders into a single zip file.
+
+    Args:
+        folder1_path (str): Path to the first folder.
+        folder2_path (str): Path to the second folder.
+        zip_file_path (str): Path to the output zip file.
+
+    Returns:
+        None
+    """
+    # Create a temporary directory
+    with tempfile.TemporaryDirectory() as temp_dir:
+        # Copy both folders into the temporary directory
+        shutil.copytree(
+            folder1_path,
+            os.path.join(temp_dir, os.path.basename(folder1_path)),
+        )
+        shutil.copytree(
+            folder2_path,
+            os.path.join(temp_dir, os.path.basename(folder2_path)),
+        )
+        # Create a zip file that contains the temporary directory
+        shutil.make_archive(zip_file_path, "zip", temp_dir)