ai-mirow-projects
/
swarms
mirror of https://github.com/kyegomez/swarms
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

openAIEmbeddings + agent readme page

Browse Source
pull/39/head
Kye 1 year ago
parent fbeefa140b
commit c112c5d98e
3 changed files with 186 additions and 97 deletions
Show Stats Download Patch File Download Diff File

130
DOCS/ideas/IDEAS.MD
Unescape View File

@ -251,4 +251,132 @@ In the context of swarm LLMs, one could consider an **Omni-Vector Embedding Data
* Handling absurdly long sequences => first transform the objective if it's more than 1000tokens into a txt file similiar to how Claude works => then chunk it into sizes of 8000 seq length embeddings => then embed it and store in the vector database => then connext the agent model to it
* Handling absurdly long sequences => first transform the objective if it's more than 1000tokens into a txt file similiar to how Claude works => then chunk it into sizes of 8000 seq length embeddings => then embed it and store in the vector database => then connext the agent model to it
Given the complexity of the topic, please note that these simplified markdown documents are quite abstract and high level. They can be used as a starting point for further detailed design and implementation:
### Document 1: Hierarchical Swarms
#### Overall Architecture
1. Leader Agent (LA): This agent has the authority to manage and distribute tasks to the Worker Agents (WA).
2. Worker Agents (WAs): These agents perform the tasks assigned by the LA.
#### Simplified Requirements
1. LA should be able to distribute tasks to WAs.
2. WAs should be able to execute tasks and return results to LA.
3. LA should be able to consolidate and process results.
#### Pseudocode
```
create LA
create WAs
for each task in tasks:
LA.distribute_task(WAs, task)
for each WA in WAs:
WA.execute_task()
LA.collect_results(WAs)
LA.process_results()
```
#### General Classes
```python
class LeaderAgent:
def distribute_task(self, WAs, task):
pass
def collect_results(self, WAs):
pass
def process_results(self):
pass
class WorkerAgent:
def execute_task(self):
pass
```
### Document 2: Collaborative Swarms
#### Overall Architecture
1. Collaborative Agents (CAs): These agents work in parallel on different aspects of a task and then collectively determine the best output.
#### Simplified Requirements
1. CAs should be able to work on tasks in parallel.
2. CAs should be able to collaborate to determine the best result.
#### Pseudocode
```
create CAs
for each task in tasks:
for each CA in CAs:
CA.execute_task(task)
CA.collaborate()
```
#### General Classes
```python
class CollaborativeAgent:
def execute_task(self, task):
pass
def collaborate(self):
pass
```
### Document 3: Competitive Swarms
#### Overall Architecture
1. Competitive Agents (CompAs): These agents work independently on the same tasks, and the best result is selected.
#### Simplified Requirements
1. CompAs should be able to work independently on tasks.
2. An evaluation method should be used to select the best result.
#### Pseudocode
```
create CompAs
for each task in tasks:
for each CompA in CompAs:
CompA.execute_task(task)
evaluate_results(CompAs)
```
#### General Classes
```python
class CompetitiveAgent:
def execute_task(self, task):
pass
def evaluate_results(CompAs):
pass
```
Note: In the real world, the complexity of the architecture and requirements will significantly exceed what is presented here. These examples provide a basic starting point but should be expanded upon based on the specifics of the task or problem you're trying to solve.
# Swarms
BabyAGI -> Autogpt's -> tools -> other agents

138
swarms/agents/README.MD
Unescape View File

@ -1,126 +1,80 @@
Given the complexity of the topic, please note that these simplified markdown documents are quite abstract and high level. They can be used as a starting point for further detailed design and implementation:
Introduction to Agents in Swarms
================================
### Document 1: Hierarchical Swarms
Welcome to the revolutionary world of Agents in Swarms. If you're familiar with my philosophy from the Linux world, you'll know that I'm a big believer in simplicity, modularity, and the power of open collaboration. The same principles apply here.
#### Overall Architecture
Agents are the individual building blocks in a swarm. They are the worker bees, each with a specific task, but all working together towards a common goal. In our case, an agent is a combination of a Language Model (LLM), Long Term Memory, and Tools.
1. Leader Agent (LA): This agent has the authority to manage and distribute tasks to the Worker Agents (WA).
2. Worker Agents (WAs): These agents perform the tasks assigned by the LA.
In other words, an agent is:
#### Simplified Requirements
`LLM => Long Term Memory => Tools`
1. LA should be able to distribute tasks to WAs.
2. WAs should be able to execute tasks and return results to LA.
3. LA should be able to consolidate and process results.
That's it. That's as simple as it can get.
#### Pseudocode
Why does this work? Because each component has a specific, well-defined role. The Language Model is the driving force, generating text based on a given prompt. The Long Term Memory stores information that the agent can draw upon to make its output more coherent and contextually relevant. The Tools provide additional capabilities, such as the ability to parse text, search the web, or interact with APIs.
```
create LA
create WAs
for each task in tasks:
LA.distribute_task(WAs, task)
for each WA in WAs:
WA.execute_task()
LA.collect_results(WAs)
LA.process_results()
```
#### General Classes
```python
class LeaderAgent:
def distribute_task(self, WAs, task):
pass
def collect_results(self, WAs):
pass
def process_results(self):
pass
class WorkerAgent:
def execute_task(self):
pass
```
### Document 2: Collaborative Swarms
#### Overall Architecture
1. Collaborative Agents (CAs): These agents work in parallel on different aspects of a task and then collectively determine the best output.
But the real beauty of this system is not just in the individual components, but in how they work together. The output of one becomes the input of another, creating a feedback loop of continuous learning and improvement.
#### Simplified Requirements
And the best part? Our Agent classes are designed to be as simple as humanely possible. They are plug-and-play with any of our language model classes, vector stores, and tools. This means you can easily swap out one component for another, allowing for endless customization and flexibility.
1. CAs should be able to work on tasks in parallel.
2. CAs should be able to collaborate to determine the best result.
#### Pseudocode
The file structure is equally straightforward:
```
create CAs
* memory
* models
* tools
* utils
for each task in tasks:
for each CA in CAs:
CA.execute_task(task)
CA.collaborate()
```
#### General Classes
Each directory contains different components of the swarm. The `models` directory contains the language models, the `memory` directory contains the long-term memory, the `tools` directory contains the tools, the `utils` directory contains various utility functions.
```python
class CollaborativeAgent:
def execute_task(self, task):
pass
Let's see how simple it is to use these components with some examples:
def collaborate(self):
pass
```
```python
# Import the necessary classes
from swarms.agents import Anthropic, HuggingFaceLLM
### Document 3: Competitive Swarms
# Create an instance of the Anthropic class
anthropic = Anthropic(model="claude-2", max_tokens_to_sample=100, temperature=0.8)
#### Overall Architecture
# Use the Anthropic instance to generate text
prompt = "Once upon a time"
stop = ["The end"]
print("Anthropic output:")
print(anthropic.generate(prompt, stop))
1. Competitive Agents (CompAs): These agents work independently on the same tasks, and the best result is selected.
# Create an instance of the HuggingFaceLLM class
huggingface = HuggingFaceLLM(model_id="gpt2", device="cpu", max_length=50)
#### Simplified Requirements
# Use the HuggingFaceLLM instance to generate text
prompt = "Once upon a time"
print("\nHuggingFaceLLM output:")
print(huggingface.generate(prompt))
```
1. CompAs should be able to work independently on tasks.
2. An evaluation method should be used to select the best result.
#### Pseudocode
And to build an agent:
```
create CompAs
```python
from swarms.agents import vectorstore, tool, Agent
for each task in tasks:
for each CompA in CompAs:
CompA.execute_task(task)
# Create an instance of the Agent class
agent = Agent(
llm=huggingface,
memory=vectorstore,
tools=tool,
)
evaluate_results(CompAs)
agent.run("Make me an instagram clone")
```
#### General Classes
```python
class CompetitiveAgent:
def execute_task(self, task):
pass
In conclusion, the Agents in Swarms represent a new way of thinking about AI. They are simple, modular, and highly customizable, allowing you to create powerful AI systems that are more than the sum of their parts. And as always, we're just getting started. There's always room for improvement, for simplification, for making things even better. That's the spirit of open collaboration. That's the spirit of Swarms.
def evaluate_results(CompAs):
pass
```
Thanks for becoming an alpha build user, email kye@apac.ai with all complaints.
Note: In the real world, the complexity of the architecture and requirements will significantly exceed what is presented here. These examples provide a basic starting point but should be expanded upon based on the specifics of the task or problem you're trying to solve.
# Swarms
BabyAGI -> Autogpt's -> tools -> other agents

15
swarms/utils/embeddings/openai.py
Unescape View File

@ -26,8 +26,17 @@ from tenacity import (
wait_exponential,
)
from langchain.embeddings.base import Embeddings
from langchain.utils import get_from_dict_or_env, get_pydantic_field_names
from swarms.utils.embeddings.base import Embeddings
def get_from_dict_or_env(values: dict, key: str, env_key: str, default: Any = None) -> Any:
import os
return values.get(key) or os.getenv(env_key) or default
def get_pydantic_field_names(cls: Any) -> Set[str]:
return set(cls.__annotations__.keys())
logger = logging.getLogger(__name__)
@ -303,8 +312,6 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
} # type: ignore[assignment] # noqa: E501
return openai_args
# please refer to
# https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb
def _get_len_safe_embeddings(
self, texts: List[str], *, engine: str, chunk_size: Optional[int] = None
) -> List[List[float]]:

Write Preview
Loading…
Cancel
Save