Your Name
4 months ago
parent
eb78e72146
commit
9011ca2b9f
@ -1,350 +0,0 @@
|
||||
# `PgVectorVectorStore` Documentation
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Introduction](#introduction)
|
||||
2. [Overview](#overview)
|
||||
3. [Class Definition](#class-definition)
|
||||
4. [Functionality and Usage](#functionality-and-usage)
|
||||
- [Setting Up the Database](#setting-up-the-database)
|
||||
- [Upserting Vectors](#upserting-vectors)
|
||||
- [Loading Vector Entries](#loading-vector-entries)
|
||||
- [Querying Vectors](#querying-vectors)
|
||||
5. [Additional Information](#additional-information)
|
||||
6. [References and Resources](#references-and-resources)
|
||||
|
||||
---
|
||||
|
||||
## 1. Introduction <a name="introduction"></a>
|
||||
|
||||
Welcome to the documentation for the Swarms `PgVectorVectorStore` class! Swarms is a library that provides various memory and storage options for high-dimensional vectors. In this documentation, we will focus on the `PgVectorVectorStore` class, which is a vector storage driver that uses PostgreSQL with the PGVector extension as the underlying storage engine.
|
||||
|
||||
### 1.1 Purpose
|
||||
|
||||
The `PgVectorVectorStore` class allows you to interact with a PostgreSQL database and store high-dimensional vectors efficiently. By using Swarms with PostgreSQL and PGVector, you can manage and work with vector data in your applications with ease.
|
||||
|
||||
### 1.2 Key Features
|
||||
|
||||
- Integration with PostgreSQL and PGVector for vector storage.
|
||||
- Simple and convenient API for upserting vectors, querying, and loading entries.
|
||||
- Support for creating and managing vector collections in PostgreSQL.
|
||||
|
||||
---
|
||||
|
||||
## 2. Overview <a name="overview"></a>
|
||||
|
||||
Before diving into the details of the `PgVectorVectorStore` class, let's provide an overview of its purpose and functionality.
|
||||
|
||||
The `PgVectorVectorStore` class is designed to:
|
||||
|
||||
- Store high-dimensional vectors in a PostgreSQL database with the PGVector extension.
|
||||
- Offer a seamless and efficient way to upsert vectors into the database.
|
||||
- Provide methods for loading individual vector entries or all vector entries in a collection.
|
||||
- Support vector queries, allowing you to find vectors similar to a given query vector.
|
||||
|
||||
In the following sections, we will explore the class definition, its parameters, and how to use it effectively.
|
||||
|
||||
---
|
||||
|
||||
## 3. Class Definition <a name="class-definition"></a>
|
||||
|
||||
Let's start by examining the class definition of `PgVectorVectorStore`, including its attributes and parameters.
|
||||
|
||||
```python
|
||||
class PgVectorVectorStore(BaseVectorStore):
|
||||
"""
|
||||
A vector store driver to Postgres using the PGVector extension.
|
||||
|
||||
Attributes:
|
||||
connection_string: An optional string describing the target Postgres database instance.
|
||||
create_engine_params: Additional configuration params passed when creating the database connection.
|
||||
engine: An optional sqlalchemy Postgres engine to use.
|
||||
table_name: Optionally specify the name of the table to used to store vectors.
|
||||
...
|
||||
"""
|
||||
```
|
||||
|
||||
Attributes:
|
||||
|
||||
- `connection_string` (Optional[str]): An optional string describing the target Postgres database instance.
|
||||
- `create_engine_params` (dict): Additional configuration parameters passed when creating the database connection.
|
||||
- `engine` (Optional[Engine]): An optional SQLAlchemy Postgres engine to use.
|
||||
- `table_name` (str): Optionally specify the name of the table to be used to store vectors.
|
||||
|
||||
### 3.1 Attribute Validators
|
||||
|
||||
The class includes validators for the `connection_string` and `engine` attributes to ensure their proper usage. These validators help maintain consistency in attribute values.
|
||||
|
||||
### 3.2 Initialization
|
||||
|
||||
During initialization, the class checks if an engine is provided. If an engine is not provided, it creates a new database connection using the `connection_string` and `create_engine_params`.
|
||||
|
||||
---
|
||||
|
||||
## 4. Functionality and Usage <a name="functionality-and-usage"></a>
|
||||
|
||||
In this section, we will explore the functionality of the `PgVectorVectorStore` class and provide detailed instructions on how to use it effectively.
|
||||
|
||||
### 4.1 Setting Up the Database <a name="setting-up-the-database"></a>
|
||||
|
||||
Before using the `PgVectorVectorStore` to store and query vectors, you need to set up the database. This includes creating the necessary extensions and database schema. You can do this using the `setup` method.
|
||||
|
||||
```python
|
||||
def setup(
|
||||
self,
|
||||
create_schema: bool = True,
|
||||
install_uuid_extension: bool = True,
|
||||
install_vector_extension: bool = True,
|
||||
) -> None:
|
||||
"""
|
||||
Provides a mechanism to initialize the database schema and extensions.
|
||||
|
||||
Parameters:
|
||||
- create_schema (bool): If True, creates the necessary database schema for vector storage. Default: True.
|
||||
- install_uuid_extension (bool): If True, installs the UUID extension in the database. Default: True.
|
||||
- install_vector_extension (bool): If True, installs the PGVector extension in the database. Default: True.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Example 1: Setting Up the Database
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(
|
||||
connection_string="your-db-connection-string", table_name="your-table-name"
|
||||
)
|
||||
|
||||
# Set up the database with default settings
|
||||
vector_store.setup()
|
||||
```
|
||||
|
||||
#### Example 2: Customized Database Setup
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(
|
||||
connection_string="your-db-connection-string", table_name="your-table-name"
|
||||
)
|
||||
|
||||
# Set up the database with customized settings
|
||||
vector_store.setup(
|
||||
create_schema=False, install_uuid_extension=True, install_vector_extension=True
|
||||
)
|
||||
```
|
||||
|
||||
### 4.2 Upserting Vectors <a name="upserting-vectors"></a>
|
||||
|
||||
The `upsert_vector` method allows you to insert or update a vector in the collection. You can specify the vector, an optional vector ID, namespace, and metadata.
|
||||
|
||||
```python
|
||||
def upsert_vector(
|
||||
self,
|
||||
vector: list[float],
|
||||
vector_id: Optional[str] = None,
|
||||
namespace: Optional[str] = None,
|
||||
meta: Optional[dict] = None,
|
||||
**kwargs,
|
||||
) -> str:
|
||||
"""
|
||||
Inserts or updates a vector in the collection.
|
||||
|
||||
Parameters:
|
||||
- vector (list[float]): The vector to upsert.
|
||||
- vector_id (Optional[str]): An optional ID for the vector. If not provided, a unique ID will be generated.
|
||||
- namespace (Optional[str]): An optional namespace for the vector.
|
||||
- meta (Optional[dict]): An optional metadata dictionary associated with the vector.
|
||||
- **kwargs: Additional keyword arguments.
|
||||
|
||||
Returns:
|
||||
- str: The ID of the upserted vector.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Example: Upserting a Vector
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(
|
||||
connection_string="your-db-connection-string", table_name="your-table-name"
|
||||
)
|
||||
|
||||
# Define a vector and upsert it
|
||||
vector = [0.1, 0.2, 0.3, 0.4]
|
||||
vector_id = "unique-vector-id"
|
||||
namespace = "your-namespace"
|
||||
meta = {"key1": "value1", "key2": "value2"}
|
||||
|
||||
vector_store.upsert_vector(
|
||||
vector=vector, vector_id=vector_id, namespace=namespace, meta=meta
|
||||
)
|
||||
```
|
||||
|
||||
### 4.3 Loading Vector Entries <a name="loading-vector-entries"></a>
|
||||
|
||||
You can load vector entries from the collection using the `load_entry` and `load_entries` methods.
|
||||
|
||||
#### 4
|
||||
|
||||
.3.1 Loading a Single Entry
|
||||
|
||||
The `load_entry` method allows you to load a specific vector entry based on its identifier and optional namespace.
|
||||
|
||||
```python
|
||||
def load_entry(
|
||||
self, vector_id: str, namespace: Optional[str] = None
|
||||
) -> BaseVectorStore.Entry:
|
||||
"""
|
||||
Retrieves a specific vector entry from the collection based on its identifier and optional namespace.
|
||||
|
||||
Parameters:
|
||||
- vector_id (str): The ID of the vector to retrieve.
|
||||
- namespace (Optional[str]): An optional namespace for filtering. Default: None.
|
||||
|
||||
Returns:
|
||||
- BaseVectorStore.Entry: The loaded vector entry.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Example: Loading a Single Entry
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(connection_string="your-db-connection-string", table_name="your-table-name")
|
||||
|
||||
# Load a specific vector entry
|
||||
loaded_entry = vector_store.load_entry(vector_id="unique-vector-id", namespace="your-namespace")
|
||||
|
||||
if loaded_entry is not None:
|
||||
loaded_vector = loaded_entry.vector
|
||||
loaded_meta = loaded_entry.meta
|
||||
# Use the loaded vector and metadata as needed
|
||||
else:
|
||||
# Vector not found
|
||||
```
|
||||
|
||||
#### 4.3.2 Loading Multiple Entries
|
||||
|
||||
The `load_entries` method allows you to load all vector entries from the collection, optionally filtering by namespace.
|
||||
|
||||
```python
|
||||
def load_entries(self, namespace: Optional[str] = None) -> list[BaseVectorStore.Entry]:
|
||||
"""
|
||||
Retrieves all vector entries from the collection, optionally filtering to only those that match the provided namespace.
|
||||
|
||||
Parameters:
|
||||
- namespace (Optional[str]): An optional namespace for filtering. Default: None.
|
||||
|
||||
Returns:
|
||||
- list[BaseVectorStore.Entry]: A list of loaded vector entries.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Example: Loading Multiple Entries
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(
|
||||
connection_string="your-db-connection-string", table_name="your-table-name"
|
||||
)
|
||||
|
||||
# Load all vector entries in the specified namespace
|
||||
entries = vector_store.load_entries(namespace="your-namespace")
|
||||
|
||||
# Process the loaded entries
|
||||
for entry in entries:
|
||||
vector_id = entry.id
|
||||
vector = entry.vector
|
||||
meta = entry.meta
|
||||
|
||||
# Handle the loaded entries as needed
|
||||
```
|
||||
|
||||
### 4.4 Querying Vectors <a name="querying-vectors"></a>
|
||||
|
||||
You can perform vector queries to find vectors similar to a given query vector using the `query` method. You can specify the query string, the maximum number of results to return, and other options.
|
||||
|
||||
```python
|
||||
def query(
|
||||
self,
|
||||
query: str,
|
||||
count: Optional[int] = BaseVectorStore.DEFAULT_QUERY_COUNT,
|
||||
namespace: Optional[str] = None,
|
||||
include_vectors: bool = False,
|
||||
distance_metric: str = "cosine_distance",
|
||||
**kwargs,
|
||||
) -> list[BaseVectorStore.QueryResult]:
|
||||
"""
|
||||
Performs a search on the collection to find vectors similar to the provided input vector,
|
||||
optionally filtering to only those that match the provided namespace.
|
||||
|
||||
Parameters:
|
||||
- query (str): The query string to find similar vectors.
|
||||
- count (Optional[int]): Maximum number of results to return. Default: BaseVectorStore.DEFAULT_QUERY_COUNT.
|
||||
- namespace (Optional[str]): An optional namespace for filtering. Default: None.
|
||||
- include_vectors (bool): If True, includes vectors in the query results. Default: False.
|
||||
- distance_metric (str): The distance metric to use for similarity measurement.
|
||||
Options: "cosine_distance", "l2_distance", "inner_product". Default: "cosine_distance".
|
||||
- **kwargs: Additional keyword arguments.
|
||||
|
||||
Returns:
|
||||
- list[BaseVectorStore.QueryResult]: A list of query results, each containing vector ID, vector (if included), score, and metadata.
|
||||
"""
|
||||
```
|
||||
|
||||
#### Example: Querying Vectors
|
||||
|
||||
```python
|
||||
# Initialize the PgVectorVectorStore instance
|
||||
vector_store = PgVectorVectorStore(
|
||||
connection_string="your-db-connection-string", table_name="your-table-name"
|
||||
)
|
||||
|
||||
# Perform a vector query
|
||||
query_string = "your-query-string"
|
||||
count = 10 # Maximum number of results to return
|
||||
namespace = "your-namespace"
|
||||
include_vectors = False # Set to True to include vectors in results
|
||||
distance_metric = "cosine_distance"
|
||||
|
||||
results = vector_store.query(
|
||||
query=query_string,
|
||||
count=count,
|
||||
namespace=namespace,
|
||||
include_vectors=include_vectors,
|
||||
distance_metric=distance_metric,
|
||||
)
|
||||
|
||||
# Process the query results
|
||||
for result in results:
|
||||
vector_id = result.id
|
||||
vector = result.vector
|
||||
score = result.score
|
||||
meta = result.meta
|
||||
|
||||
# Handle the results as needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Additional Information <a name="additional-information"></a>
|
||||
|
||||
Here are some additional tips and information for using the `PgVectorVectorStore` class effectively:
|
||||
|
||||
- When upserting vectors, you can generate a unique vector ID using a hash of the vector's content to ensure uniqueness.
|
||||
- Consider using namespaces to organize and categorize vectors within your PostgreSQL database.
|
||||
- You can choose from different distance metrics (cosine distance, L2 distance, inner product) for vector querying based on your application's requirements.
|
||||
- Keep your database connection string secure and follow best practices for database access control.
|
||||
|
||||
---
|
||||
|
||||
## 6. References and Resources <a name="references-and-resources"></a>
|
||||
|
||||
Here are some references and resources for further information on Swarms and PostgreSQL with PGVector:
|
||||
|
||||
- [Swarms GitHub Repository](https://github.com/swarms): Swarms library on GitHub for updates and contributions.
|
||||
- [PostgreSQL Official Website](https://www.postgresql.org/): Official PostgreSQL website for documentation and resources.
|
||||
- [PGVector GitHub Repository](https://github.com/ankane/pgvector): PGVector extension on GitHub for detailed information.
|
||||
|
||||
---
|
||||
|
||||
This concludes the documentation for the Swarms `PgVectorVectorStore` class. You now have a comprehensive understanding of how to use Swarms with PostgreSQL and PGVector for vector storage. If you have any further questions or need assistance, please refer to the provided references and resources. Happy coding!
|
||||
@ -1,293 +0,0 @@
|
||||
# `PineconeDB` Documentation
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Introduction](#introduction)
|
||||
2. [PineconeVector Class](#pineconevector-class)
|
||||
3. [Installation](#installation)
|
||||
4. [Usage](#usage)
|
||||
- [Creating a PineconeVector Instance](#creating-a-pineconevector-instance)
|
||||
- [Creating an Index](#creating-an-index)
|
||||
- [Upserting Vectors](#upserting-vectors)
|
||||
- [Querying the Index](#querying-the-index)
|
||||
- [Loading an Entry](#loading-an-entry)
|
||||
- [Loading Entries](#loading-entries)
|
||||
5. [Additional Information](#additional-information)
|
||||
6. [References and Resources](#references-and-resources)
|
||||
|
||||
---
|
||||
|
||||
## 1. Introduction <a name="introduction"></a>
|
||||
|
||||
Welcome to the Swarms documentation! Swarms is a library that provides various memory and storage options for high-dimensional vectors. In this documentation, we will focus on the `PineconeVector` class, which is a vector storage driver that uses Pinecone as the underlying storage engine.
|
||||
|
||||
### 1.1 Purpose
|
||||
|
||||
The `PineconeVector` class allows you to interact with Pinecone, a vector database that enables the storage, search, and retrieval of high-dimensional vectors with speed and low latency. By using Swarms with Pinecone, you can easily manage and work with vector data in your applications without the need to manage infrastructure.
|
||||
|
||||
### 1.2 Key Features
|
||||
|
||||
- Seamless integration with Pinecone for vector storage.
|
||||
- Simple and convenient API for upserting vectors, querying, and loading entries.
|
||||
- Support for creating and managing indexes.
|
||||
|
||||
---
|
||||
|
||||
## 2. PineconeVector Class <a name="pineconevector-class"></a>
|
||||
|
||||
The `PineconeVector` class is the core component of Swarms that interacts with Pinecone for vector storage. Below, we will provide an in-depth overview of this class, including its purpose, parameters, and methods.
|
||||
|
||||
### 2.1 Class Definition
|
||||
|
||||
```python
|
||||
class PineconeVector(BaseVector):
|
||||
```
|
||||
|
||||
### 2.2 Parameters
|
||||
|
||||
The `PineconeVector` class accepts the following parameters during initialization:
|
||||
|
||||
- `api_key` (str): The API key for your Pinecone account.
|
||||
- `index_name` (str): The name of the index to use.
|
||||
- `environment` (str): The environment to use. Either "us-west1-gcp" or "us-east1-gcp".
|
||||
- `project_name` (str, optional): The name of the project to use. Defaults to `None`.
|
||||
- `index` (pinecone.Index, optional): The Pinecone index to use. Defaults to `None`.
|
||||
|
||||
### 2.3 Methods
|
||||
|
||||
The `PineconeVector` class provides several methods for interacting with Pinecone:
|
||||
|
||||
#### 2.3.1 `upsert_vector`
|
||||
|
||||
```python
|
||||
def upsert_vector(
|
||||
self,
|
||||
vector: list[float],
|
||||
vector_id: Optional[str] = None,
|
||||
namespace: Optional[str] = None,
|
||||
meta: Optional[dict] = None,
|
||||
**kwargs
|
||||
) -> str:
|
||||
```
|
||||
|
||||
Upserts a vector into the index.
|
||||
|
||||
- `vector` (list[float]): The vector to upsert.
|
||||
- `vector_id` (Optional[str]): An optional ID for the vector. If not provided, a unique ID will be generated.
|
||||
- `namespace` (Optional[str]): An optional namespace for the vector.
|
||||
- `meta` (Optional[dict]): An optional metadata dictionary associated with the vector.
|
||||
- `**kwargs`: Additional keyword arguments.
|
||||
|
||||
#### 2.3.2 `load_entry`
|
||||
|
||||
```python
|
||||
def load_entry(
|
||||
self, vector_id: str, namespace: Optional[str] = None
|
||||
) -> Optional[BaseVector.Entry]:
|
||||
```
|
||||
|
||||
Loads a single vector from the index.
|
||||
|
||||
- `vector_id` (str): The ID of the vector to load.
|
||||
- `namespace` (Optional[str]): An optional namespace for the vector.
|
||||
|
||||
#### 2.3.3 `load_entries`
|
||||
|
||||
```python
|
||||
def load_entries(self, namespace: Optional[str] = None) -> list[BaseVector.Entry]:
|
||||
```
|
||||
|
||||
Loads all vectors from the index.
|
||||
|
||||
- `namespace` (Optional[str]): An optional namespace for the vectors.
|
||||
|
||||
#### 2.3.4 `query`
|
||||
|
||||
```python
|
||||
def query(
|
||||
self,
|
||||
query: str,
|
||||
count: Optional[int] = None,
|
||||
namespace: Optional[str] = None,
|
||||
include_vectors: bool = False,
|
||||
include_metadata=True,
|
||||
**kwargs
|
||||
) -> list[BaseVector.QueryResult]:
|
||||
```
|
||||
|
||||
Queries the index for vectors similar to the given query string.
|
||||
|
||||
- `query` (str): The query string.
|
||||
- `count` (Optional[int]): The maximum number of results to return. If not provided, a default value is used.
|
||||
- `namespace` (Optional[str]): An optional namespace for the query.
|
||||
- `include_vectors` (bool): Whether to include vectors in the query results.
|
||||
- `include_metadata` (bool): Whether to include metadata in the query results.
|
||||
- `**kwargs`: Additional keyword arguments.
|
||||
|
||||
#### 2.3.5 `create_index`
|
||||
|
||||
```python
|
||||
def create_index(self, name: str, **kwargs) -> None:
|
||||
```
|
||||
|
||||
Creates a new index.
|
||||
|
||||
- `name` (str): The name of the index to create.
|
||||
- `**kwargs`: Additional keyword arguments.
|
||||
|
||||
---
|
||||
|
||||
## 3. Installation <a name="installation"></a>
|
||||
|
||||
To use the Swarms library and the `PineconeVector` class, you will need to install the library and its dependencies. Follow these steps to get started:
|
||||
|
||||
1. Install Swarms:
|
||||
|
||||
```bash
|
||||
pip install swarms
|
||||
```
|
||||
|
||||
2. Install Pinecone:
|
||||
|
||||
You will also need a Pinecone account and API key. Follow the instructions on the Pinecone website to create an account and obtain an API key.
|
||||
|
||||
3. Import the necessary modules in your Python code:
|
||||
|
||||
```python
|
||||
from swarms_memory import PineconeVector
|
||||
```
|
||||
|
||||
Now you're ready to use the `PineconeVector` class to work with Pinecone for vector storage.
|
||||
|
||||
---
|
||||
|
||||
## 4. Usage <a name="usage"></a>
|
||||
|
||||
In this section, we will provide detailed examples of how to use the `PineconeVector` class for vector storage with Pinecone.
|
||||
|
||||
### 4.1 Creating a PineconeVector Instance <a name="creating-a-pineconevector-instance"></a>
|
||||
|
||||
To get started, you need to create an instance of the `PineconeVector` class. You will need your Pinecone API key, the name of the index you want to use, and the environment. You can also specify an optional project name if you have one.
|
||||
|
||||
```python
|
||||
pv = PineconeVector(
|
||||
api_key="your-api-key",
|
||||
index_name="your-index-name",
|
||||
environment="us-west1-gcp",
|
||||
project_name="your-project-name",
|
||||
)
|
||||
```
|
||||
|
||||
### 4.2 Creating an Index <a name="creating-an-index"></a>
|
||||
|
||||
Before you can upsert vectors, you need to create an index in Pinecone. You can use the `create_index` method for this purpose.
|
||||
|
||||
```python
|
||||
pv.create_index("your-index-name")
|
||||
```
|
||||
|
||||
### 4.3 Upserting Vectors <a name="upserting-vectors"></a>
|
||||
|
||||
You can upsert vectors into the Pine
|
||||
|
||||
cone index using the `upsert_vector` method. This method allows you to specify the vector, an optional vector ID, namespace, and metadata.
|
||||
|
||||
```python
|
||||
vector = [0.1, 0.2, 0.3, 0.4]
|
||||
vector_id = "unique-vector-id"
|
||||
namespace = "your-namespace"
|
||||
meta = {"key1": "value1", "key2": "value2"}
|
||||
|
||||
pv.upsert_vector(vector=vector, vector_id=vector_id, namespace=namespace, meta=meta)
|
||||
```
|
||||
|
||||
### 4.4 Querying the Index <a name="querying-the-index"></a>
|
||||
|
||||
You can query the Pinecone index to find vectors similar to a given query string using the `query` method. You can specify the query string, the maximum number of results to return, and other options.
|
||||
|
||||
```python
|
||||
query_string = "your-query-string"
|
||||
count = 10 # Maximum number of results to return
|
||||
namespace = "your-namespace"
|
||||
include_vectors = False # Set to True to include vectors in results
|
||||
include_metadata = True
|
||||
|
||||
results = pv.query(
|
||||
query=query_string,
|
||||
count=count,
|
||||
namespace=namespace,
|
||||
include_vectors=include_vectors,
|
||||
include_metadata=include_metadata,
|
||||
)
|
||||
|
||||
# Process the query results
|
||||
for result in results:
|
||||
vector_id = result.id
|
||||
vector = result.vector
|
||||
score = result.score
|
||||
meta = result.meta
|
||||
|
||||
# Handle the results as needed
|
||||
```
|
||||
|
||||
### 4.5 Loading an Entry <a name="loading-an-entry"></a>
|
||||
|
||||
You can load a single vector entry from the Pinecone index using the `load_entry` method. Provide the vector ID and an optional namespace.
|
||||
|
||||
```python
|
||||
vector_id = "your-vector-id"
|
||||
namespace = "your-namespace"
|
||||
|
||||
entry = pv.load_entry(vector_id=vector_id, namespace=namespace)
|
||||
|
||||
if entry is not None:
|
||||
loaded_vector = entry.vector
|
||||
loaded_meta = entry.meta
|
||||
|
||||
# Use the loaded vector and metadata
|
||||
else:
|
||||
# Vector not found
|
||||
```
|
||||
|
||||
### 4.6 Loading Entries <a name="loading-entries"></a>
|
||||
|
||||
To load all vectors from the Pinecone index, you can use the `load_entries` method. You can also specify an optional namespace.
|
||||
|
||||
```python
|
||||
namespace = "your-namespace"
|
||||
|
||||
entries = pv.load_entries(namespace=namespace)
|
||||
|
||||
# Process the loaded entries
|
||||
for entry in entries:
|
||||
vector_id = entry.id
|
||||
vector = entry.vector
|
||||
meta = entry.meta
|
||||
|
||||
# Handle the loaded entries as needed
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 5. Additional Information <a name="additional-information"></a>
|
||||
|
||||
In this section, we provide additional information and tips for using the `PineconeVector` class effectively.
|
||||
|
||||
- When upserting vectors, you can generate a unique vector ID using a hash of the vector's content to ensure uniqueness.
|
||||
- Consider using namespaces to organize and categorize vectors within your Pinecone index.
|
||||
- Pinecone provides powerful querying capabilities, so be sure to explore and leverage its features to retrieve relevant vectors efficiently.
|
||||
- Keep your Pinecone API key secure and follow Pinecone's best practices for API key management.
|
||||
|
||||
---
|
||||
|
||||
## 6. References and Resources <a name="references-and-resources"></a>
|
||||
|
||||
Here are some references and resources for further information on Pinecone and Swarms:
|
||||
|
||||
- [Pinecone Website](https://www.pinecone.io/): Official Pinecone website for documentation and resources.
|
||||
- [Pinecone Documentation](https://docs.pinecone.io/): Detailed documentation for Pinecone.
|
||||
- [Swarms GitHub Repository](https://github.com/swarms): Swarms library on GitHub for updates and contributions.
|
||||
|
||||
---
|
||||
|
||||
This concludes the documentation for the Swarms library and the `PineconeVector` class. You now have a deep understanding of how to use Swarms with Pinecone for vector storage. If you have any further questions or need assistance, please refer to the provided references and resources. Happy coding!
|
||||
@ -1,86 +0,0 @@
|
||||
# Qdrant Client Library
|
||||
|
||||
## Overview
|
||||
|
||||
The Qdrant Client Library is designed for interacting with the Qdrant vector database, allowing efficient storage and retrieval of high-dimensional vector data. It integrates with machine learning models for embedding and is particularly suited for search and recommendation systems.
|
||||
|
||||
## Installation
|
||||
|
||||
```python
|
||||
pip install qdrant-client sentence-transformers httpx
|
||||
```
|
||||
|
||||
## Class Definition: Qdrant
|
||||
|
||||
```python
|
||||
class Qdrant:
|
||||
def __init__(
|
||||
self,
|
||||
api_key: str,
|
||||
host: str,
|
||||
port: int = 6333,
|
||||
collection_name: str = "qdrant",
|
||||
model_name: str = "BAAI/bge-small-en-v1.5",
|
||||
https: bool = True,
|
||||
):
|
||||
...
|
||||
```
|
||||
|
||||
### Constructor Parameters
|
||||
|
||||
| Parameter | Type | Description | Default Value |
|
||||
|-----------------|---------|--------------------------------------------------|-----------------------|
|
||||
| api_key | str | API key for authentication. | - |
|
||||
| host | str | Host address of the Qdrant server. | - |
|
||||
| port | int | Port number for the Qdrant server. | 6333 |
|
||||
| collection_name | str | Name of the collection to be used or created. | "qdrant" |
|
||||
| model_name | str | Name of the sentence transformer model. | "BAAI/bge-small-en-v1.5" |
|
||||
| https | bool | Flag to use HTTPS for connection. | True |
|
||||
|
||||
### Methods
|
||||
|
||||
#### `_load_embedding_model(model_name: str)`
|
||||
|
||||
Loads the sentence embedding model.
|
||||
|
||||
#### `_setup_collection()`
|
||||
|
||||
Checks if the specified collection exists in Qdrant; if not, creates it.
|
||||
|
||||
#### `add_vectors(docs: List[dict]) -> OperationResponse`
|
||||
|
||||
Adds vectors to the Qdrant collection.
|
||||
|
||||
#### `search_vectors(query: str, limit: int = 3) -> SearchResult`
|
||||
|
||||
Searches the Qdrant collection for vectors similar to the query vector.
|
||||
|
||||
## Usage Examples
|
||||
|
||||
### Example 1: Setting Up the Qdrant Client
|
||||
|
||||
```python
|
||||
from qdrant_client import Qdrant
|
||||
|
||||
qdrant_client = Qdrant(api_key="your_api_key", host="localhost", port=6333)
|
||||
```
|
||||
|
||||
### Example 2: Adding Vectors to a Collection
|
||||
|
||||
```python
|
||||
documents = [{"page_content": "Sample text 1"}, {"page_content": "Sample text 2"}]
|
||||
|
||||
operation_info = qdrant_client.add_vectors(documents)
|
||||
print(operation_info)
|
||||
```
|
||||
|
||||
### Example 3: Searching for Vectors
|
||||
|
||||
```python
|
||||
search_result = qdrant_client.search_vectors("Sample search query")
|
||||
print(search_result)
|
||||
```
|
||||
|
||||
## Further Information
|
||||
|
||||
Refer to the [Qdrant Documentation](https://qdrant.tech/docs) for more details on the Qdrant vector database.
|
||||
@ -1,250 +0,0 @@
|
||||
# Short-Term Memory Module Documentation
|
||||
|
||||
## Introduction
|
||||
The Short-Term Memory module is a component of the SWARMS framework designed for managing short-term and medium-term memory in a multi-agent system. This documentation provides a detailed explanation of the Short-Term Memory module, its purpose, functions, and usage.
|
||||
|
||||
### Purpose
|
||||
The Short-Term Memory module serves the following purposes:
|
||||
1. To store and manage messages in short-term memory.
|
||||
2. To provide functions for retrieving, updating, and clearing memory.
|
||||
3. To facilitate searching for specific terms within the memory.
|
||||
4. To enable saving and loading memory data to/from a file.
|
||||
|
||||
### Class Definition
|
||||
```python
|
||||
class ShortTermMemory(BaseStructure):
|
||||
def __init__(
|
||||
self,
|
||||
return_str: bool = True,
|
||||
autosave: bool = True,
|
||||
*args,
|
||||
**kwargs,
|
||||
):
|
||||
...
|
||||
```
|
||||
|
||||
#### Parameters
|
||||
| Parameter | Type | Default Value | Description |
|
||||
|---------------------|----------|---------------|------------------------------------------------------------------------------------------------------------------|
|
||||
| `return_str` | bool | True | If True, returns memory as a string. |
|
||||
| `autosave` | bool | True | If True, enables automatic saving of memory data to a file. |
|
||||
| `*args`, `**kwargs` | | | Additional arguments and keyword arguments (not used in the constructor but allowed for flexibility). |
|
||||
|
||||
### Functions
|
||||
|
||||
#### 1. `add`
|
||||
```python
|
||||
def add(self, role: str = None, message: str = None, *args, **kwargs):
|
||||
```
|
||||
|
||||
- Adds a message to the short-term memory.
|
||||
- Parameters:
|
||||
- `role` (str, optional): Role associated with the message.
|
||||
- `message` (str, optional): The message to be added.
|
||||
- Returns: The added memory.
|
||||
|
||||
##### Example 1: Adding a Message to Short-Term Memory
|
||||
```python
|
||||
memory.add(role="Agent 1", message="Received task assignment.")
|
||||
```
|
||||
|
||||
##### Example 2: Adding Multiple Messages to Short-Term Memory
|
||||
```python
|
||||
messages = [("Agent 1", "Received task assignment."), ("Agent 2", "Task completed.")]
|
||||
for role, message in messages:
|
||||
memory.add(role=role, message=message)
|
||||
```
|
||||
|
||||
#### 2. `get_short_term`
|
||||
```python
|
||||
def get_short_term(self):
|
||||
```
|
||||
|
||||
- Retrieves the short-term memory.
|
||||
- Returns: The contents of the short-term memory.
|
||||
|
||||
##### Example: Retrieving Short-Term Memory
|
||||
```python
|
||||
short_term_memory = memory.get_short_term()
|
||||
for entry in short_term_memory:
|
||||
print(entry["role"], ":", entry["message"])
|
||||
```
|
||||
|
||||
#### 3. `get_medium_term`
|
||||
```python
|
||||
def get_medium_term(self):
|
||||
```
|
||||
|
||||
- Retrieves the medium-term memory.
|
||||
- Returns: The contents of the medium-term memory.
|
||||
|
||||
##### Example: Retrieving Medium-Term Memory
|
||||
```python
|
||||
medium_term_memory = memory.get_medium_term()
|
||||
for entry in medium_term_memory:
|
||||
print(entry["role"], ":", entry["message"])
|
||||
```
|
||||
|
||||
#### 4. `clear_medium_term`
|
||||
```python
|
||||
def clear_medium_term(self):
|
||||
```
|
||||
|
||||
- Clears the medium-term memory.
|
||||
|
||||
##### Example: Clearing Medium-Term Memory
|
||||
```python
|
||||
memory.clear_medium_term()
|
||||
```
|
||||
|
||||
#### 5. `get_short_term_memory_str`
|
||||
```python
|
||||
def get_short_term_memory_str(self, *args, **kwargs):
|
||||
```
|
||||
|
||||
- Retrieves the short-term memory as a string.
|
||||
- Returns: A string representation of the short-term memory.
|
||||
|
||||
##### Example: Getting Short-Term Memory as a String
|
||||
```python
|
||||
short_term_memory_str = memory.get_short_term_memory_str()
|
||||
print(short_term_memory_str)
|
||||
```
|
||||
|
||||
#### 6. `update_short_term`
|
||||
```python
|
||||
def update_short_term(self, index, role: str, message: str, *args, **kwargs):
|
||||
```
|
||||
|
||||
- Updates a message in the short-term memory.
|
||||
- Parameters:
|
||||
- `index` (int): The index of the message to update.
|
||||
- `role` (str): New role for the message.
|
||||
- `message` (str): New message content.
|
||||
- Returns: None.
|
||||
|
||||
##### Example: Updating a Message in Short-Term Memory
|
||||
```python
|
||||
memory.update_short_term(
|
||||
index=0, role="Updated Role", message="Updated message content."
|
||||
)
|
||||
```
|
||||
|
||||
#### 7. `clear`
|
||||
```python
|
||||
def clear(self):
|
||||
```
|
||||
|
||||
- Clears the short-term memory.
|
||||
|
||||
##### Example: Clearing Short-Term Memory
|
||||
```python
|
||||
memory.clear()
|
||||
```
|
||||
|
||||
#### 8. `search_memory`
|
||||
```python
|
||||
def search_memory(self, term):
|
||||
```
|
||||
|
||||
- Searches the memory for a specific term.
|
||||
- Parameters:
|
||||
- `term` (str): The term to search for.
|
||||
- Returns: A dictionary containing search results for short-term and medium-term memory.
|
||||
|
||||
##### Example: Searching Memory for a Term
|
||||
```python
|
||||
search_results = memory.search_memory("task")
|
||||
print("Short-Term Memory Results:", search_results["short_term"])
|
||||
print("Medium-Term Memory Results:", search_results["medium_term"])
|
||||
```
|
||||
|
||||
#### 9. `return_shortmemory_as_str`
|
||||
```python
|
||||
def return_shortmemory_as_str(self):
|
||||
```
|
||||
|
||||
- Returns the memory as a string.
|
||||
|
||||
##### Example: Returning Short-Term Memory as a String
|
||||
```python
|
||||
short_term_memory_str = memory.return_shortmemory_as_str()
|
||||
print(short_term_memory_str)
|
||||
```
|
||||
|
||||
#### 10. `move_to_medium_term`
|
||||
```python
|
||||
def move_to_medium_term(self, index):
|
||||
```
|
||||
|
||||
- Moves a message from the short-term memory to the medium-term memory.
|
||||
- Parameters:
|
||||
- `index` (int): The index of the message to move.
|
||||
|
||||
##### Example: Moving a Message to Medium-Term Memory
|
||||
```python
|
||||
memory.move_to_medium_term(index=0)
|
||||
```
|
||||
|
||||
#### 11. `return_medium_memory_as_str`
|
||||
```python
|
||||
def return_medium_memory_as_str(self):
|
||||
```
|
||||
|
||||
- Returns the medium-term memory as a string.
|
||||
|
||||
##### Example: Returning Medium-Term Memory as a String
|
||||
```python
|
||||
medium_term_memory_str = memory.return_medium_memory_as_str()
|
||||
print(medium_term_memory_str)
|
||||
```
|
||||
|
||||
#### 12. `save_to_file`
|
||||
```python
|
||||
def save_to_file(self, filename: str):
|
||||
```
|
||||
|
||||
- Saves the memory data to a file.
|
||||
- Parameters:
|
||||
- `filename` (str): The name of the file to save the data to.
|
||||
|
||||
##### Example: Saving Memory Data to a File
|
||||
```python
|
||||
memory.save_to_file("memory_data.json")
|
||||
```
|
||||
|
||||
#### 13. `load_from_file`
|
||||
```python
|
||||
def load_from_file(self, filename: str, *args, **kwargs):
|
||||
```
|
||||
|
||||
- Loads memory data from a file.
|
||||
- Parameters:
|
||||
- `filename` (str): The name of the file to load data from.
|
||||
|
||||
##### Example: Loading Memory Data from a File
|
||||
```python
|
||||
memory.load_from_file("memory_data.json")
|
||||
```
|
||||
|
||||
### Additional Information and Tips
|
||||
|
||||
- To use the Short-Term Memory module effectively, consider the following tips:
|
||||
- Use the `add` function to store messages in short-term memory.
|
||||
-
|
||||
|
||||
Retrieve memory contents using `get_short_term` and `get_medium_term` functions.
|
||||
- Clear memory as needed using `clear` and `clear_medium_term` functions.
|
||||
- Search for specific terms within the memory using the `search_memory` function.
|
||||
- Save and load memory data to/from files using `save_to_file` and `load_from_file` functions.
|
||||
|
||||
- Ensure proper exception handling when using memory functions to handle potential errors gracefully.
|
||||
|
||||
- When using the `search_memory` function, iterate through the results dictionary to access search results for short-term and medium-term memory.
|
||||
|
||||
### References and Resources
|
||||
|
||||
- For more information on multi-agent systems and memory management, refer to the SWARMS framework documentation: [SWARMS Documentation](https://docs.swarms.world/).
|
||||
|
||||
- For advanced memory management and customization, explore the SWARMS framework source code.
|
||||
|
||||
@ -1,204 +0,0 @@
|
||||
# Weaviate API Client Documentation
|
||||
|
||||
## Overview
|
||||
|
||||
The Weaviate API Client is an interface to Weaviate, a vector database with a GraphQL API. This client allows you to interact with Weaviate programmatically, making it easier to create collections, add objects, query data, update objects, and delete objects within your Weaviate instance.
|
||||
|
||||
This documentation provides a comprehensive guide on how to use the Weaviate API Client, including its initialization, methods, and usage examples.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Installation](#installation)
|
||||
- [Initialization](#initialization)
|
||||
- [Methods](#methods)
|
||||
- [create_collection](#create-collection)
|
||||
- [add](#add)
|
||||
- [query](#query)
|
||||
- [update](#update)
|
||||
- [delete](#delete)
|
||||
- [Examples](#examples)
|
||||
|
||||
## Installation
|
||||
|
||||
Before using the Weaviate API Client, make sure to install the `swarms` library. You can install it using pip:
|
||||
|
||||
```bash
|
||||
pip install swarms
|
||||
```
|
||||
|
||||
## Initialization
|
||||
|
||||
To use the Weaviate API Client, you need to initialize an instance of the `WeaviateDB` class. Here are the parameters you can pass to the constructor:
|
||||
|
||||
| Parameter | Type | Description |
|
||||
|----------------------|----------------|----------------------------------------------------------------------------------------------------------------------------------|
|
||||
| `http_host` | str | The HTTP host of the Weaviate server. |
|
||||
| `http_port` | str | The HTTP port of the Weaviate server. |
|
||||
| `http_secure` | bool | Whether to use HTTPS. |
|
||||
| `grpc_host` | Optional[str] | The gRPC host of the Weaviate server. (Optional) |
|
||||
| `grpc_port` | Optional[str] | The gRPC port of the Weaviate server. (Optional) |
|
||||
| `grpc_secure` | Optional[bool] | Whether to use gRPC over TLS. (Optional) |
|
||||
| `auth_client_secret` | Optional[Any] | The authentication client secret. (Optional) |
|
||||
| `additional_headers` | Optional[Dict[str, str]] | Additional headers to send with requests. (Optional) |
|
||||
| `additional_config` | Optional[weaviate.AdditionalConfig] | Additional configuration for the client. (Optional) |
|
||||
| `connection_params` | Dict[str, Any] | Dictionary containing connection parameters. This parameter is used internally and can be ignored in most cases. |
|
||||
|
||||
Here's an example of how to initialize a WeaviateDB:
|
||||
|
||||
```python
|
||||
from swarms_memory import WeaviateDB
|
||||
|
||||
weaviate_client = WeaviateDB(
|
||||
http_host="YOUR_HTTP_HOST",
|
||||
http_port="YOUR_HTTP_PORT",
|
||||
http_secure=True,
|
||||
grpc_host="YOUR_gRPC_HOST",
|
||||
grpc_port="YOUR_gRPC_PORT",
|
||||
grpc_secure=True,
|
||||
auth_client_secret="YOUR_APIKEY",
|
||||
additional_headers={"X-OpenAI-Api-Key": "YOUR_OPENAI_APIKEY"},
|
||||
additional_config=None, # You can pass additional configuration here
|
||||
)
|
||||
```
|
||||
|
||||
## Methods
|
||||
|
||||
### `create_collection`
|
||||
|
||||
The `create_collection` method allows you to create a new collection in Weaviate. A collection is a container for storing objects with specific properties.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `name` (str): The name of the collection.
|
||||
- `properties` (List[Dict[str, Any]]): A list of dictionaries specifying the properties of objects to be stored in the collection.
|
||||
- `vectorizer_config` (Any, optional): Additional vectorizer configuration for the collection. (Optional)
|
||||
|
||||
#### Usage
|
||||
|
||||
```python
|
||||
weaviate_client.create_collection(
|
||||
name="my_collection",
|
||||
properties=[
|
||||
{"name": "property1", "dataType": ["string"]},
|
||||
{"name": "property2", "dataType": ["int"]},
|
||||
],
|
||||
vectorizer_config=None, # Optional vectorizer configuration
|
||||
)
|
||||
```
|
||||
|
||||
### `add`
|
||||
|
||||
The `add` method allows you to add an object to a specified collection in Weaviate.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `collection_name` (str): The name of the collection where the object will be added.
|
||||
- `properties` (Dict[str, Any]): A dictionary specifying the properties of the object to be added.
|
||||
|
||||
#### Usage
|
||||
|
||||
```python
|
||||
weaviate_client.add(
|
||||
collection_name="my_collection", properties={"property1": "value1", "property2": 42}
|
||||
)
|
||||
```
|
||||
|
||||
### `query`
|
||||
|
||||
The `query` method allows you to query objects from a specified collection in Weaviate.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `collection_name` (str): The name of the collection to query.
|
||||
- `query` (str): The query string specifying the search criteria.
|
||||
- `limit` (int, optional): The maximum number of results to return. (Default: 10)
|
||||
|
||||
#### Usage
|
||||
|
||||
```python
|
||||
results = weaviate_client.query(
|
||||
collection_name="my_collection",
|
||||
query="property1:value1",
|
||||
limit=20 # Optional, specify the limit
|
||||
|
||||
if needed
|
||||
)
|
||||
```
|
||||
|
||||
### `update`
|
||||
|
||||
The `update` method allows you to update an object in a specified collection in Weaviate.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `collection_name` (str): The name of the collection where the object exists.
|
||||
- `object_id` (str): The ID of the object to be updated.
|
||||
- `properties` (Dict[str, Any]): A dictionary specifying the properties to update.
|
||||
|
||||
#### Usage
|
||||
|
||||
```python
|
||||
weaviate_client.update(
|
||||
collection_name="my_collection",
|
||||
object_id="object123",
|
||||
properties={"property1": "new_value", "property2": 99},
|
||||
)
|
||||
```
|
||||
|
||||
### `delete`
|
||||
|
||||
The `delete` method allows you to delete an object from a specified collection in Weaviate.
|
||||
|
||||
#### Parameters
|
||||
|
||||
- `collection_name` (str): The name of the collection from which to delete the object.
|
||||
- `object_id` (str): The ID of the object to delete.
|
||||
|
||||
#### Usage
|
||||
|
||||
```python
|
||||
weaviate_client.delete(collection_name="my_collection", object_id="object123")
|
||||
```
|
||||
|
||||
## Examples
|
||||
|
||||
Here are three examples demonstrating how to use the Weaviate API Client for common tasks:
|
||||
|
||||
### Example 1: Creating a Collection
|
||||
|
||||
```python
|
||||
weaviate_client.create_collection(
|
||||
name="people",
|
||||
properties=[
|
||||
{"name": "name", "dataType": ["string"]},
|
||||
{"name": "age", "dataType": ["int"]},
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
### Example 2: Adding an Object
|
||||
|
||||
```python
|
||||
weaviate_client.add(collection_name="people", properties={"name": "John", "age": 30})
|
||||
```
|
||||
|
||||
### Example 3: Querying Objects
|
||||
|
||||
```python
|
||||
results = weaviate_client.query(collection_name="people", query="name:John", limit=5)
|
||||
```
|
||||
|
||||
These examples cover the basic operations of creating collections, adding objects, and querying objects using the Weaviate API Client.
|
||||
|
||||
## Additional Information and Tips
|
||||
|
||||
- If you encounter any errors during the operations, the client will raise exceptions with informative error messages.
|
||||
- You can explore more advanced features and configurations in the Weaviate documentation.
|
||||
- Make sure to handle authentication and security appropriately when using the client in production environments.
|
||||
|
||||
## References and Resources
|
||||
|
||||
- [Weaviate Documentation](https://weaviate.readthedocs.io/en/latest/): Official documentation for Weaviate.
|
||||
- [Weaviate GitHub Repository](https://github.com/semi-technologies/weaviate): The source code and issue tracker for Weaviate.
|
||||
|
||||
This documentation provides a comprehensive guide on using the Weaviate API Client to interact with Weaviate, making it easier to manage and query your data.
|
||||
Loading…
Reference in new issue