Kye Gomez
7 months ago
parent
96e9cfd496
commit
7f577acca3
File diff suppressed because it is too large
Load Diff
@ -0,0 +1,150 @@
|
|||||||
|
# Documentation for GPT4o Module
|
||||||
|
|
||||||
|
## Overview and Introduction
|
||||||
|
|
||||||
|
The `GPT4o` module is a multi-modal conversational model based on OpenAI's GPT-4 architecture. It extends the functionality of the `BaseMultiModalModel` class, enabling it to handle both text and image inputs for generating diverse and contextually rich responses. This module leverages the power of the GPT-4 model to enhance interactions by integrating visual information with textual prompts, making it highly relevant for applications requiring multi-modal understanding and response generation.
|
||||||
|
|
||||||
|
### Key Concepts
|
||||||
|
- **Multi-Modal Model**: A model that can process and generate responses based on multiple types of inputs, such as text and images.
|
||||||
|
- **System Prompt**: A predefined prompt to guide the conversation flow.
|
||||||
|
- **Temperature**: A parameter that controls the randomness of the response generation.
|
||||||
|
- **Max Tokens**: The maximum number of tokens (words or word pieces) in the generated response.
|
||||||
|
|
||||||
|
## Class Definition
|
||||||
|
|
||||||
|
### `GPT4o` Class
|
||||||
|
|
||||||
|
|
||||||
|
### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Description |
|
||||||
|
|-----------------|--------|--------------------------------------------------------------------------------------|
|
||||||
|
| `system_prompt` | `str` | The system prompt to be used in the conversation. |
|
||||||
|
| `temperature` | `float`| The temperature parameter for generating diverse responses. Default is `0.1`. |
|
||||||
|
| `max_tokens` | `int` | The maximum number of tokens in the generated response. Default is `300`. |
|
||||||
|
| `openai_api_key`| `str` | The API key for accessing the OpenAI GPT-4 API. |
|
||||||
|
| `*args` | | Additional positional arguments. |
|
||||||
|
| `**kwargs` | | Additional keyword arguments. |
|
||||||
|
|
||||||
|
## Functionality and Usage
|
||||||
|
|
||||||
|
### `encode_image` Function
|
||||||
|
|
||||||
|
The `encode_image` function is used to encode an image file into a base64 string format, which can then be included in the request to the GPT-4 API.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Description |
|
||||||
|
|---------------|--------|----------------------------------------------|
|
||||||
|
| `image_path` | `str` | The local path to the image file to be encoded. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Return Type | Description |
|
||||||
|
|-------------|---------------------------------|
|
||||||
|
| `str` | The base64 encoded string of the image. |
|
||||||
|
|
||||||
|
### `GPT4o.__init__` Method
|
||||||
|
|
||||||
|
The constructor for the `GPT4o` class initializes the model with the specified parameters and sets up the OpenAI client.
|
||||||
|
|
||||||
|
### `GPT4o.run` Method
|
||||||
|
|
||||||
|
The `run` method executes the GPT-4o model to generate a response based on the provided task and optional image.
|
||||||
|
|
||||||
|
#### Parameters
|
||||||
|
|
||||||
|
| Parameter | Type | Description |
|
||||||
|
|---------------|--------|----------------------------------------------------|
|
||||||
|
| `task` | `str` | The task or user prompt for the conversation. |
|
||||||
|
| `local_img` | `str` | The local path to the image file. |
|
||||||
|
| `img` | `str` | The URL of the image. |
|
||||||
|
| `*args` | | Additional positional arguments. |
|
||||||
|
| `**kwargs` | | Additional keyword arguments. |
|
||||||
|
|
||||||
|
#### Returns
|
||||||
|
|
||||||
|
| Return Type | Description |
|
||||||
|
|-------------|--------------------------------------------------|
|
||||||
|
| `str` | The generated response from the GPT-4o model. |
|
||||||
|
|
||||||
|
## Usage Examples
|
||||||
|
|
||||||
|
### Example 1: Basic Text Prompt
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import GPT4o
|
||||||
|
|
||||||
|
# Initialize the model
|
||||||
|
model = GPT4o(
|
||||||
|
system_prompt="You are a helpful assistant.",
|
||||||
|
temperature=0.7,
|
||||||
|
max_tokens=150,
|
||||||
|
openai_api_key="your_openai_api_key"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Define the task
|
||||||
|
task = "What is the capital of France?"
|
||||||
|
|
||||||
|
# Generate response
|
||||||
|
response = model.run(task)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 2: Text Prompt with Local Image
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import GPT4o
|
||||||
|
|
||||||
|
# Initialize the model
|
||||||
|
model = GPT4o(
|
||||||
|
system_prompt="Describe the image content.",
|
||||||
|
temperature=0.5,
|
||||||
|
max_tokens=200,
|
||||||
|
openai_api_key="your_openai_api_key"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Define the task and image path
|
||||||
|
task = "Describe the content of this image."
|
||||||
|
local_img = "path/to/your/image.jpg"
|
||||||
|
|
||||||
|
# Generate response
|
||||||
|
response = model.run(task, local_img=local_img)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Example 3: Text Prompt with Image URL
|
||||||
|
|
||||||
|
```python
|
||||||
|
from swarms import GPT4o
|
||||||
|
|
||||||
|
# Initialize the model
|
||||||
|
model = GPT4o(
|
||||||
|
system_prompt="You are a visual assistant.",
|
||||||
|
temperature=0.6,
|
||||||
|
max_tokens=250,
|
||||||
|
openai_api_key="your_openai_api_key"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Define the task and image URL
|
||||||
|
task = "What can you tell about the scenery in this image?"
|
||||||
|
img_url = "http://example.com/image.jpg"
|
||||||
|
|
||||||
|
# Generate response
|
||||||
|
response = model.run(task, img=img_url)
|
||||||
|
print(response)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Additional Information and Tips
|
||||||
|
|
||||||
|
- **API Key Management**: Ensure that your OpenAI API key is securely stored and managed. Do not hard-code it in your scripts. Use environment variables or secure storage solutions.
|
||||||
|
- **Image Encoding**: The `encode_image` function is crucial for converting images to a base64 format suitable for API requests. Ensure that the images are accessible and properly formatted.
|
||||||
|
- **Temperature Parameter**: Adjust the `temperature` parameter to control the creativity of the model's responses. Lower values make the output more deterministic, while higher values increase randomness.
|
||||||
|
- **Token Limit**: Be mindful of the `max_tokens` parameter to avoid exceeding the API's token limits. This parameter controls the length of the generated responses.
|
||||||
|
|
||||||
|
## References and Resources
|
||||||
|
|
||||||
|
- [OpenAI API Documentation](https://beta.openai.com/docs/)
|
||||||
|
- [Python Base64 Encoding](https://docs.python.org/3/library/base64.html)
|
||||||
|
- [dotenv Documentation](https://saurabh-kumar.com/python-dotenv/)
|
||||||
|
- [BaseMultiModalModel Documentation](https://swarms.apac.ai)
|
||||||
Loading…
Reference in new issue