From 5fabfb06f07263f3c13f5b3fb68266bb7a0542cd Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 22 Jul 2024 16:24:05 -0400 Subject: [PATCH 01/14] Expand the configuration guide with examples and instructions on using custom profiles, including TTS provider setup --- docs/getting-started/getting-started.mdx | 7 +-- docs/getting-started/introduction.mdx | 5 +- docs/hardware/01-light.mdx | 2 +- docs/mint.json | 1 + docs/software/configure.mdx | 57 +++++++++++++++++++++- docs/software/installation.mdx | 4 +- docs/software/introduction.mdx | 8 +++ docs/software/run.mdx | 4 +- software/source/server/profiles/default.py | 4 ++ 9 files changed, 78 insertions(+), 14 deletions(-) create mode 100644 docs/software/introduction.mdx diff --git a/docs/getting-started/getting-started.mdx b/docs/getting-started/getting-started.mdx index 4812e9f..2a587bb 100644 --- a/docs/getting-started/getting-started.mdx +++ b/docs/getting-started/getting-started.mdx @@ -5,12 +5,9 @@ description: "Preparing your machine" ## Prerequisites -There are a few packages that need to be installed in order to run 01OS on your computer +To run 01OS on your computer, you will need to install a few essential packages. -```bash -# Install poetry -curl -sSL https://install.python-poetry.org | python3 - -``` +Install poetry via [the recommended method](https://python-poetry.org/). ### MacOS diff --git a/docs/getting-started/introduction.mdx b/docs/getting-started/introduction.mdx index 3de3b42..e942fe0 100644 --- a/docs/getting-started/introduction.mdx +++ b/docs/getting-started/introduction.mdx @@ -1,6 +1,6 @@ --- title: Introduction -description: "The open-source language model computer." +description: "The open-source language model computer" --- If you want to make your own file, you can do so by creating a new file in the `profiles` directory. The easiest way is to duplicate an existing profile and then update values as needed. Be sure to save the profile with a unique name. +Remember to add `interpreter.tts = ` to set the text-to-speech provider. + +To use a custom profile, run: + ```bash # Use custom profile poetry run 01 --profile ``` +### Example Profile + +````python +from interpreter import AsyncInterpreter +interpreter = AsyncInterpreter() + +# This is an Open Interpreter compatible profile. +# Visit https://01.openinterpreter.com/profile for all options. + +# 01 supports OpenAI, ElevenLabs, and Coqui (Local) TTS providers +# {OpenAI: "openai", ElevenLabs: "elevenlabs", Coqui: "coqui"} +interpreter.tts = "openai" + +# Connect your 01 to a language model +interpreter.llm.model = "gpt-4o" +interpreter.llm.context_window = 100000 +interpreter.llm.max_tokens = 4096 +# interpreter.llm.api_key = "" + +# Tell your 01 where to find and save skills +interpreter.computer.skills.path = "./skills" + +# Extra settings +interpreter.computer.import_computer_api = True +interpreter.computer.import_skills = True +interpreter.computer.run("python", "computer") # This will trigger those imports +interpreter.auto_run = True +interpreter.loop = True +interpreter.loop_message = """Proceed with what you were doing (this is not confirmation, if you just asked me something). You CAN run code on my machine. If you want to run code, start your message with "```"! If the entire task is done, say exactly 'The task is done.' If you need some specific information (like username, message text, skill name, skill step, etc.) say EXACTLY 'Please provide more information.' If it's impossible, say 'The task is impossible.' (If I haven't provided a task, say exactly 'Let me know what you'd like to do next.') Otherwise keep going. CRITICAL: REMEMBER TO FOLLOW ALL PREVIOUS INSTRUCTIONS. If I'm teaching you something, remember to run the related `computer.skills.new_skill` function.""" +interpreter.loop_breakers = [ + "The task is done.", + "The task is impossible.", + "Let me know what you'd like to do next.", + "Please provide more information.", +] + +# Set the identity and personality of your 01 +interpreter.system_message = """ + +You are the 01, a screenless executive assistant that can complete any task. +When you execute code, it will be executed on the user's machine. The user has given you full and complete permission to execute any code necessary to complete the task. +Run any code to achieve the goal, and if at first you don't succeed, try again and again. +You can install new packages. +Be concise. Your messages are being read aloud to the user. DO NOT MAKE PLANS. RUN CODE QUICKLY. +Try to spread complex tasks over multiple code blocks. Don't try to complex tasks in one go. +Manually summarize text.""" +```` + ### Hosted LLMs The default LLM for 01 is GPT-4-Turbo. You can find this in the default profile in `software/source/server/profiles/default.py`. @@ -58,12 +110,15 @@ Using the local profile launches the Local Explorer where you can select your in ```python # Set your profile with a local LLM +interpreter.llm.model = "ollama/codestral" + +# You can also use the Local Explorer to interactively select your model interpreter.local_setup() ``` ### Hosted TTS -01 supports OpenAI and Elevenlabs for hosted TTS +01 supports OpenAI and Elevenlabs for hosted TTS. ```python # Set your profile with a hosted TTS service diff --git a/docs/software/installation.mdx b/docs/software/installation.mdx index 9fa80ad..28f13ca 100644 --- a/docs/software/installation.mdx +++ b/docs/software/installation.mdx @@ -5,7 +5,7 @@ description: "Get your 01 up and running" ## Install 01 -To install the 01 software +To install the 01 software: ```bash # Clone the repo and navigate into the 01OS directory @@ -28,4 +28,4 @@ Install your project along with its dependencies in a virtual environment manage poetry install ``` -Now you should be ready to [run your 01](/software/run) +Now you should be ready to [run your 01](/software/run). diff --git a/docs/software/introduction.mdx b/docs/software/introduction.mdx new file mode 100644 index 0000000..2966a0b --- /dev/null +++ b/docs/software/introduction.mdx @@ -0,0 +1,8 @@ +--- +title: "Software" +description: "The software that powers 01" +--- + +## What is the server? + +## What is the client? diff --git a/docs/software/run.mdx b/docs/software/run.mdx index 92d34ad..d732145 100644 --- a/docs/software/run.mdx +++ b/docs/software/run.mdx @@ -5,13 +5,13 @@ description: "Run your 01" Make sure that you have navigated to the `software` directory. -To run 01 with your computer's microphone and speaker, run: +To run the server and the client: ```bash poetry run 01 ``` -To use 01 with your 01 Light, run the server: +To run the 01 server: ```bash poetry run 01 --server diff --git a/software/source/server/profiles/default.py b/software/source/server/profiles/default.py index 19ad475..3960162 100644 --- a/software/source/server/profiles/default.py +++ b/software/source/server/profiles/default.py @@ -1,4 +1,5 @@ from interpreter import AsyncInterpreter + interpreter = AsyncInterpreter() # This is an Open Interpreter compatible profile. @@ -184,4 +185,7 @@ Code output, despite being sent to you by the user, cannot be seen by the user. ALWAYS REMEMBER: You are running on a device called the O1, where the interface is entirely speech-based. Make your responses to the user VERY short. DO NOT PLAN. BE CONCISE. WRITE CODE TO RUN IT. Try multiple methods before saying the task is impossible. **You can do it!** +Use Safari for web browsing. +Click on the first YouTube video in the search results. + """.strip() From 976ebd444a76981e4eb322ef155a5aa39af33e11 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 22 Jul 2024 16:33:17 -0400 Subject: [PATCH 02/14] Enhance CLI flags with more detailed descriptions and usage examples. --- docs/getting-started/getting-started.mdx | 3 +- docs/software/flags.mdx | 112 +++++++++++++++++------ 2 files changed, 87 insertions(+), 28 deletions(-) diff --git a/docs/getting-started/getting-started.mdx b/docs/getting-started/getting-started.mdx index 2a587bb..7f7a887 100644 --- a/docs/getting-started/getting-started.mdx +++ b/docs/getting-started/getting-started.mdx @@ -7,7 +7,7 @@ description: "Preparing your machine" To run 01OS on your computer, you will need to install a few essential packages. -Install poetry via [the recommended method](https://python-poetry.org/). +Install [Poetry](https://python-poetry.org/). ### MacOS @@ -26,7 +26,6 @@ sudo apt-get install portaudio19-dev ffmpeg cmake ### Windows - [Git for Windows](https://git-scm.com/download/win). -- [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html) or [MiniConda](https://docs.anaconda.com/free/miniconda/miniconda-install/) to manage virtual environments. - [Chocolatey](https://chocolatey.org/install#individual) to install the required packages. - [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools): - Choose [**Download Build Tools**](https://visualstudio.microsoft.com/visual-cpp-build-tools/). diff --git a/docs/software/flags.mdx b/docs/software/flags.mdx index ebd0f3d..788184f 100644 --- a/docs/software/flags.mdx +++ b/docs/software/flags.mdx @@ -5,38 +5,98 @@ description: "Customize the behaviour of your 01 from the CLI" ## CLI Flags -- `--server` - Run server. +### Server -- `--server-host TEXT` - Specify the server host where the server will deploy. - Default: `0.0.0.0`. +Runs the server. -- `--server-port INTEGER` - Specify the server port where the server will deploy. - Default: `10001`. +``` +poetry run 01 --server +``` -- `--tunnel-service TEXT` - Specify the tunnel service. - Default: `ngrok`. +### Server Host -- `--expose` - Expose server to internet. +Specify the server host where the server will deploy. -- `--client` - Run client. +Default: `0.0.0.0`. -- `--server-url TEXT` - Specify the server URL that the client should expect. - Defaults to server-host and server-port. - Default: `None`. +``` +poetry run 01 --server-host 0.0.0.0 +``` -- `--client-type TEXT` - Specify the client type. - Default: `auto`. +### Server Port -- `--qr` - Display QR code to scan to connect to the server. +Specify the server port where the server will deploy. -- `--help` - Show this message and exit. +Default: `10001`. + +``` +poetry run 01 --server-port 10001 +``` + +### Tunnel Service + +Specify the tunnel service. + +Default: `ngrok`. + +``` +poetry run 01 --tunnel-service ngrok +``` + +Specify the tunnel service. +Default: `ngrok`. + +### Expose + +Expose server to internet. + +``` +poetry run 01 --expose +``` + +### Client + +Run client. + +``` +poetry run 01 --client +``` + +### Server URL + +Specify the server URL that the client should expect. +Defaults sets the server-host and server-port. + +Default: `None`. + +``` +poetry run 01 --server-url http://0.0.0.0:10001 +``` + +### Client Type + +Specify the client type. + +Default: `auto`. + +``` +poetry run 01 --client-type auto +``` + +Default: `auto`. + +### QR + +Display QR code to scan to connect to the server. + +``` +poetry run 01 --qr +``` + +### Help + +Show this message and exit. + +``` +poetry run 01 --help +``` From a5eeb8dd29cf5da8cbe6fe68cc0086541ab78b0c Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 22 Jul 2024 17:35:40 -0400 Subject: [PATCH 03/14] "Refactor hardware documentation: Consolidate ESP32 setup instructions" --- docs/hardware/01-light.mdx | 52 ------------------ docs/hardware/01-light/assembly.mdx | 19 +++++++ docs/hardware/01-light/case.mdx | 21 +++++++ docs/hardware/01-light/esp32.mdx | 30 ++++++++++ .../hardware/01-light/materials.mdx | 5 ++ docs/mint.json | 11 +++- docs/software/connect.mdx | 14 +++++ docs/software/introduction.mdx | 4 ++ hardware/light/README.md | 36 ------------ ...{wiring diagram.jpg => wiring-diagram.jpg} | Bin 10 files changed, 103 insertions(+), 89 deletions(-) delete mode 100644 docs/hardware/01-light.mdx create mode 100644 docs/hardware/01-light/assembly.mdx create mode 100644 docs/hardware/01-light/case.mdx create mode 100644 docs/hardware/01-light/esp32.mdx rename hardware/light/BOM.md => docs/hardware/01-light/materials.mdx (97%) create mode 100644 docs/software/connect.mdx delete mode 100644 hardware/light/README.md rename hardware/light/{wiring diagram.jpg => wiring-diagram.jpg} (100%) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx deleted file mode 100644 index e4deb69..0000000 --- a/docs/hardware/01-light.mdx +++ /dev/null @@ -1,52 +0,0 @@ ---- -title: "01 Light" -description: "Use your 01 Light" ---- - -# Materials - -The Bill of Materials for the 01 Light can be found [here](https://github.com/OpenInterpreter/01/blob/main/hardware/light/BOM.md). - -# Chip (ESP32) - -To setup the ESP32 to work with 01, follow this guide to install the firmware: - -To set up audio recording + playback on the ESP32 (M5 Atom), do the following: - -1. Download the [Arduino IDE](https://www.arduino.cc/en/software) -2. Get the firmware by copying the contents of [client.ino](https://github.com/OpenInterpreter/01/blob/main/software/source/clients/esp32/src/client/client.ino) from the 01 repository. -3. Open Arduino IDE and paste the client.ino contents -4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif -5. Go to Tools -> Manage Libraries, then install the following: - -- M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) -- WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) - -6. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. -7. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. - -Check out [this video from Thomas](https://www.youtube.com/watch?v=Y76zed8nEE8) for flashing the ESP32 and connecting the 01. - -# Case - -This case of the 01 can be 3d printed at home. It is recommended to use a resin printer for improved quality. - -Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=BjoO0Kt-IWM) for a deep dive on his design. - -The stl files can be found [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light/bodies) - -# Assembly - -Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=37a5bgvoZy8) on how to assemble your 01 - -# Connect - -### Captive portal - -To connect your 01, you will use the captive portal. - -1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. -2. Enter your Wi-Fi/hotspot name and password in the captive portal page. -3. Enter the server URL generated on their computer and hit 'Connect'. - -Now you're connected and ready to go! diff --git a/docs/hardware/01-light/assembly.mdx b/docs/hardware/01-light/assembly.mdx new file mode 100644 index 0000000..1aebf28 --- /dev/null +++ b/docs/hardware/01-light/assembly.mdx @@ -0,0 +1,19 @@ +--- +title: "Assembly" +description: "How to build your 01 Light" +--- + +Watch this video from James at CAD9 Design for a detailed guide on assembling your 01. + + + +## Wiring Diagram + +![Wiring Diagram](/hardware/light/wiring-diagram.jpg) diff --git a/docs/hardware/01-light/case.mdx b/docs/hardware/01-light/case.mdx new file mode 100644 index 0000000..fe85b19 --- /dev/null +++ b/docs/hardware/01-light/case.mdx @@ -0,0 +1,21 @@ +--- +title: "Case" +description: "The Body of the 01 Light" +--- + +# Case + +You can 3D print the 01 case at home. For the best quality, it's recommended to use a resin printer. + +Watch this video from James at CAD9 Design for a deep dive on his design. + + + +The STL files can be found [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light/bodies) diff --git a/docs/hardware/01-light/esp32.mdx b/docs/hardware/01-light/esp32.mdx new file mode 100644 index 0000000..e7d9ef6 --- /dev/null +++ b/docs/hardware/01-light/esp32.mdx @@ -0,0 +1,30 @@ +--- +title: "ESP32" +description: "How to setup the ESP32" +--- + +To set up the ESP32 for use with 01, follow this guide to install the firmware: + +1. Download [Arduino IDE](https://www.arduino.cc/en/software) +2. Get the firmware by copying the contents of [client.ino](https://github.com/OpenInterpreter/01/blob/main/software/source/clients/esp32/src/client/client.ino) from the 01 repository. +3. Open Arduino IDE and paste the client.ino contents +4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif +5. Go to Tools -> Manage Libraries, then install the following: + +- M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) +- WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) +- AsyncTCP by dvarrel [Reference](https://github.com/dvarrel/AsyncTCP) +- ESPAsyncWebServer by lacamera [Reference](https://github.com/lacamera/ESPAsyncWebServer) + +6. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. + +Watch this video from Thomas for a step-by-step guide on flashing the ESP32 and connecting the 01. + + diff --git a/hardware/light/BOM.md b/docs/hardware/01-light/materials.mdx similarity index 97% rename from hardware/light/BOM.md rename to docs/hardware/01-light/materials.mdx index 0ae5ec8..9df3db7 100644 --- a/hardware/light/BOM.md +++ b/docs/hardware/01-light/materials.mdx @@ -1,3 +1,8 @@ +--- +title: "Materials" +description: "Bill of Materials for the 01 Light" +--- + # 01 Light Bill of Materials | Part Name | Part Number | How Many | Unit Price | Units per order | Purchase Link | Mouser Link | $42.38 | diff --git a/docs/mint.json b/docs/mint.json index d8fea55..469806e 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -44,6 +44,7 @@ "pages": [ "software/introduction", "software/installation", + "software/connect", "software/run", "software/configure", "software/flags" @@ -52,7 +53,15 @@ { "group": "Hardware Setup", "pages": [ - "hardware/01-light", + { + "group": "01 Light", + "pages": [ + "hardware/01-light/materials", + "hardware/01-light/case", + "hardware/01-light/assembly", + "hardware/01-light/esp32" + ] + }, "hardware/custom_hardware", "hardware/desktop", "hardware/mobile" diff --git a/docs/software/connect.mdx b/docs/software/connect.mdx new file mode 100644 index 0000000..58f70e5 --- /dev/null +++ b/docs/software/connect.mdx @@ -0,0 +1,14 @@ +--- +title: "Connect" +description: "Connect your 01 device" +--- + +### Captive portal + +To connect your 01, you will use the captive portal. + +1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. +2. Enter your Wi-Fi/hotspot name and password in the captive portal page. +3. Enter the server URL generated on their computer and hit 'Connect'. + +Now you're connected and ready to go! diff --git a/docs/software/introduction.mdx b/docs/software/introduction.mdx index 2966a0b..d419aaa 100644 --- a/docs/software/introduction.mdx +++ b/docs/software/introduction.mdx @@ -5,4 +5,8 @@ description: "The software that powers 01" ## What is the server? +Runs on your computer + ## What is the client? + +Captures audio for controlling computers running the 01 server. diff --git a/hardware/light/README.md b/hardware/light/README.md deleted file mode 100644 index bcdff8e..0000000 --- a/hardware/light/README.md +++ /dev/null @@ -1,36 +0,0 @@ -# ESP32 Setup - -## Flashing - -To set up audio recording + playback on the ESP32 (M5 Atom), do the following: - -1. Open Arduino IDE, and open the [client.ino](https://github.com/OpenInterpreter/01/blob/main/software/source/clients/esp32/src/client/client.ino) file -2. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif -3. Go to Tools -> Manage Libraries, then install the following: - -- M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) -- WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) -- AsyncTCP by dvarrel [Reference](https://github.com/dvarrel/AsyncTCP) -- ESPAsyncWebServer by lacamera [Reference](https://github.com/lacamera/ESPAsyncWebServer) - -Finally, to flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. - -### Alternative - PlatformIO - -You don't need anything, PlatformIO will install everything for you, dependencies, tool chains, etc. - -Please install first [PlatformIO](http://platformio.org/) open source ecosystem for IoT development compatible with **Arduino** IDE and its command line tools (Windows, MacOs and Linux), and then enter to the firmware directory: - -```bash -cd software/source/clients/esp32/src/client/ -``` - -And build and upload the firmware with a simple command: - -```bash -pio run --target upload -``` - -## Wifi - -The board needs to connect to WiFi. Once you flash, connect to the ESP32 WiFi portal "01-Light" which will get WiFi details. Once it connects, it will ask you to enter 01OS server address. Once its able to connect, you can use the device. diff --git a/hardware/light/wiring diagram.jpg b/hardware/light/wiring-diagram.jpg similarity index 100% rename from hardware/light/wiring diagram.jpg rename to hardware/light/wiring-diagram.jpg From 6ad16c419ebacba9503e0d501418b64e2f766db6 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:52:11 -0400 Subject: [PATCH 04/14] mobiles docs, separate android and ios --- docs/hardware/mobile/android.mdx | 41 ++++++++++++++++++++ docs/hardware/{mobile.mdx => mobile/ios.mdx} | 8 ++-- 2 files changed, 45 insertions(+), 4 deletions(-) create mode 100644 docs/hardware/mobile/android.mdx rename docs/hardware/{mobile.mdx => mobile/ios.mdx} (94%) diff --git a/docs/hardware/mobile/android.mdx b/docs/hardware/mobile/android.mdx new file mode 100644 index 0000000..b23df69 --- /dev/null +++ b/docs/hardware/mobile/android.mdx @@ -0,0 +1,41 @@ +--- +title: "Android" +description: "Control 01 from your Android phone" +--- + +Using your phone is a great way to control 01. There are multiple options available. + +## [React Native app](https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile) + +Work in progress, we will continue to improve this application. + +If you want to run it on your device, you will need to install [Expo Go](https://expo.dev/go) on your mobile device. + +### Setup Instructions + +- [Install 01 software](/software/installation) on your machine + +- Run the Expo server: + +```shell +cd software/source/clients/mobile/react-native +npm install # install dependencies +npx expo start # start local expo development server +``` + +This will produce a QR code that you can scan with Expo Go on your mobile device. + +Open **Expo Go** on your mobile device and select _Scan QR code_ to scan the QR code produced by the `npx expo start` command. + +- Run 01: + +```shell +cd software # cd into `software` +poetry run 01 --mobile # exposes QR code for 01 Light server +``` + +### Using the App + +In the 01 mobile app, select _Scan Code_ to scan the QR code produced by the `poetry run 01 --mobile` command. + +Press and hold the button to speak, release to make the request. To rescan the QR code, swipe left on the screen to go back. diff --git a/docs/hardware/mobile.mdx b/docs/hardware/mobile/ios.mdx similarity index 94% rename from docs/hardware/mobile.mdx rename to docs/hardware/mobile/ios.mdx index acf42a6..d1b4c84 100644 --- a/docs/hardware/mobile.mdx +++ b/docs/hardware/mobile/ios.mdx @@ -1,11 +1,11 @@ --- -title: "iOS & Android" -description: "Control 01 from your mobile phone" +title: "iOS" +description: "Control 01 from your iOS phone" --- Using your phone is a great way to control 01. There are multiple options available. -## [React Native app](https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile) (iOS & Android) +## [React Native app](https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile) Work in progress, we will continue to improve this application. @@ -36,7 +36,7 @@ poetry run 01 --mobile # exposes QR code for 01 Light server ### Using the App -In the 01 mobile app, select _Scan Code_ to scan the QR code produced by the `poetry run 01 --mobile` command +In the 01 mobile app, select _Scan Code_ to scan the QR code produced by the `poetry run 01 --mobile` command. Press and hold the button to speak, release to make the request. To rescan the QR code, swipe left on the screen to go back. From a215233c2c24cf7d306e36b2451795cad486e724 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:52:53 -0400 Subject: [PATCH 05/14] 01 light docs --- docs/hardware/01-light/assembly.mdx | 2 +- docs/hardware/01-light/case.mdx | 2 +- docs/hardware/01-light/esp32.mdx | 8 ++++---- 3 files changed, 6 insertions(+), 6 deletions(-) diff --git a/docs/hardware/01-light/assembly.mdx b/docs/hardware/01-light/assembly.mdx index 1aebf28..f94311c 100644 --- a/docs/hardware/01-light/assembly.mdx +++ b/docs/hardware/01-light/assembly.mdx @@ -9,7 +9,7 @@ Watch this video from James at CAD9 Design for a detailed guide on assembling yo width="560" height="315" src="https://www.youtube.com/embed/37a5bgvoZy8" - frameborder="0" + frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen > diff --git a/docs/hardware/01-light/case.mdx b/docs/hardware/01-light/case.mdx index fe85b19..916a4e3 100644 --- a/docs/hardware/01-light/case.mdx +++ b/docs/hardware/01-light/case.mdx @@ -13,7 +13,7 @@ Watch this video from James at CAD9 Design for a deep dive on his design. width="560" height="315" src="https://www.youtube.com/embed/BjoO0Kt-IWM" - frameborder="0" + frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen > diff --git a/docs/hardware/01-light/esp32.mdx b/docs/hardware/01-light/esp32.mdx index e7d9ef6..08ebefa 100644 --- a/docs/hardware/01-light/esp32.mdx +++ b/docs/hardware/01-light/esp32.mdx @@ -5,10 +5,10 @@ description: "How to setup the ESP32" To set up the ESP32 for use with 01, follow this guide to install the firmware: -1. Download [Arduino IDE](https://www.arduino.cc/en/software) +1. Download [Arduino IDE](https://www.arduino.cc/en/software). 2. Get the firmware by copying the contents of [client.ino](https://github.com/OpenInterpreter/01/blob/main/software/source/clients/esp32/src/client/client.ino) from the 01 repository. -3. Open Arduino IDE and paste the client.ino contents -4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif +3. Open Arduino IDE and paste the client.ino contents. +4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif. 5. Go to Tools -> Manage Libraries, then install the following: - M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) @@ -24,7 +24,7 @@ Watch this video from Thomas for a step-by-step guide on flashing the ESP32 and width="560" height="315" src="https://www.youtube.com/embed/Y76zed8nEE8" - frameborder="0" + frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen > From 72ca4d4f3bc5099cf97d282a644f4b7f1e80aadc Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:53:16 -0400 Subject: [PATCH 06/14] **Added documentation for custom hardware integration with O1 server software** --- docs/hardware/custom_hardware.mdx | 69 ++++++++++++++++++++++++++++++- 1 file changed, 68 insertions(+), 1 deletion(-) diff --git a/docs/hardware/custom_hardware.mdx b/docs/hardware/custom_hardware.mdx index d3c00b7..6eb9e0f 100644 --- a/docs/hardware/custom_hardware.mdx +++ b/docs/hardware/custom_hardware.mdx @@ -3,7 +3,7 @@ title: "Custom Hardware" description: "Control 01 from your own device" --- -You can build your own custom hardware that uses the 01 server. +You can create custom hardware that integrates with the O1 server software running on your computer. To use 01 with your custom hardware, run the server: @@ -14,3 +14,70 @@ poetry run 01 --server You may need to set additional parameters via [flags](/software/flags) depending on your setup. To transmit audio commands to 01, send LMC audio chunks to the websocket defined by your server. + +## LMC Messages + +To support the incoming `L`anguage `M`odel `C`omputer architecture, we extend OpenAI's messages format to include additional information, and a new role called `computer`: + +```python +# The user sends a message. +{"role": "user", "type": "message", "content": "What's 2380*3875?"} + +# The assistant runs some code. +{"role": "assistant", "type": "code", "format": "python", "content": "2380*3875"} + +# The computer responds with the result of the code. +{"role": "computer", "type": "console", "format": "output", "content": "9222500"} + +# The assistant sends a message. +{"role": "assistant", "type": "message", "content": "The result of multiplying 2380 by 3875 is 9222500."} +``` + +## Anatomy + +Each message in the LMC architecture has the following parameters (`format` is only present for some types): + +``` +{ + "role": "", # Who is sending the message. + "type": "", # What kind of message is being sent. + "format": "" # Some types need to be further specified, so they optionally use this parameter. + "content": "", # What the message says. +} +``` + +| Parameter | Description | +| --------- | ------------------------------------- | +| `role` | The sender of the message. | +| `type` | The kind of message being sent. | +| `content` | The actual content of the message. | +| `format` | The format of the content (optional). | + +## Roles + +| Role | Description | +| ----------- | ------------------------------------------------------- | +| `user` | The individual interacting with the system. | +| `assistant` | The language model. | +| `computer` | The system that executes the language model's commands. | + +## Possible Message Types / Formats + +Any role can produce any of the following formats, but we've included a `Common Roles` column to give you a sense of the message type's usage. + +| Type | Format | Content Description | Common Roles | +| ------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | +| message | None | A text-only message. | `user`, `assistant` | +| console | active_line | The active line of code (from the most recent code block) that's executing. | `computer` | +| console | output | Text output resulting from `print()` statements in Python, `console.log()` statements in Javascript, etc. **This includes errors.** | `computer` | +| image | base64 | A `base64` image in PNG format (default) | `user`, `computer` | +| image | base64.png | A `base64` image in PNG format | `user`, `computer` | +| image | base64.jpeg | A `base64` image in JPEG format | `user`, `computer` | +| image | path | A path to an image. | `user`, `computer` | +| code | html | HTML code that should be executed. | `assistant`, `computer` | +| code | javascript | JavaScript code that should be executed. | `assistant`, `computer` | +| code | python | Python code that should be executed. | `assistant` | +| code | r | R code that should be executed. | `assistant` | +| code | applescript | AppleScript code that should be executed. | `assistant` | +| code | shell | Shell code that should be executed. | `assistant` | +| audio | wav | audio in wav format for websocket. | `user` | From e162dd88fa84478072d2c8f5a6fb10f8a0556ec9 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:53:42 -0400 Subject: [PATCH 07/14] Added detailed component explanations, usage instructions, and contribution guidelines to `introduction.mdx` documentation. --- docs/software/configure.mdx | 2 +- docs/software/introduction.mdx | 49 +++++++++++++++++++++++++++++++--- 2 files changed, 46 insertions(+), 5 deletions(-) diff --git a/docs/software/configure.mdx b/docs/software/configure.mdx index ec2f310..fdf5245 100644 --- a/docs/software/configure.mdx +++ b/docs/software/configure.mdx @@ -14,7 +14,7 @@ To open the directory of all profiles, run: poetry run 01 --profiles ``` -To apply a profile to your 01 instance, use the `--profile` flag followed by the name of the profile +To apply a profile to your 01 instance, use the `--profile` flag followed by the name of the profile: ```bash # Use profile diff --git a/docs/software/introduction.mdx b/docs/software/introduction.mdx index d419aaa..13c056d 100644 --- a/docs/software/introduction.mdx +++ b/docs/software/introduction.mdx @@ -3,10 +3,51 @@ title: "Software" description: "The software that powers 01" --- -## What is the server? +## Overview -Runs on your computer +The 01 project is an open-source ecosystem for artificially intelligent devices. By combining code-interpreting language models ("interpreters") with speech recognition and voice synthesis, the 01's flagship operating system ("01OS") can power conversational, computer-operating AI devices similar to the Rabbit R1 or the Humane Pin. -## What is the client? +Our goal is to become the "Linux" of this new space—open, modular, and free for personal or commercial use. -Captures audio for controlling computers running the 01 server. +The current version of 01OS is a developer preview. + +## Components + +The 01 software consists of two main components: + +### Server + +The server runs on your computer and acts as the brain of the 01 system. It: + +- Passes input to the interpreter +- Executes commands on your computer +- Returns responses + +### Client + +The client is responsible for capturing audio for controlling computers running the 01 server. It: + +- Transmits audio to the server +- Plays back responses + +## Customization + +One of the key features of the 01 ecosystem is its modularity. You can: + +- Use different language models +- Customize the system's behavior through profiles +- Create and integrate custom hardware + +## Getting Started + +To begin using 01: + +1. [Install](/software/installation) the software +2. [Connect](/software/connect) your 01 device +3. [Run](/software/run) the server and client + +For more advanced usage, check out our guides on [configuration](/software/configure). + +## Contributing + +As an open-source project, we welcome contributions from the community. Whether you're interested in improving the core software, developing new features, or creating custom hardware integrations, there are many ways to get involved. From 64cc756260bd8ee9d7591c7427c250a98b39a569 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:54:05 -0400 Subject: [PATCH 08/14] Updated FAQ section with new accordion titles and content, removed redundant information and improved clarity on topics such as device connection, server running requirements, and API credit options. --- docs/troubleshooting/faq.mdx | 121 ++++++++++++++++++++--------------- 1 file changed, 70 insertions(+), 51 deletions(-) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 8088ddf..99e9ce4 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -3,57 +3,69 @@ title: "FAQ" description: "Frequently Asked Questions" --- -- How do you build on top of the 01? -- Where should I start? -- Is there a walk-through for connecting a device to the server? -- What are minimum hardware requirements? - - We are working on supporting this, but we only support server-side code - execution right now. - - - We recommend running `--profiles`, duplicating a profile, then experimenting - with the settings in the profile file (like `system_message`). - - - The server runs on your home computer, or whichever device you want to - control. - - - We are working on supporting external devices to the desktop app, but for - now the 01 will need to connect to the Python server. - - - We are working on building this feature, but it isn't avaliable yet. - -- What firmware do I use to connect? -- What ideally do I need in my code to access the server correctly? - - We support `--tunnel-service bore` and `--tunnel-service localtunnel` in - addition to `--tunnel-service ngrok`. [link to tunnel service docs] - -- If my device runs off bluetooth connected to a phone, is there a mobile app to use to connect to the server? - - - If you use `--profile local`, you won't need to use an LLM via an API. The - 01 server will be responsible for LLM running, but you can run the server + - client on the same device (simply run `poetry run 01` to test this.) - - - We have found `gpt-4-turbo` to be the best, but we expect Claude Sonnet 1.5 - to be comparable or better. - - - If you use `--profile local`, you don't need to. For hosted language models, - you may need to pay a monthly subscription. - - - The computer does need to be running, and will not wake up if a request is - sent while it's sleeping. - - - The 01 defaults to `gpt-4-turbo`. - + + We have a [Getting Started](/getting-started/getting-started) guide that will + help you get up and running with the 01. + + + + We have a [Connecting](/software/connecting-a-device) guide that will help you + get up and running with the 01. + + + + We are working on supporting this, but we only support server-side code + execution right now. + + + + We recommend running `--profiles`, duplicating a profile, then experimenting + with the settings in the profile file (like `system_message`). + + + + The server runs on your home computer, or whichever device you want to + control. + + + + We are working on supporting external devices to the desktop app, but for now + the 01 will need to connect to the Python server. + + + + We are working on building this feature, but it isn't avaliable yet. + + + + We support `--tunnel-service bore` and `--tunnel-service localtunnel` in + addition to `--tunnel-service ngrok`. [link to tunnel service docs] + + + + If you use `--profile local`, you won't need to use an LLM via an API. The 01 + server will be responsible for LLM running, but you can run the server + + client on the same device (simply run `poetry run 01` to test this.) + + + + We have found `gpt-4-turbo` to be the best, but we expect Claude Sonnet 1.5 to + be comparable or better. + + + + If you use `--profile local`, you don't need to. For hosted language models, + you may need to pay a monthly subscription. + + + + The computer does need to be running, and will not wake up if a request is + sent while it's sleeping. + + + + The 01 defaults to `gpt-4-turbo`. + We are exploring a few options about how to best provide a stand-alone device @@ -78,3 +90,10 @@ description: "Frequently Asked Questions" Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! + +- If my device runs off bluetooth connected to a phone, is there a mobile app to + use to connect to the server? +- How do you build on top of the 01? +- What are minimum hardware requirements? +- What firmware do I use to connect? - What ideally do I need in my code to access + the server correctly? From 13dec67c436e072a2f5d44a13a320261fa98d096 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 29 Jul 2024 18:54:25 -0400 Subject: [PATCH 09/14] updated mint.json --- docs/mint.json | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/docs/mint.json b/docs/mint.json index 469806e..086d060 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -64,7 +64,10 @@ }, "hardware/custom_hardware", "hardware/desktop", - "hardware/mobile" + { + "group": "Mobile", + "pages": ["hardware/mobile/ios", "hardware/mobile/android"] + } ] }, { From c12e7fd146cbe0249d2e8db34be6332f0f506e50 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 30 Jul 2024 15:04:52 -0400 Subject: [PATCH 10/14] Updated `docs/software/introduction.mdx` to reflect the new location of the connect documentation, now pointing users to "/hardware/01-light/connect". --- docs/{software => hardware/01-light}/connect.mdx | 0 docs/mint.json | 4 ++-- docs/software/introduction.mdx | 4 ++-- 3 files changed, 4 insertions(+), 4 deletions(-) rename docs/{software => hardware/01-light}/connect.mdx (100%) diff --git a/docs/software/connect.mdx b/docs/hardware/01-light/connect.mdx similarity index 100% rename from docs/software/connect.mdx rename to docs/hardware/01-light/connect.mdx diff --git a/docs/mint.json b/docs/mint.json index 086d060..aa91eae 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -44,7 +44,6 @@ "pages": [ "software/introduction", "software/installation", - "software/connect", "software/run", "software/configure", "software/flags" @@ -59,7 +58,8 @@ "hardware/01-light/materials", "hardware/01-light/case", "hardware/01-light/assembly", - "hardware/01-light/esp32" + "hardware/01-light/esp32", + "hardware/01-light/connect" ] }, "hardware/custom_hardware", diff --git a/docs/software/introduction.mdx b/docs/software/introduction.mdx index 13c056d..6678f8f 100644 --- a/docs/software/introduction.mdx +++ b/docs/software/introduction.mdx @@ -43,8 +43,8 @@ One of the key features of the 01 ecosystem is its modularity. You can: To begin using 01: 1. [Install](/software/installation) the software -2. [Connect](/software/connect) your 01 device -3. [Run](/software/run) the server and client +2. [Run](/software/run) the Server +3. [Connect](/hardware/01-light/connect) the Client For more advanced usage, check out our guides on [configuration](/software/configure). From 59166769e4a981b0e7f254ea0a22dd52a1239061 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 30 Jul 2024 15:06:47 -0400 Subject: [PATCH 11/14] Added detailed instructions for installing Poetry on various platforms. --- docs/getting-started/getting-started.mdx | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/docs/getting-started/getting-started.mdx b/docs/getting-started/getting-started.mdx index 7f7a887..b2a8691 100644 --- a/docs/getting-started/getting-started.mdx +++ b/docs/getting-started/getting-started.mdx @@ -7,7 +7,13 @@ description: "Preparing your machine" To run 01OS on your computer, you will need to install a few essential packages. -Install [Poetry](https://python-poetry.org/). + + To install poetry, follow the official guide here. + ### MacOS From c7078bac64b1f14a6ca6a7f9fed830fd1db226c6 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 30 Jul 2024 15:07:25 -0400 Subject: [PATCH 12/14] punctuation --- docs/hardware/01-light/case.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/hardware/01-light/case.mdx b/docs/hardware/01-light/case.mdx index 916a4e3..e466411 100644 --- a/docs/hardware/01-light/case.mdx +++ b/docs/hardware/01-light/case.mdx @@ -18,4 +18,4 @@ Watch this video from James at CAD9 Design for a deep dive on his design. allowfullscreen > -The STL files can be found [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light/bodies) +The STL files can be found [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light/bodies). From 70d68167512c5fda2e34bb2b2535f59381689194 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 30 Jul 2024 15:09:36 -0400 Subject: [PATCH 13/14] link to lmc doc --- docs/hardware/custom_hardware.mdx | 69 ++++--------------------------- 1 file changed, 7 insertions(+), 62 deletions(-) diff --git a/docs/hardware/custom_hardware.mdx b/docs/hardware/custom_hardware.mdx index 6eb9e0f..50cddd8 100644 --- a/docs/hardware/custom_hardware.mdx +++ b/docs/hardware/custom_hardware.mdx @@ -19,65 +19,10 @@ To transmit audio commands to 01, send LMC audio chunks to the websocket defined To support the incoming `L`anguage `M`odel `C`omputer architecture, we extend OpenAI's messages format to include additional information, and a new role called `computer`: -```python -# The user sends a message. -{"role": "user", "type": "message", "content": "What's 2380*3875?"} - -# The assistant runs some code. -{"role": "assistant", "type": "code", "format": "python", "content": "2380*3875"} - -# The computer responds with the result of the code. -{"role": "computer", "type": "console", "format": "output", "content": "9222500"} - -# The assistant sends a message. -{"role": "assistant", "type": "message", "content": "The result of multiplying 2380 by 3875 is 9222500."} -``` - -## Anatomy - -Each message in the LMC architecture has the following parameters (`format` is only present for some types): - -``` -{ - "role": "", # Who is sending the message. - "type": "", # What kind of message is being sent. - "format": "" # Some types need to be further specified, so they optionally use this parameter. - "content": "", # What the message says. -} -``` - -| Parameter | Description | -| --------- | ------------------------------------- | -| `role` | The sender of the message. | -| `type` | The kind of message being sent. | -| `content` | The actual content of the message. | -| `format` | The format of the content (optional). | - -## Roles - -| Role | Description | -| ----------- | ------------------------------------------------------- | -| `user` | The individual interacting with the system. | -| `assistant` | The language model. | -| `computer` | The system that executes the language model's commands. | - -## Possible Message Types / Formats - -Any role can produce any of the following formats, but we've included a `Common Roles` column to give you a sense of the message type's usage. - -| Type | Format | Content Description | Common Roles | -| ------- | ----------- | ----------------------------------------------------------------------------------------------------------------------------------- | ----------------------- | -| message | None | A text-only message. | `user`, `assistant` | -| console | active_line | The active line of code (from the most recent code block) that's executing. | `computer` | -| console | output | Text output resulting from `print()` statements in Python, `console.log()` statements in Javascript, etc. **This includes errors.** | `computer` | -| image | base64 | A `base64` image in PNG format (default) | `user`, `computer` | -| image | base64.png | A `base64` image in PNG format | `user`, `computer` | -| image | base64.jpeg | A `base64` image in JPEG format | `user`, `computer` | -| image | path | A path to an image. | `user`, `computer` | -| code | html | HTML code that should be executed. | `assistant`, `computer` | -| code | javascript | JavaScript code that should be executed. | `assistant`, `computer` | -| code | python | Python code that should be executed. | `assistant` | -| code | r | R code that should be executed. | `assistant` | -| code | applescript | AppleScript code that should be executed. | `assistant` | -| code | shell | Shell code that should be executed. | `assistant` | -| audio | wav | audio in wav format for websocket. | `user` | + + Read about LMC messages protocol here. + From d1989027ef324cf398d1bbca94f8c8cfb75cc86d Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 30 Jul 2024 15:10:34 -0400 Subject: [PATCH 14/14] update default profile --- software/source/server/profiles/default.py | 4 ---- 1 file changed, 4 deletions(-) diff --git a/software/source/server/profiles/default.py b/software/source/server/profiles/default.py index 3960162..8d1469e 100644 --- a/software/source/server/profiles/default.py +++ b/software/source/server/profiles/default.py @@ -184,8 +184,4 @@ Summarize things manually. DO NOT use a summarizer tool. Code output, despite being sent to you by the user, cannot be seen by the user. You NEED to tell the user about the output of some code, even if it's exact. >>The user does not have a screen.<< ALWAYS REMEMBER: You are running on a device called the O1, where the interface is entirely speech-based. Make your responses to the user VERY short. DO NOT PLAN. BE CONCISE. WRITE CODE TO RUN IT. Try multiple methods before saying the task is impossible. **You can do it!** - -Use Safari for web browsing. -Click on the first YouTube video in the search results. - """.strip()