From 7bafa0db2835c5cf043c08fdfe9437c931b7146c Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Mon, 8 Jul 2024 14:39:38 -0400 Subject: [PATCH 01/30] remove references to 01 heavy --- README.md | 4 +--- TASKS.md | 4 ++-- docs/README_CN.md | 1 - docs/README_FR.md | 5 ++--- docs/README_JA.md | 3 +-- docs/bodies/01-heavy.mdx | 6 ------ docs/mint.json | 2 +- 7 files changed, 7 insertions(+), 18 deletions(-) delete mode 100644 docs/bodies/01-heavy.mdx diff --git a/README.md b/README.md index ebd8488..c65dba3 100644 --- a/README.md +++ b/README.md @@ -10,11 +10,10 @@
- | [日本語](docs/README_JP.md) | [English](README.md) | +| [日本語](docs/README_JP.md) | [English](README.md) |
-
![OI-O1-BannerDemo-2](https://www.openinterpreter.com/OI-O1-BannerDemo-3.jpg) @@ -65,7 +64,6 @@ poetry run 01 # Runs the 01 Light simulator (hold your spacebar, speak, release) - The **01 Light** is an ESP32-based voice interface. Build instructions are [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light). A list of what to buy [here](https://github.com/OpenInterpreter/01/blob/main/hardware/light/BOM.md). - It works in tandem with the **01 Server** ([setup guide below](https://github.com/OpenInterpreter/01/blob/main/README.md#01-server)) running on your home computer. - **Mac OSX** and **Ubuntu** are supported by running `poetry run 01` (**Windows** is supported experimentally). This uses your spacebar to simulate the 01 Light. -- (coming soon) The **01 Heavy** is a standalone device that runs everything locally. **We need your help supporting & building more hardware.** The 01 should be able to run on any device with input (microphone, keyboard, etc.), output (speakers, screens, motors, etc.), and an internet connection (or sufficient compute to run everything locally). [Contribution Guide →](https://github.com/OpenInterpreter/01/blob/main/CONTRIBUTING.md) diff --git a/TASKS.md b/TASKS.md index 2477fb2..946e214 100644 --- a/TASKS.md +++ b/TASKS.md @@ -5,7 +5,7 @@ - [ ] Test it end-to-end. - [ ] Make sure it works with computer.skills.search (it should already work) - [ ] Create computer.skills.teach() - - [ ] Displays a tkinter message asking users to complete the task via text (eventually voice) in the most generalizable way possible. OI should use computer.mouse and computer.keyboard to fulfill each step, then save the generalized instruction as a skill. Clicking the mouse cancels teach mode. When OI invokes this skill in the future, it will just list those steps (it needs to figure out how to flexibly accomplish each step). + - [ ] Displays a tkinter message asking users to complete the task via text (eventually voice) in the most generalizable way possible. OI should use computer.mouse and computer.keyboard to fulfill each step, then save the generalized instruction as a skill. Clicking the mouse cancels teach mode. When OI invokes this skill in the future, it will just list those steps (it needs to figure out how to flexibly accomplish each step). - [ ] Computer: "What do you want to name this skill?" - [ ] User: Enters name in textbox - [ ] Computer: "Whats the First Step" @@ -73,7 +73,6 @@ - [ ] Connect the Jetson or Pi to a battery. - [ ] Make a rudimentary case for the Seeed Sense + speaker. Optional. - [ ] Make a rudimentary case for the Jetson or Pi. Optional. -- [ ] Determine recommended minimal hardware for the light & heavy. **Release Day** @@ -81,6 +80,7 @@ - [ ] Create form to get pre-release feedback from 200 interested people (who responded to Killian's tweet) **DONE** + - [ ] Get Local TTS working on Mac [Shiven] - [ ] Get Local SST working on Mac [Zohaib + Shiven] - [ ] Debug level logging/printing [Tom] diff --git a/docs/README_CN.md b/docs/README_CN.md index e77d2b3..8e7e53e 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -53,7 +53,6 @@ poetry run 01 # Runs the 01 Light simulator (hold your spacebar, speak, release) - **01 Light** 是基于 ESP32 的语音接口。 [构建说明在这里。](https://github.com/OpenInterpreter/01/tree/main/hardware/light) 它与运行在你家庭电脑上的 **01 Server** ([下面有设置指南](https://github.com/OpenInterpreter/01/blob/main/README.md#01-server)) 配合使用。 - **Mac OSX** and **Ubuntu** 支持通过运行 `poetry run 01`。 这会使用你的空格键来模拟 01 Light。 -- （即将推出） **01 Heavy** 是一个独立设备，可以在本地运行所有功能。 **我们需要您的帮助来支持和构建更多硬件。** 01 应该能够在任何具有输入（麦克风、键盘等）、输出（扬声器、屏幕、电机等）和互联网连接（或足够的计算资源以在本地运行所有内容）的设备上运行。 [ 贡献指南 →](https://github.com/OpenInterpreter/01/blob/main/CONTRIBUTING.md) diff --git a/docs/README_FR.md b/docs/README_FR.md index 26297eb..cdb0025 100644 --- a/docs/README_FR.md +++ b/docs/README_FR.md @@ -57,7 +57,6 @@ poetry run 01 # Exécute le simulateur 01 Light (maintenez votre barre d'espace, - Le **01 Light** est une interface vocale basée sur ESP32. Les instructions de construction sont [ici]. (https://github.com/OpenInterpreter/01/tree/main/hardware/light). Une liste de ce qu'il faut acheter se trouve [ici](https://github.com/OpenInterpreter/01/blob/main/hardware/light/BOM.md). - Il fonctionne en tandem avec le **Server 01** ([guide d'installation ci-dessous](https://github.com/OpenInterpreter/01/blob/main/README.md#01-server)) fonctionnant sur votre ordinateur. - **Mac OSX** et **Ubuntu** sont pris en charge en exécutant `poetry run 01` (**Windows** est pris en charge de manière expérimentale). Cela utilise votre barre d'espace pour simuler le 01 Light. -- (prochainement) Le **01 Heavy** est un dispositif autonome qui exécute tout localement. **Nous avons besoin de votre aide pour soutenir et construire plus de hardware.** Le 01 devrait pouvoir fonctionner sur tout dispositif avec entrée (microphone, clavier, etc.), sortie (haut-parleurs, écrans, moteurs, etc.) et connexion internet (ou suffisamment de puissance de calcul pour tout exécuter localement). [Guide de Contribution →](https://github.com/OpenInterpreter/01/blob/main/CONTRIBUTING.md) @@ -65,7 +64,7 @@ poetry run 01 # Exécute le simulateur 01 Light (maintenez votre barre d'espace, # Comment ça marche ? -Le 01 expose un websocket de *speech-to-speech* à l'adresse `localhost:10001`. +Le 01 expose un websocket de _speech-to-speech_ à l'adresse `localhost:10001`. Si vous diffusez des octets audio bruts vers `/` au [format de streaming LMC](https://docs.openinterpreter.com/guides/streaming-response), vous recevrez sa réponse dans le même format. @@ -81,7 +80,7 @@ Le 01 l'encapsule dans une interface vocale : ## Messages LMC -Pour communiquer avec les différents composants du système, nous introduisons le [format de messages LMC](https://docs.openinterpreter.com/protocols/lmc-messages), une extension du format de message d'OpenAI qui inclut un nouveau rôle "*computer*": +Pour communiquer avec les différents composants du système, nous introduisons le [format de messages LMC](https://docs.openinterpreter.com/protocols/lmc-messages), une extension du format de message d'OpenAI qui inclut un nouveau rôle "_computer_": https://github.com/OpenInterpreter/01/assets/63927363/8621b075-e052-46ba-8d2e-d64b9f2a5da9 diff --git a/docs/README_JA.md b/docs/README_JA.md index 868dae8..d8eeb7c 100644 --- a/docs/README_JA.md +++ b/docs/README_JA.md @@ -12,7 +12,7 @@ ![OI-O1-BannerDemo-2](https://www.openinterpreter.com/OI-O1-BannerDemo-3.jpg) -あなたのビルドをサポートします。[1対1のサポートを申し込む。](https://0ggfznkwh4j.typeform.com/to/kkStE8WF) +あなたのビルドをサポートします。[1 対 1 のサポートを申し込む。](https://0ggfznkwh4j.typeform.com/to/kkStE8WF)
@@ -56,7 +56,6 @@ poetry run 01 # 01 Light シミュレーターを作動させる（スペース - **01 Light** は ESP32 ベースの音声インターフェースです。ビルド手順は[こちら](https://github.com/OpenInterpreter/01/tree/main/hardware/light)。買うべきもののリストは[こちら](https://github.com/OpenInterpreter/01/blob/main/hardware/light/BOM.md)。 - ご自宅のコンピューターで動作している **01 サーバー**（[下記のセットアップガイド](https://github.com/OpenInterpreter/01/blob/main/README.md#01-server)）と連動して動作します。 - **Mac OSX** と **Ubuntu** は `poetry run 01` を実行することでサポートされます（**Windows** は実験的にサポートされている）。これはスペースキーを使って 01 Light をシミュレートします。 -- (近日発表) **01 Heavy** は、ローカルですべてを実行するスタンドアローンデバイスです。 **より多くのハードウェアをサポートし、構築するためには、皆さんの協力が必要です。** 01 は、入力（マイク、キーボードなど）、出力（スピーカー、スクリーン、モーターなど）、インターネット接続（またはローカルですべてを実行するのに十分な計算能力）があれば、どのようなデバイスでも実行できるはずです。[コントリビューションガイド →](https://github.com/OpenInterpreter/01/blob/main/CONTRIBUTING.md) diff --git a/docs/bodies/01-heavy.mdx b/docs/bodies/01-heavy.mdx deleted file mode 100644 index cac5f1f..0000000 --- a/docs/bodies/01-heavy.mdx +++ /dev/null @@ -1,6 +0,0 @@ ---- -title: "01 Heavy" -description: "Build your 01 Heavy" ---- - -runs fully locally + coming soon diff --git a/docs/mint.json b/docs/mint.json index 1ac5338..3c7cd36 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -54,7 +54,7 @@ }, { "group": "Bodies", - "pages": ["bodies/01-light", "bodies/01-heavy"] + "pages": ["bodies/01-light"] }, { "group": "Legal", From ca28f165b5262b96280d88049079a145f569235e Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:02:02 -0400 Subject: [PATCH 02/30] update docs structure --- docs/bodies/01-light.mdx | 12 --- docs/getting-started/getting-started.mdx | 39 ++++++++ docs/getting-started/introduction.mdx | 30 +----- docs/getting-started/setup.mdx | 93 ------------------- docs/{services => guides}/language-model.mdx | 0 docs/{services => guides}/speech-to-text.mdx | 0 docs/{services => guides}/text-to-speech.mdx | 0 docs/hardware/01-light.mdx | 22 +++++ docs/mint.json | 33 ++++--- .../{client/setup.mdx => software/client.mdx} | 0 docs/software/installation.mdx | 41 ++++++++ docs/{server => software}/setup.mdx | 0 docs/troubleshooting/faq.mdx | 4 + 13 files changed, 126 insertions(+), 148 deletions(-) delete mode 100644 docs/bodies/01-light.mdx create mode 100644 docs/getting-started/getting-started.mdx delete mode 100644 docs/getting-started/setup.mdx rename docs/{services => guides}/language-model.mdx (100%) rename docs/{services => guides}/speech-to-text.mdx (100%) rename docs/{services => guides}/text-to-speech.mdx (100%) create mode 100644 docs/hardware/01-light.mdx rename docs/{client/setup.mdx => software/client.mdx} (100%) create mode 100644 docs/software/installation.mdx rename docs/{server => software}/setup.mdx (100%) create mode 100644 docs/troubleshooting/faq.mdx diff --git a/docs/bodies/01-light.mdx b/docs/bodies/01-light.mdx deleted file mode 100644 index 4bf8f3a..0000000 --- a/docs/bodies/01-light.mdx +++ /dev/null @@ -1,12 +0,0 @@ ---- -title: "01 Light" -description: "Build your 01 Light" ---- - -## ESP32 client - -Instructions to set up your ESP32 client can be found here - -## Suppliementary files - -For CAD files, wiring diagram, and images, please visit the [01 Light hardware repository](https://github.com/OpenInterpreter/01/tree/main/hardware/light). diff --git a/docs/getting-started/getting-started.mdx b/docs/getting-started/getting-started.mdx new file mode 100644 index 0000000..769feae --- /dev/null +++ b/docs/getting-started/getting-started.mdx @@ -0,0 +1,39 @@ +--- +title: "Getting Started" +description: "Preparing your machine" +--- + +## Prerequisites + +There are a few packages that need to be installed in order to run 01OS on your computer + +```bash +# Install poetry +curl -sSL https://install.python-poetry.org | python3 - +``` + +```bash +# MacOS +brew install portaudio ffmpeg cmake + +# Ubuntu (wayland not supported, only ubuntu 20.04 and below) +sudo apt-get install portaudio19-dev ffmpeg cmake + +# Windows +On Windows you will need to install the following: +``` + +- [Git for Windows](https://git-scm.com/download/win). +- [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html) or [MiniConda](https://docs.anaconda.com/free/miniconda/miniconda-install/) to manage virtual environments. +- [Chocolatey](https://chocolatey.org/install#individual) to install the required packages. +- [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools): + - Choose [**Download Build Tools**](https://visualstudio.microsoft.com/visual-cpp-build-tools/). + - Run the downloaded file **vs_BuildTools.exe**. + - In the installer, select **Workloads** > **Desktop & Mobile** > **Desktop Development with C++**. + +With these installed, you can run the following commands in a **PowerShell terminal as an administrator**: + +```powershell +# Install the required packages +choco install -y ffmpeg +``` diff --git a/docs/getting-started/introduction.mdx b/docs/getting-started/introduction.mdx index 1b14e3a..69b5d22 100644 --- a/docs/getting-started/introduction.mdx +++ b/docs/getting-started/introduction.mdx @@ -1,12 +1,12 @@ --- title: Introduction -description: 'The open-source language model computer.' +description: "The open-source language model computer." --- thumbnail The 01 project is an open-source ecosystem for artificially intelligent devices. @@ -15,30 +15,4 @@ By combining code-interpreting language models ("interpreters") with speech reco We intend to become the “Linux” of this new space— open, modular, and free for personal or commercial use. -## Quick Start - -### Install dependencies - -```bash -# MacOS -brew install portaudio ffmpeg cmake - -# Ubuntu -sudo apt-get install portaudio19-dev ffmpeg cmake -``` - -For windows, please refer to the [setup guide](/getting-started/setup#windows). - -### Install and run the 01 CLI - -```bash -# Clone the repo and navigate into the software directory -git clone https://github.com/OpenInterpreter/01.git -cd software - -# Install dependencies and run 01 -poetry install -poetry run 01 -``` - _Disclaimer:_ The current version of 01OS is a developer preview diff --git a/docs/getting-started/setup.mdx b/docs/getting-started/setup.mdx deleted file mode 100644 index ae2aa1c..0000000 --- a/docs/getting-started/setup.mdx +++ /dev/null @@ -1,93 +0,0 @@ ---- -title: 'Setup' -description: 'Get your 01 up and running' ---- - -## Captive portal - -To connect your 01, you will use the captive portal. - -1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. -2. Enter your Wi-Fi/hotspot name and password in the captive portal page. -3. Enter the server URL generated on their computer and hit 'Connect'. - -Now you're connected and ready to go! - -# Local 01OS - -## Prerequisites - -There are a few packages that need to be installed in order to run 01OS on your computer - -```bash -# MacOS -brew install portaudio ffmpeg cmake - -# Ubuntu (wayland not supported, only ubuntu 20.04 and below) -sudo apt-get install portaudio19-dev ffmpeg cmake - -# Install poetry -curl -sSL https://install.python-poetry.org | python3 - -``` - -#### Windows - -On Windows you will need to install the following: - -- [Git for Windows](https://git-scm.com/download/win). -- [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html) or [MiniConda](https://docs.anaconda.com/free/miniconda/miniconda-install/) to manage virtual environments. -- [Chocolatey](https://chocolatey.org/install#individual) to install the required packages. -- [Microsoft C++ Build Tools](https://visualstudio.microsoft.com/visual-cpp-build-tools): - - Choose [**Download Build Tools**](https://visualstudio.microsoft.com/visual-cpp-build-tools/). - - Run the downloaded file **vs_BuildTools.exe**. - - In the installer, select **Workloads** > **Desktop & Mobile** > **Desktop Development with C++**. - -With these installed, you can run the following commands in a **PowerShell terminal as an administrator**: - -```powershell -# Install the required packages -choco install -y ffmpeg -``` - -## Install 01 - -To install the 01 CLI - -```bash -# Clone the repo and navigate into the 01OS directory -git clone https://github.com/OpenInterpreter/01.git -``` - -## Run the 01 - -In order to run 01 on your computer, use [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer). - -Navigate to the project's software directory: - -```bash -cd software -``` - -Install your project along with its dependencies in a virtual environment managed by Poetry. - -```bash -poetry install -``` - -Run your local version of 01 with: - -```bash -poetry run 01 -``` - -## Swap out service providers - -You have the ability to set your LLM, STT, and TTS service providers - -## Server setup - -You are able to run just the server. - -## Client setup - -You are able to run just the client. diff --git a/docs/services/language-model.mdx b/docs/guides/language-model.mdx similarity index 100% rename from docs/services/language-model.mdx rename to docs/guides/language-model.mdx diff --git a/docs/services/speech-to-text.mdx b/docs/guides/speech-to-text.mdx similarity index 100% rename from docs/services/speech-to-text.mdx rename to docs/guides/speech-to-text.mdx diff --git a/docs/services/text-to-speech.mdx b/docs/guides/text-to-speech.mdx similarity index 100% rename from docs/services/text-to-speech.mdx rename to docs/guides/text-to-speech.mdx diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx new file mode 100644 index 0000000..76608bf --- /dev/null +++ b/docs/hardware/01-light.mdx @@ -0,0 +1,22 @@ +--- +title: "01 Light" +description: "Use your 01 Light" +--- + +## Captive portal + +To connect your 01, you will use the captive portal. + +1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. +2. Enter your Wi-Fi/hotspot name and password in the captive portal page. +3. Enter the server URL generated on their computer and hit 'Connect'. + +Now you're connected and ready to go! + +## ESP32 client + +Instructions to set up your ESP32 client can be found here + +## Suppliementary files + +For CAD files, wiring diagram, and images, please visit the [01 Light hardware repository](https://github.com/OpenInterpreter/01/tree/main/hardware/light). diff --git a/docs/mint.json b/docs/mint.json index 3c7cd36..efbf9e2 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -34,27 +34,30 @@ "navigation": [ { "group": "Getting Started", - "pages": ["getting-started/introduction", "getting-started/setup"] + "pages": [ + "getting-started/introduction", + "getting-started/getting-started" + ] }, { - "group": "Server", - "pages": ["server/setup"] + "group": "Software Setup", + "pages": ["software/installation", "software/setup", "software/client"] }, { - "group": "Services", - "pages": [ - "services/language-model", - "services/speech-to-text", - "services/text-to-speech" - ] + "group": "Hardware Setup", + "pages": ["hardware/01-light"] }, { - "group": "Client", - "pages": ["client/setup"] + "group": "Using 01", + "pages": [ + "guides/language-model", + "guides/speech-to-text", + "guides/text-to-speech" + ] }, { - "group": "Bodies", - "pages": ["bodies/01-light"] + "group": "Troubleshooting", + "pages": ["troubleshooting/faq"] }, { "group": "Legal", @@ -66,7 +69,7 @@ }, "footerSocials": { "twitter": "https://x.com/OpenInterpreter", - "github": "https://github.com/KillianLucas/01", - "discord": "https://discord.gg/E2XTbkj4JF" + "github": "https://github.com/OpenInterpreter/01", + "discord": "https://discord.com/invite/Hvz9Axh84z" } } diff --git a/docs/client/setup.mdx b/docs/software/client.mdx similarity index 100% rename from docs/client/setup.mdx rename to docs/software/client.mdx diff --git a/docs/software/installation.mdx b/docs/software/installation.mdx new file mode 100644 index 0000000..1d54c78 --- /dev/null +++ b/docs/software/installation.mdx @@ -0,0 +1,41 @@ +--- +title: "Installation" +description: "Get your 01 up and running" +--- + +## Install 01 + +To install the 01 software + +```bash +# Clone the repo and navigate into the 01OS directory +git clone https://github.com/OpenInterpreter/01.git +``` + +## Run the 01 + +In order to run 01 on your computer, use [Poetry](https://python-poetry.org/docs/#installing-with-the-official-installer). + +Navigate to the project's software directory: + +```bash +cd software +``` + +Install your project along with its dependencies in a virtual environment managed by Poetry. + +```bash +poetry install +``` + +Run your local version of 01 with: + +```bash +poetry run 01 +``` + +Use 01 with your 01 Light + +```bash +poetry run 01 --server +``` diff --git a/docs/server/setup.mdx b/docs/software/setup.mdx similarity index 100% rename from docs/server/setup.mdx rename to docs/software/setup.mdx diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx new file mode 100644 index 0000000..91122a6 --- /dev/null +++ b/docs/troubleshooting/faq.mdx @@ -0,0 +1,4 @@ +--- +title: "FAQ" +description: "Frequently Asked Questions" +--- From a5d345a55d5ef72fbe467c29dbab1cd2ab907684 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:05:47 -0400 Subject: [PATCH 03/30] m5 atom page --- .../client.mdx => hardware/m5atom.mdx} | 28 ++----------------- docs/mint.json | 4 +-- 2 files changed, 4 insertions(+), 28 deletions(-) rename docs/{software/client.mdx => hardware/m5atom.mdx} (76%) diff --git a/docs/software/client.mdx b/docs/hardware/m5atom.mdx similarity index 76% rename from docs/software/client.mdx rename to docs/hardware/m5atom.mdx index df1a628..84e9155 100644 --- a/docs/software/client.mdx +++ b/docs/hardware/m5atom.mdx @@ -1,6 +1,6 @@ --- -title: "Setup" -description: "Get your 01 client up and running" +title: "M5 Atom" +description: "Prep your M5 Atom" --- ## ESP32 Playback @@ -16,27 +16,3 @@ To set up audio recording + playback on the ESP32 (M5 Atom), do the following: 4. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. 5. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. - -## Desktop - -### Server with a client - -```bash -# run 01 with no args -poetry run 01 -``` - -### Client only - -```bash -poetry run 01 --client -``` - -### Flags - -- `--client` - Run client. - -- `--client-type TEXT` - Specify the client type. - Default: `auto`. diff --git a/docs/mint.json b/docs/mint.json index efbf9e2..09b4f76 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -41,11 +41,11 @@ }, { "group": "Software Setup", - "pages": ["software/installation", "software/setup", "software/client"] + "pages": ["software/installation", "software/setup"] }, { "group": "Hardware Setup", - "pages": ["hardware/01-light"] + "pages": ["hardware/01-light", "hardware/m5atom"] }, { "group": "Using 01", From e92555898c93c2995add2b2cd37096595c0b0eb1 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:11:43 -0400 Subject: [PATCH 04/30] better m5 atom instructions --- docs/hardware/m5atom.mdx | 18 ++++++++++++------ 1 file changed, 12 insertions(+), 6 deletions(-) diff --git a/docs/hardware/m5atom.mdx b/docs/hardware/m5atom.mdx index 84e9155..e7d34fa 100644 --- a/docs/hardware/m5atom.mdx +++ b/docs/hardware/m5atom.mdx @@ -3,16 +3,22 @@ title: "M5 Atom" description: "Prep your M5 Atom" --- -## ESP32 Playback +The recommended hardware for the developer kit is based around the M5 Atom. It contains an ESP32 board, as well as a microphone and speaker. + +The ESP32 needs to be setup to work with the 01OS server. We've created the firmware needed, just follow this guide. + +## ESP32 To set up audio recording + playback on the ESP32 (M5 Atom), do the following: -1. Open Arduino IDE, and open the client/client.ino file -2. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif -3. Go to Tools -> Manage Libraries, then install the following: +1. Download the [Arduino IDE](https://www.arduino.cc/en/software) +2. Get the firmware by copying the contents of `software/source/clients/esp32/src/client/client.ino` from the 01 repository. +3. Open Arduino IDE and paste the client.ino contents +4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif +5. Go to Tools -> Manage Libraries, then install the following: - M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) - WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) -4. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. -5. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. +6. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. +7. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. From 9ba89177d22f614aaf5355a3e91732723bc44585 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:13:53 -0400 Subject: [PATCH 05/30] better 01 light instructions --- docs/hardware/01-light.mdx | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index 76608bf..98ee592 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -3,6 +3,8 @@ title: "01 Light" description: "Use your 01 Light" --- +If you ordered your 01 Light and it was pre-assembled, you just need to connect it to your computer using the Captive Portal. If you're assembling it yourself, be sure to follow the M5 Atom instructions. + ## Captive portal To connect your 01, you will use the captive portal. @@ -13,9 +15,9 @@ To connect your 01, you will use the captive portal. Now you're connected and ready to go! -## ESP32 client +## M5 Atom Prep -Instructions to set up your ESP32 client can be found here +Instructions to set up your M5 Atom can be found here ## Suppliementary files From 8d10679d5eb961b3f5c47b21034c80e3df187e60 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:17:04 -0400 Subject: [PATCH 06/30] add page for flags --- docs/mint.json | 2 +- docs/software/flags.mdx | 77 +++++++++++++++++++++++++++++++++++++++++ docs/software/setup.mdx | 73 -------------------------------------- 3 files changed, 78 insertions(+), 74 deletions(-) create mode 100644 docs/software/flags.mdx diff --git a/docs/mint.json b/docs/mint.json index 09b4f76..d2d6edc 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -41,7 +41,7 @@ }, { "group": "Software Setup", - "pages": ["software/installation", "software/setup"] + "pages": ["software/installation", "software/setup", "software/flags"] }, { "group": "Hardware Setup", diff --git a/docs/software/flags.mdx b/docs/software/flags.mdx new file mode 100644 index 0000000..acfd369 --- /dev/null +++ b/docs/software/flags.mdx @@ -0,0 +1,77 @@ +--- +title: "Flags" +description: "Customize the behaviour of your 01" +--- + +## CLI Flags + +- `--server` + Run server. + +- `--server-host TEXT` + Specify the server host where the server will deploy. + Default: `0.0.0.0`. + +- `--server-port INTEGER` + Specify the server port where the server will deploy. + Default: `10001`. + +- `--tunnel-service TEXT` + Specify the tunnel service. + Default: `ngrok`. + +- `--expose` + Expose server to internet. + +- `--server-url TEXT` + Specify the server URL that the client should expect. + Defaults to server-host and server-port. + Default: `None`. + +- `--llm-service TEXT` + Specify the LLM service. + Default: `litellm`. + +- `--model TEXT` + Specify the model. + Default: `gpt-4`. + +- `--llm-supports-vision` + Specify if the LLM service supports vision. + +- `--llm-supports-functions` + Specify if the LLM service supports functions. + +- `--context-window INTEGER` + Specify the context window size. + Default: `2048`. + +- `--max-tokens INTEGER` + Specify the maximum number of tokens. + Default: `4096`. + +- `--temperature FLOAT` + Specify the temperature for generation. + Default: `0.8`. + +- `--tts-service TEXT` + Specify the TTS service. + Default: `openai`. + +- `--stt-service TEXT` + Specify the STT service. + Default: `openai`. + +- `--local` + Use recommended local services for LLM, STT, and TTS. + +- `--install-completion [bash|zsh|fish|powershell|pwsh]` + Install completion for the specified shell. + Default: `None`. + +- `--show-completion [bash|zsh|fish|powershell|pwsh]` + Show completion for the specified shell, to copy it or customize the installation. + Default: `None`. + +- `--help` + Show this message and exit. diff --git a/docs/software/setup.mdx b/docs/software/setup.mdx index f500687..31f7665 100644 --- a/docs/software/setup.mdx +++ b/docs/software/setup.mdx @@ -41,76 +41,3 @@ You are able to pass CLI flags `--tts-service` and/or `--stt-service` with the d These different service providers can be found in `/services/stt` and `/services/tts` For more information, please read about speech-to-text and text-to-speech - -## CLI Flags - -- `--server` - Run server. - -- `--server-host TEXT` - Specify the server host where the server will deploy. - Default: `0.0.0.0`. - -- `--server-port INTEGER` - Specify the server port where the server will deploy. - Default: `10001`. - -- `--tunnel-service TEXT` - Specify the tunnel service. - Default: `ngrok`. - -- `--expose` - Expose server to internet. - -- `--server-url TEXT` - Specify the server URL that the client should expect. - Defaults to server-host and server-port. - Default: `None`. - -- `--llm-service TEXT` - Specify the LLM service. - Default: `litellm`. - -- `--model TEXT` - Specify the model. - Default: `gpt-4`. - -- `--llm-supports-vision` - Specify if the LLM service supports vision. - -- `--llm-supports-functions` - Specify if the LLM service supports functions. - -- `--context-window INTEGER` - Specify the context window size. - Default: `2048`. - -- `--max-tokens INTEGER` - Specify the maximum number of tokens. - Default: `4096`. - -- `--temperature FLOAT` - Specify the temperature for generation. - Default: `0.8`. - -- `--tts-service TEXT` - Specify the TTS service. - Default: `openai`. - -- `--stt-service TEXT` - Specify the STT service. - Default: `openai`. - -- `--local` - Use recommended local services for LLM, STT, and TTS. - -- `--install-completion [bash|zsh|fish|powershell|pwsh]` - Install completion for the specified shell. - Default: `None`. - -- `--show-completion [bash|zsh|fish|powershell|pwsh]` - Show completion for the specified shell, to copy it or customize the installation. - Default: `None`. - -- `--help` - Show this message and exit. From 251b5b7adbd3d260bf29f40fefd98357d39ec920 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 14:19:25 -0400 Subject: [PATCH 07/30] reorg OS pre-reqs --- docs/getting-started/getting-started.mdx | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-) diff --git a/docs/getting-started/getting-started.mdx b/docs/getting-started/getting-started.mdx index 769feae..4812e9f 100644 --- a/docs/getting-started/getting-started.mdx +++ b/docs/getting-started/getting-started.mdx @@ -12,17 +12,22 @@ There are a few packages that need to be installed in order to run 01OS on your curl -sSL https://install.python-poetry.org | python3 - ``` +### MacOS + ```bash -# MacOS brew install portaudio ffmpeg cmake +``` -# Ubuntu (wayland not supported, only ubuntu 20.04 and below) -sudo apt-get install portaudio19-dev ffmpeg cmake +### Ubuntu + +Wayland not supported, only Ubuntu 20.04 and below -# Windows -On Windows you will need to install the following: +```bash +sudo apt-get install portaudio19-dev ffmpeg cmake ``` +### Windows + - [Git for Windows](https://git-scm.com/download/win). - [virtualenv](https://virtualenv.pypa.io/en/latest/installation.html) or [MiniConda](https://docs.anaconda.com/free/miniconda/miniconda-install/) to manage virtual environments. - [Chocolatey](https://chocolatey.org/install#individual) to install the required packages. From 83d472ed33e3f77f99753fb0378f0d084546f87b Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 15:38:36 -0400 Subject: [PATCH 08/30] add h1s to 01 light page --- docs/hardware/01-light.mdx | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index 98ee592..aa9b745 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -5,7 +5,9 @@ description: "Use your 01 Light" If you ordered your 01 Light and it was pre-assembled, you just need to connect it to your computer using the Captive Portal. If you're assembling it yourself, be sure to follow the M5 Atom instructions. -## Captive portal +# Pre-assembled + +### Captive portal To connect your 01, you will use the captive portal. @@ -15,10 +17,12 @@ To connect your 01, you will use the captive portal. Now you're connected and ready to go! -## M5 Atom Prep +# Build Your Own + +### M5 Atom Prep Instructions to set up your M5 Atom can be found here -## Suppliementary files +### Suppliementary files For CAD files, wiring diagram, and images, please visit the [01 Light hardware repository](https://github.com/OpenInterpreter/01/tree/main/hardware/light). From 89b91d2365d4f53d0182c689be8e2a6857432eb5 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Tue, 9 Jul 2024 15:53:51 -0400 Subject: [PATCH 09/30] improve software doc structure --- docs/mint.json | 7 ++++++- docs/software/{setup.mdx => configure.mdx} | 12 ++---------- docs/software/flags.mdx | 2 +- docs/software/installation.mdx | 12 +----------- docs/software/run.mdx | 18 ++++++++++++++++++ 5 files changed, 28 insertions(+), 23 deletions(-) rename docs/software/{setup.mdx => configure.mdx} (89%) create mode 100644 docs/software/run.mdx diff --git a/docs/mint.json b/docs/mint.json index d2d6edc..c28f93c 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -41,7 +41,12 @@ }, { "group": "Software Setup", - "pages": ["software/installation", "software/setup", "software/flags"] + "pages": [ + "software/installation", + "software/run", + "software/configure", + "software/flags" + ] }, { "group": "Hardware Setup", diff --git a/docs/software/setup.mdx b/docs/software/configure.mdx similarity index 89% rename from docs/software/setup.mdx rename to docs/software/configure.mdx index 31f7665..a1f81dc 100644 --- a/docs/software/setup.mdx +++ b/docs/software/configure.mdx @@ -1,16 +1,8 @@ --- -title: "Setup" -description: "Get your 01 server up and running" +title: "Configure" +description: "Configure your 01 instance" --- -## Run Server - -```bash -poetry run 01 --server -``` - -## Configure - A core part of the 01 server is the interpreter which is an instance of Open Interpreter. Open Interpreter is highly configurable and only requires updating a single file. diff --git a/docs/software/flags.mdx b/docs/software/flags.mdx index acfd369..2400252 100644 --- a/docs/software/flags.mdx +++ b/docs/software/flags.mdx @@ -1,6 +1,6 @@ --- title: "Flags" -description: "Customize the behaviour of your 01" +description: "Customize the behaviour of your 01 from the CLI" --- ## CLI Flags diff --git a/docs/software/installation.mdx b/docs/software/installation.mdx index 1d54c78..9cbbc93 100644 --- a/docs/software/installation.mdx +++ b/docs/software/installation.mdx @@ -28,14 +28,4 @@ Install your project along with its dependencies in a virtual environment manage poetry install ``` -Run your local version of 01 with: - -```bash -poetry run 01 -``` - -Use 01 with your 01 Light - -```bash -poetry run 01 --server -``` +Now you should be ready to [run your 01](/software/run) diff --git a/docs/software/run.mdx b/docs/software/run.mdx new file mode 100644 index 0000000..92d34ad --- /dev/null +++ b/docs/software/run.mdx @@ -0,0 +1,18 @@ +--- +title: "Run" +description: "Run your 01" +--- + + Make sure that you have navigated to the `software` directory. + +To run 01 with your computer's microphone and speaker, run: + +```bash +poetry run 01 +``` + +To use 01 with your 01 Light, run the server: + +```bash +poetry run 01 --server +``` From e4b656f59faa599c8d8f23560ca318784dd3ef38 Mon Sep 17 00:00:00 2001 From: Ben Steinher <130510994+human-bee@users.noreply.github.com> Date: Tue, 9 Jul 2024 14:18:15 -0700 Subject: [PATCH 10/30] Update faq.mdx Added FAQs that I haven't given clear answers to yet. --- docs/troubleshooting/faq.mdx | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 91122a6..56301a2 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -2,3 +2,22 @@ title: "FAQ" description: "Frequently Asked Questions" --- + +- How do you build on top of the 01? +- Where should I start? +- Is there a walk-through for connecting a device to the server? +- What are minimum hardware requirements? +- How do I have code run on the client-side? +- How do I build a profile? +- Where does the server run? + - Can an 01 device connect to the desktop app, or do general customers/consumers need to set it up in their terminal? + - Can I on/off certain tools? +- What firmware do I use to connect? + - What ideally do I need in my code to access the server correctly? +- Alternatives to nGrok? +- If my device runs off bluetooth connected to a phone, is there a mobile app to use to connect to the server? +- Uses a huge deal of API credits, what options do I have for using local models? Can these be run on the client device? +- Which model is best? +- Do I need to pay for a monthly subscription? +- Does the computer the O1 connects to need to always be on and running? If its in sleep mode will it wake up when I call on it? +- Which Model does 01 use? From 1ae939db9217f98ccd0d8061cf7ab26a7e7473e3 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 10:43:26 -0400 Subject: [PATCH 11/30] change configure to only reference profiles --- docs/software/configure.mdx | 26 ++++++-------------------- docs/software/installation.mdx | 2 +- 2 files changed, 7 insertions(+), 21 deletions(-) diff --git a/docs/software/configure.mdx b/docs/software/configure.mdx index a1f81dc..9482115 100644 --- a/docs/software/configure.mdx +++ b/docs/software/configure.mdx @@ -4,32 +4,18 @@ description: "Configure your 01 instance" --- A core part of the 01 server is the interpreter which is an instance of Open Interpreter. -Open Interpreter is highly configurable and only requires updating a single file. +Open Interpreter is highly configurable and only requires updating a profile. ```bash -# Edit i.py -software/source/server/i.py +# View profiles +poetry run 01 --profiles ``` Properties such as `model`, `context_window`, and many more can be updated here. -### LLM service provider - -If you wish to use a local model, you can use the `--llm-service` flag: +To assign a profile to your 01 instance, use the `--profile` flag: ```bash -# use llamafile -poetry run 01 --server --llm-service llamafile +# Use profile +poetry run 01 --profile ``` - -For more information about LLM service providers, check out the page on Language Models. - -### Voice Interface - -Both speech-to-text and text-to-speech can be configured in 01OS. - -You are able to pass CLI flags `--tts-service` and/or `--stt-service` with the desired service provider to swap out different services - -These different service providers can be found in `/services/stt` and `/services/tts` - -For more information, please read about speech-to-text and text-to-speech diff --git a/docs/software/installation.mdx b/docs/software/installation.mdx index 9cbbc93..9fa80ad 100644 --- a/docs/software/installation.mdx +++ b/docs/software/installation.mdx @@ -1,5 +1,5 @@ --- -title: "Installation" +title: "Install" description: "Get your 01 up and running" --- From ca1f7155db3968c540b6ab767455e1129e93cc63 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 11:22:56 -0400 Subject: [PATCH 12/30] update flags --- docs/software/flags.mdx | 51 +++++++---------------------------------- 1 file changed, 8 insertions(+), 43 deletions(-) diff --git a/docs/software/flags.mdx b/docs/software/flags.mdx index 2400252..ebd0f3d 100644 --- a/docs/software/flags.mdx +++ b/docs/software/flags.mdx @@ -23,55 +23,20 @@ description: "Customize the behaviour of your 01 from the CLI" - `--expose` Expose server to internet. +- `--client` + Run client. + - `--server-url TEXT` Specify the server URL that the client should expect. Defaults to server-host and server-port. Default: `None`. -- `--llm-service TEXT` - Specify the LLM service. - Default: `litellm`. - -- `--model TEXT` - Specify the model. - Default: `gpt-4`. - -- `--llm-supports-vision` - Specify if the LLM service supports vision. - -- `--llm-supports-functions` - Specify if the LLM service supports functions. - -- `--context-window INTEGER` - Specify the context window size. - Default: `2048`. - -- `--max-tokens INTEGER` - Specify the maximum number of tokens. - Default: `4096`. - -- `--temperature FLOAT` - Specify the temperature for generation. - Default: `0.8`. +- `--client-type TEXT` + Specify the client type. + Default: `auto`. -- `--tts-service TEXT` - Specify the TTS service. - Default: `openai`. - -- `--stt-service TEXT` - Specify the STT service. - Default: `openai`. - -- `--local` - Use recommended local services for LLM, STT, and TTS. - -- `--install-completion [bash|zsh|fish|powershell|pwsh]` - Install completion for the specified shell. - Default: `None`. - -- `--show-completion [bash|zsh|fish|powershell|pwsh]` - Show completion for the specified shell, to copy it or customize the installation. - Default: `None`. +- `--qr` + Display QR code to scan to connect to the server. - `--help` Show this message and exit. From 88b702de19b83b1009050bbf6693696e930522c3 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 14:44:23 -0400 Subject: [PATCH 13/30] profiles --- docs/software/configure.mdx | 27 +++++++++++++++++++++++---- 1 file changed, 23 insertions(+), 4 deletions(-) diff --git a/docs/software/configure.mdx b/docs/software/configure.mdx index 9482115..bdda328 100644 --- a/docs/software/configure.mdx +++ b/docs/software/configure.mdx @@ -4,18 +4,37 @@ description: "Configure your 01 instance" --- A core part of the 01 server is the interpreter which is an instance of Open Interpreter. -Open Interpreter is highly configurable and only requires updating a profile. +Open Interpreter is highly configurable and only requires updating or creating a profile. +Properties such as `model`, `context_window`, and many more can be updated here. + +To open the directory of all profiles, run: ```bash # View profiles poetry run 01 --profiles ``` -Properties such as `model`, `context_window`, and many more can be updated here. - -To assign a profile to your 01 instance, use the `--profile` flag: +To apply a profile to your 01 instance, use the `--profile` flag followed by the name of the profile ```bash # Use profile poetry run 01 --profile ``` + +### Standard Profiles + +`default.py` is the default profile that is used when no profile is specified. The default TTS is OpenAI. + +`fast.py` uses elevenlabs and groq, which are the fastest providers. + +`local.py` uses coqui TTS and runs the --local explorer from Open Interpreter. + +### Custom Profiles + +If you want to make your own file, you can do so by creating a new file in the `profiles` directory. +The easiest way is to duplicate an existing profile and then update values as needed. Be sure to save the profile with a unique name. + +```bash +# Use custom profile +poetry run 01 --profile +``` From 3b1f767011483fd663928bea6908c6bab3ef631c Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 14:47:19 -0400 Subject: [PATCH 14/30] remove references to i.py --- README.md | 2 +- docs/README_CN.md | 4 ++-- docs/README_FR.md | 4 ++-- docs/README_JA.md | 4 ++-- 4 files changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index c65dba3..66f7489 100644 --- a/README.md +++ b/README.md @@ -96,7 +96,7 @@ https://github.com/OpenInterpreter/01/assets/63927363/8621b075-e052-46ba-8d2e-d6 Dynamic System Messages enable you to execute code inside the LLM's system message, moments before it appears to the AI. ```python -# Edit the following settings in i.py +# Edit the following settings in Profiles interpreter.system_message = r" The time is {{time.time()}}. " # Anything in double brackets will be executed as Python interpreter.chat("What time is it?") # It will know, without making a tool/API call ``` diff --git a/docs/README_CN.md b/docs/README_CN.md index 8e7e53e..7ccea0f 100644 --- a/docs/README_CN.md +++ b/docs/README_CN.md @@ -85,7 +85,7 @@ https://github.com/OpenInterpreter/01/assets/63927363/8621b075-e052-46ba-8d2e-d6 动态系统消息使您能够在 LLM 系统消息出现在 AI 前的片刻内执行代码。 ```python -# Edit the following settings in i.py +# Edit the following settings in Profiles interpreter.system_message = r" The time is {{time.time()}}. " # Anything in double brackets will be executed as Python interpreter.chat("What time is it?") # It will know, without making a tool/API call ``` @@ -114,7 +114,7 @@ poetry run 01 --local ## 自定义 -要自定义系统的行为，请编辑 `i.py` 中的 [系统消息、模型、技能库路径](https://docs.openinterpreter.com/settings/all-settings) 等。这个文件设置了一个解释器，并由 Open Interpreter 提供支持。 +要自定义系统的行为，请编辑 Profiles 中的 [系统消息、模型、技能库路径](https://docs.openinterpreter.com/settings/all-settings) 等。这个文件设置了一个解释器，并由 Open Interpreter 提供支持。 ## Ubuntu 依赖项 diff --git a/docs/README_FR.md b/docs/README_FR.md index cdb0025..10fed0b 100644 --- a/docs/README_FR.md +++ b/docs/README_FR.md @@ -89,7 +89,7 @@ https://github.com/OpenInterpreter/01/assets/63927363/8621b075-e052-46ba-8d2e-d6 Les Messages Systèmes Dynamiques vous permettent d'exécuter du code à l'intérieur du message système du LLM, juste avant qu'il n'apparaisse à l'IA. ```python -# Modifiez les paramètres suivants dans i.py +# Modifiez les paramètres suivants dans Profiles interpreter.system_message = r" The time is {{time.time()}}. " # Tout ce qui est entre doubles crochets sera exécuté comme du Python interpreter.chat("What time is it?") # L'interpréteur connaitre la réponse, sans faire appel à un outil ou une API ``` @@ -118,7 +118,7 @@ Si vous souhaitez exécuter localement du speech-to-text en utilisant Whisper, v ## Personnalisation -Pour personnaliser le comportement du système, modifie [`system message`, `model`, `skills library path`,](https://docs.openinterpreter.com/settings/all-settings) etc. in `i.py`. Ce fichier configure un interprète alimenté par Open Interpreter. +Pour personnaliser le comportement du système, modifie [`system message`, `model`, `skills library path`,](https://docs.openinterpreter.com/settings/all-settings) etc. in Profiles. Ce fichier configure un interprète alimenté par Open Interpreter. ## Dépendances Ubuntu diff --git a/docs/README_JA.md b/docs/README_JA.md index d8eeb7c..2b837ee 100644 --- a/docs/README_JA.md +++ b/docs/README_JA.md @@ -88,7 +88,7 @@ https://github.com/OpenInterpreter/01/assets/63927363/8621b075-e052-46ba-8d2e-d6 ダイナミックシステムメッセージは、LLM のシステムメッセージが AI に表示される一瞬前に、その中でコードを実行することを可能にします。 ```python -# i.py の以下の設定を編集 +# Profiles の以下の設定を編集 interpreter.system_message = r" The time is {{time.time()}}. " # 二重括弧の中は Python として実行されます interpreter.chat("What time is it?") # ツール/API を呼び出すことなく、次のことが分かります ``` @@ -117,7 +117,7 @@ Whisper を使ってローカル音声合成を実行したい場合、Rust を ## カスタマイズ -システムの動作をカスタマイズするには、`i.py` 内の[システムメッセージ、モデル、スキルライブラリのパス](https://docs.openinterpreter.com/settings/all-settings)などを編集します。このファイルはインタープリターをセットアップするもので、Open Interpreter によって動作します。 +システムの動作をカスタマイズするには、Profiles 内の[システムメッセージ、モデル、スキルライブラリのパス](https://docs.openinterpreter.com/settings/all-settings)などを編集します。このファイルはインタープリターをセットアップするもので、Open Interpreter によって動作します。 ## Ubuntu 依存関係 From cefa9c4669113b42641e3a6868752ed56f70bf1a Mon Sep 17 00:00:00 2001 From: killian <63927363+KillianLucas@users.noreply.github.com> Date: Wed, 10 Jul 2024 11:49:20 -0700 Subject: [PATCH 15/30] Added more to FAQ --- docs/troubleshooting/faq.mdx | 28 +++++++++++++++++++++++++--- 1 file changed, 25 insertions(+), 3 deletions(-) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 56301a2..3a81b2e 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -8,16 +8,38 @@ description: "Frequently Asked Questions" - Is there a walk-through for connecting a device to the server? - What are minimum hardware requirements? - How do I have code run on the client-side? +We are working on supporting this, but we only support server-side code execution right now. - How do I build a profile? +We recommend running `--profiles`, duplicating a profile, then experimenting with the settings in the profile file (like `system_message`). - Where does the server run? - - Can an 01 device connect to the desktop app, or do general customers/consumers need to set it up in their terminal? - - Can I on/off certain tools? +The server runs on your home computer, or whichever device you want to control. +- Can an 01 device connect to the desktop app, or do general customers/consumers need to set it up in their terminal? +We are working on supporting external devices to the desktop app, but for now the 01 will need to connect to the Python server. +- Can I on/off certain tools? +We are working on building this feature, but it isn't avaliable yet. - What firmware do I use to connect? - - What ideally do I need in my code to access the server correctly? +- What ideally do I need in my code to access the server correctly? - Alternatives to nGrok? +We support `--tunnel-service bore` and `--tunnel-service localtunnel` in addition to `--tunnel-service ngrok`. [link to tunnel service docs] - If my device runs off bluetooth connected to a phone, is there a mobile app to use to connect to the server? - Uses a huge deal of API credits, what options do I have for using local models? Can these be run on the client device? +If you use `--profile local`, you won't need to use an LLM via an API. The 01 server will be responsible for LLM running, but you can run the server + client on the same device (simply run `poetry run 01` to test this.) - Which model is best? +We have found `gpt-4-turbo` to be the best, but we expect Claude Sonnet 1.5 to be comparable or better. - Do I need to pay for a monthly subscription? +If you use `--profile local`, you don't need to. For hosted language models, you may need to pay a monthly subscription. - Does the computer the O1 connects to need to always be on and running? If its in sleep mode will it wake up when I call on it? +The computer does need to be running, and will not wake up if a request is sent while it's sleeping. - Which Model does 01 use? +The 01 defaults to `gpt-4-turbo`. + +(from help email templates) + +Standalone Device/Hosted Server +We are exploring a few options about how to best provide a stand-alone device connected to a virtual computer in the cloud, provided by Open Interpreter. There will be an announcement once we have figured out the right way to do it. But the idea is that it functions with the same capabilities as the demo, just controlling a computer in the cloud, not the one on your desk at home. + +How Do I Get Involved +We are figuring out the best way to activate the community to build the next phase. For now, you can read over the Repository https://github.com/OpenInterpreter/01 and join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! + +Mobile App +The official app is being developed, and you can find instructions for how to set it up and contribute to development here: https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! \ No newline at end of file From d4c4229141e7072dfed0f95416a6d237bb47b85a Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:04:03 -0400 Subject: [PATCH 16/30] language model docs --- docs/guides/language-model.mdx | 32 +++++--------------------------- 1 file changed, 5 insertions(+), 27 deletions(-) diff --git a/docs/guides/language-model.mdx b/docs/guides/language-model.mdx index eb0aa8d..03b1d32 100644 --- a/docs/guides/language-model.mdx +++ b/docs/guides/language-model.mdx @@ -3,36 +3,14 @@ title: "Language Model" description: "The LLM that powers your 01" --- -## llamafile - -llamafile lets you distribute and run LLMs with a single file. Read more about llamafile [here](https://github.com/Mozilla-Ocho/llamafile) - -```bash -# Set the LLM service to llamafile -poetry run 01 --llm-service llamafile -``` - -## Llamaedge - -llamaedge makes it easy for you to run LLM inference apps and create OpenAI-compatible API services for the Llama2 series of LLMs locally. -Read more about Llamaedge [here](https://github.com/LlamaEdge/LlamaEdge) - -```bash -# Set the LLM service to Llamaedge -poetry run 01 --llm-service llamaedge -``` - ## Hosted Models -01OS leverages liteLLM which supports [many hosted models](https://docs.litellm.ai/docs/providers/). +The default LLM for 01 is GPT-4-Turbo. You can find this in the default profile in `software/source/server/profiles/default.py`. -To select your providers +The fast profile uses Llama3-8b served by Groq. You can find this in the fast profile in `software/source/server/profiles/fast.py`. -```bash -# Set the LLM service -poetry run 01 --llm-service openai -``` +## Local Models -## Other Models +You can use local models to power 01. -More instructions coming soon! +Using the local profile launches the Local Explorer where you can select your inference provider and model. The default options include Llamafile, Jan, Ollama, and LM Studio. From 4293884c5c0a84f092b9009cce99897e9f055cb9 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:09:05 -0400 Subject: [PATCH 17/30] tts docs --- docs/guides/text-to-speech.mdx | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/docs/guides/text-to-speech.mdx b/docs/guides/text-to-speech.mdx index 4a6688e..03dab15 100644 --- a/docs/guides/text-to-speech.mdx +++ b/docs/guides/text-to-speech.mdx @@ -1,22 +1,24 @@ --- title: "Text To Speech" -description: "The service to speak the text" +description: "The voice of 01" --- -## Piper (Local) +## Local TTS -This option installs piper to allow all text to speech to be done locally on device. +For local TTS, Coqui is used. -```bash -# Set a local TTS service -01 --tts-service piper +```python +# Set your profile with a local TTS service +interpreter.tts = "coqui" ``` -## OpenAI (Hosted) +## Hosted TTS -```bash -# Set TTS service -01 --tts-service openai +01 supports OpenAI and Elevenlabs for hosted TTS + +```python +# Set your profile with a hosted TTS service +interpreter.tts = "elevenlabs" ``` ## Other Models From c1babe19f40f33a1f824a2c122971c5ad7326938 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:15:42 -0400 Subject: [PATCH 18/30] add james' assembly video --- docs/guides/speech-to-text.mdx | 24 ------------------------ docs/hardware/01-light.mdx | 2 ++ 2 files changed, 2 insertions(+), 24 deletions(-) delete mode 100644 docs/guides/speech-to-text.mdx diff --git a/docs/guides/speech-to-text.mdx b/docs/guides/speech-to-text.mdx deleted file mode 100644 index d4dc044..0000000 --- a/docs/guides/speech-to-text.mdx +++ /dev/null @@ -1,24 +0,0 @@ ---- -title: "Speech To Text" -description: "Converts your voice into text" ---- - -## Whisper (Local) - -This option installs whisper-rust to allow all speech to text to be done locally on device. - -```bash -# Set a local STT service -01 --stt-service local-whisper -``` - -## Whisper (Hosted) - -```bash -# Set STT service -01 --stt-service openai -``` - -## Other Models - -More instructions coming soon! diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index aa9b745..59d6364 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -19,6 +19,8 @@ Now you're connected and ready to go! # Build Your Own +Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=37a5bgvoZy8) on how to assemble your 01 + ### M5 Atom Prep Instructions to set up your M5 Atom can be found here From d7ecb690ad81251a6dbf31c827ce5d35dad72a0c Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:18:48 -0400 Subject: [PATCH 19/30] update mint.json --- docs/mint.json | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-) diff --git a/docs/mint.json b/docs/mint.json index c28f93c..81f0a44 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -54,11 +54,7 @@ }, { "group": "Using 01", - "pages": [ - "guides/language-model", - "guides/speech-to-text", - "guides/text-to-speech" - ] + "pages": ["guides/language-model", "guides/text-to-speech"] }, { "group": "Troubleshooting", From 0204ed5aa33e6f6d4e20038b84f038bcebf3ce22 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:35:16 -0400 Subject: [PATCH 20/30] accordion the answered faqs --- docs/troubleshooting/faq.mdx | 70 ++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 23 deletions(-) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 3a81b2e..0605ebe 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -7,31 +7,55 @@ description: "Frequently Asked Questions" - Where should I start? - Is there a walk-through for connecting a device to the server? - What are minimum hardware requirements? -- How do I have code run on the client-side? -We are working on supporting this, but we only support server-side code execution right now. -- How do I build a profile? -We recommend running `--profiles`, duplicating a profile, then experimenting with the settings in the profile file (like `system_message`). -- Where does the server run? -The server runs on your home computer, or whichever device you want to control. -- Can an 01 device connect to the desktop app, or do general customers/consumers need to set it up in their terminal? -We are working on supporting external devices to the desktop app, but for now the 01 will need to connect to the Python server. -- Can I on/off certain tools? -We are working on building this feature, but it isn't avaliable yet. + + We are working on supporting this, but we only support server-side code + execution right now. + + + We recommend running `--profiles`, duplicating a profile, then experimenting + with the settings in the profile file (like `system_message`). + + + The server runs on your home computer, or whichever device you want to + control. + + + We are working on supporting external devices to the desktop app, but for + now the 01 will need to connect to the Python server. + + + We are working on building this feature, but it isn't avaliable yet. + - What firmware do I use to connect? - What ideally do I need in my code to access the server correctly? -- Alternatives to nGrok? -We support `--tunnel-service bore` and `--tunnel-service localtunnel` in addition to `--tunnel-service ngrok`. [link to tunnel service docs] + + We support `--tunnel-service bore` and `--tunnel-service localtunnel` in + addition to `--tunnel-service ngrok`. [link to tunnel service docs] + - If my device runs off bluetooth connected to a phone, is there a mobile app to use to connect to the server? -- Uses a huge deal of API credits, what options do I have for using local models? Can these be run on the client device? -If you use `--profile local`, you won't need to use an LLM via an API. The 01 server will be responsible for LLM running, but you can run the server + client on the same device (simply run `poetry run 01` to test this.) -- Which model is best? -We have found `gpt-4-turbo` to be the best, but we expect Claude Sonnet 1.5 to be comparable or better. -- Do I need to pay for a monthly subscription? -If you use `--profile local`, you don't need to. For hosted language models, you may need to pay a monthly subscription. -- Does the computer the O1 connects to need to always be on and running? If its in sleep mode will it wake up when I call on it? -The computer does need to be running, and will not wake up if a request is sent while it's sleeping. -- Which Model does 01 use? -The 01 defaults to `gpt-4-turbo`. + {" "} + {" "} + {" "} + + If you use `--profile local`, you won't need to use an LLM via an API. The + 01 server will be responsible for LLM running, but you can run the server + + client on the same device (simply run `poetry run 01` to test this.) + + + We have found `gpt-4-turbo` to be the best, but we expect Claude Sonnet 1.5 + to be comparable or better. + + + If you use `--profile local`, you don't need to. For hosted language models, + you may need to pay a monthly subscription. + + + The computer does need to be running, and will not wake up if a request is + sent while it's sleeping. + + + The 01 defaults to `gpt-4-turbo`. + (from help email templates) @@ -42,4 +66,4 @@ How Do I Get Involved We are figuring out the best way to activate the community to build the next phase. For now, you can read over the Repository https://github.com/OpenInterpreter/01 and join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! Mobile App -The official app is being developed, and you can find instructions for how to set it up and contribute to development here: https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! \ No newline at end of file +The official app is being developed, and you can find instructions for how to set it up and contribute to development here: https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! From 833520bf4311e6093d99313b6648603589a59a1a Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 15:43:45 -0400 Subject: [PATCH 21/30] remove weird cursor artifacts --- docs/troubleshooting/faq.mdx | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 0605ebe..8fd7925 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -33,9 +33,7 @@ description: "Frequently Asked Questions" addition to `--tunnel-service ngrok`. [link to tunnel service docs] - If my device runs off bluetooth connected to a phone, is there a mobile app to use to connect to the server? - {" "} - {" "} - {" "} + If you use `--profile local`, you won't need to use an LLM via an API. The 01 server will be responsible for LLM running, but you can run the server + From a7c96bed62ff9ec0f7dc20ca5450e3fff9058900 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Wed, 10 Jul 2024 16:07:44 -0400 Subject: [PATCH 22/30] remove preorder link from README --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index afe6d56..3dd705d 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,7 @@

The open-source language model computer.
-
Preorder the Light‎ ‎ |‎ ‎ Get Updates‎ ‎ |‎ ‎ Documentation
+
Get Updates‎ ‎ |‎ ‎ Documentation

From db3e2c3638b688c4071c7174216547b47be3788d Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 11:01:04 -0400 Subject: [PATCH 23/30] merge guides into configure --- docs/guides/language-model.mdx | 16 ---------------- docs/guides/text-to-speech.mdx | 26 -------------------------- docs/mint.json | 4 ---- docs/software/configure.mdx | 30 ++++++++++++++++++++++++++++++ 4 files changed, 30 insertions(+), 46 deletions(-) delete mode 100644 docs/guides/language-model.mdx delete mode 100644 docs/guides/text-to-speech.mdx diff --git a/docs/guides/language-model.mdx b/docs/guides/language-model.mdx deleted file mode 100644 index 03b1d32..0000000 --- a/docs/guides/language-model.mdx +++ /dev/null @@ -1,16 +0,0 @@ ---- -title: "Language Model" -description: "The LLM that powers your 01" ---- - -## Hosted Models - -The default LLM for 01 is GPT-4-Turbo. You can find this in the default profile in `software/source/server/profiles/default.py`. - -The fast profile uses Llama3-8b served by Groq. You can find this in the fast profile in `software/source/server/profiles/fast.py`. - -## Local Models - -You can use local models to power 01. - -Using the local profile launches the Local Explorer where you can select your inference provider and model. The default options include Llamafile, Jan, Ollama, and LM Studio. diff --git a/docs/guides/text-to-speech.mdx b/docs/guides/text-to-speech.mdx deleted file mode 100644 index 03dab15..0000000 --- a/docs/guides/text-to-speech.mdx +++ /dev/null @@ -1,26 +0,0 @@ ---- -title: "Text To Speech" -description: "The voice of 01" ---- - -## Local TTS - -For local TTS, Coqui is used. - -```python -# Set your profile with a local TTS service -interpreter.tts = "coqui" -``` - -## Hosted TTS - -01 supports OpenAI and Elevenlabs for hosted TTS - -```python -# Set your profile with a hosted TTS service -interpreter.tts = "elevenlabs" -``` - -## Other Models - -More instructions coming soon! diff --git a/docs/mint.json b/docs/mint.json index 81f0a44..e763131 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -52,10 +52,6 @@ "group": "Hardware Setup", "pages": ["hardware/01-light", "hardware/m5atom"] }, - { - "group": "Using 01", - "pages": ["guides/language-model", "guides/text-to-speech"] - }, { "group": "Troubleshooting", "pages": ["troubleshooting/faq"] diff --git a/docs/software/configure.mdx b/docs/software/configure.mdx index bdda328..2ebe16a 100644 --- a/docs/software/configure.mdx +++ b/docs/software/configure.mdx @@ -38,3 +38,33 @@ The easiest way is to duplicate an existing profile and then update values as ne # Use custom profile poetry run 01 --profile ``` + +### Hosted LLMs + +The default LLM for 01 is GPT-4-Turbo. You can find this in the default profile in `software/source/server/profiles/default.py`. + +The fast profile uses Llama3-8b served by Groq. You can find this in the fast profile in `software/source/server/profiles/fast.py`. + +### Local LLMs + +You can use local models to power 01. + +Using the local profile launches the Local Explorer where you can select your inference provider and model. The default options include Llamafile, Jan, Ollama, and LM Studio. + +### Hosted TTS + +01 supports OpenAI and Elevenlabs for hosted TTS + +```python +# Set your profile with a hosted TTS service +interpreter.tts = "elevenlabs" +``` + +### Local TTS + +For local TTS, Coqui is used. + +```python +# Set your profile with a local TTS service +interpreter.tts = "coqui" +``` From ef1e7119865a1d6a31aaa2a8421f2163539883ad Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 11:05:01 -0400 Subject: [PATCH 24/30] add llm examples to configure --- docs/software/configure.mdx | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/software/configure.mdx b/docs/software/configure.mdx index 2ebe16a..844e122 100644 --- a/docs/software/configure.mdx +++ b/docs/software/configure.mdx @@ -45,12 +45,22 @@ The default LLM for 01 is GPT-4-Turbo. You can find this in the default profile The fast profile uses Llama3-8b served by Groq. You can find this in the fast profile in `software/source/server/profiles/fast.py`. +```python +# Set your profile with a hosted LLM +interpreter.llm.model = "gpt-4o" +``` + ### Local LLMs You can use local models to power 01. Using the local profile launches the Local Explorer where you can select your inference provider and model. The default options include Llamafile, Jan, Ollama, and LM Studio. +```python +# Set your profile with a local LLM +interpreter.local_setup() +``` + ### Hosted TTS 01 supports OpenAI and Elevenlabs for hosted TTS From d7f65209898d97aaff6a2dd8e8cb13c46ba124d2 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 11:22:27 -0400 Subject: [PATCH 25/30] Incorporate updated firmware instructions into '01-light.mdx' --- docs/hardware/01-light.mdx | 20 +++++++++++++++++++- docs/hardware/m5atom.mdx | 20 -------------------- docs/mint.json | 2 +- 3 files changed, 20 insertions(+), 22 deletions(-) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index 59d6364..8c0acf8 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -23,7 +23,25 @@ Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v ### M5 Atom Prep -Instructions to set up your M5 Atom can be found here +The recommended hardware for the developer kit is based around the M5 Atom. It contains an ESP32 board, as well as a microphone and speaker. + +The ESP32 needs to be setup to work with the 01OS server. We've created the firmware needed, just follow this guide. + +## ESP32 + +To set up audio recording + playback on the ESP32 (M5 Atom), do the following: + +1. Download the [Arduino IDE](https://www.arduino.cc/en/software) +2. Get the firmware by copying the contents of `software/source/clients/esp32/src/client/client.ino` from the 01 repository. +3. Open Arduino IDE and paste the client.ino contents +4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif +5. Go to Tools -> Manage Libraries, then install the following: + +- M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) +- WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) + +6. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. +7. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. ### Suppliementary files diff --git a/docs/hardware/m5atom.mdx b/docs/hardware/m5atom.mdx index e7d34fa..c5e6954 100644 --- a/docs/hardware/m5atom.mdx +++ b/docs/hardware/m5atom.mdx @@ -2,23 +2,3 @@ title: "M5 Atom" description: "Prep your M5 Atom" --- - -The recommended hardware for the developer kit is based around the M5 Atom. It contains an ESP32 board, as well as a microphone and speaker. - -The ESP32 needs to be setup to work with the 01OS server. We've created the firmware needed, just follow this guide. - -## ESP32 - -To set up audio recording + playback on the ESP32 (M5 Atom), do the following: - -1. Download the [Arduino IDE](https://www.arduino.cc/en/software) -2. Get the firmware by copying the contents of `software/source/clients/esp32/src/client/client.ino` from the 01 repository. -3. Open Arduino IDE and paste the client.ino contents -4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif -5. Go to Tools -> Manage Libraries, then install the following: - -- M5Atom by M5Stack [Reference](https://www.arduino.cc/reference/en/libraries/m5atom/) -- WebSockets by Markus Sattler [Reference](https://www.arduino.cc/reference/en/libraries/websockets/) - -6. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. -7. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. diff --git a/docs/mint.json b/docs/mint.json index e763131..69479db 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -50,7 +50,7 @@ }, { "group": "Hardware Setup", - "pages": ["hardware/01-light", "hardware/m5atom"] + "pages": ["hardware/01-light"] }, { "group": "Troubleshooting", From 8888fbd30eac04c3fd13c69502452a7cadb263aa Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 11:41:59 -0400 Subject: [PATCH 26/30] Created new pages (custom_hardware, desktop, and mobile) with guides on controlling 01 from different devices. --- docs/hardware/custom_hardware.mdx | 4 ++++ docs/hardware/desktop.mdx | 4 ++++ docs/hardware/m5atom.mdx | 4 ---- docs/hardware/mobile.mdx | 4 ++++ docs/mint.json | 7 ++++++- 5 files changed, 18 insertions(+), 5 deletions(-) create mode 100644 docs/hardware/custom_hardware.mdx create mode 100644 docs/hardware/desktop.mdx delete mode 100644 docs/hardware/m5atom.mdx create mode 100644 docs/hardware/mobile.mdx diff --git a/docs/hardware/custom_hardware.mdx b/docs/hardware/custom_hardware.mdx new file mode 100644 index 0000000..4430d29 --- /dev/null +++ b/docs/hardware/custom_hardware.mdx @@ -0,0 +1,4 @@ +--- +title: "Custom Hardware" +description: "Control 01 from your own device" +--- diff --git a/docs/hardware/desktop.mdx b/docs/hardware/desktop.mdx new file mode 100644 index 0000000..e27a389 --- /dev/null +++ b/docs/hardware/desktop.mdx @@ -0,0 +1,4 @@ +--- +title: "Desktop" +description: "Control 01 from your computer" +--- diff --git a/docs/hardware/m5atom.mdx b/docs/hardware/m5atom.mdx deleted file mode 100644 index c5e6954..0000000 --- a/docs/hardware/m5atom.mdx +++ /dev/null @@ -1,4 +0,0 @@ ---- -title: "M5 Atom" -description: "Prep your M5 Atom" ---- diff --git a/docs/hardware/mobile.mdx b/docs/hardware/mobile.mdx new file mode 100644 index 0000000..816a404 --- /dev/null +++ b/docs/hardware/mobile.mdx @@ -0,0 +1,4 @@ +--- +title: "iOS & Android" +description: "Control 01 from your mobile phone" +--- diff --git a/docs/mint.json b/docs/mint.json index 69479db..5c116d4 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -50,7 +50,12 @@ }, { "group": "Hardware Setup", - "pages": ["hardware/01-light"] + "pages": [ + "hardware/01-light", + "hardware/custom-hardware", + "hardware/desktop", + "hardware/mobile" + ] }, { "group": "Troubleshooting", From 19ce6cb9c1bbda0fef0a9af187638cc49b62c2ff Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 11:57:05 -0400 Subject: [PATCH 27/30] "Refactor hardware documentation for 01 Light" --- docs/hardware/01-light.mdx | 54 ++++++++++++++++++++------------------ 1 file changed, 29 insertions(+), 25 deletions(-) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index 8c0acf8..7465ff3 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -3,36 +3,18 @@ title: "01 Light" description: "Use your 01 Light" --- -If you ordered your 01 Light and it was pre-assembled, you just need to connect it to your computer using the Captive Portal. If you're assembling it yourself, be sure to follow the M5 Atom instructions. +# Materials -# Pre-assembled +The Bill of Materials for the 01 Light can be found [here](https://github.com/OpenInterpreter/01/blob/main/hardware/light/BOM.md) -### Captive portal - -To connect your 01, you will use the captive portal. - -1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. -2. Enter your Wi-Fi/hotspot name and password in the captive portal page. -3. Enter the server URL generated on their computer and hit 'Connect'. - -Now you're connected and ready to go! - -# Build Your Own - -Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=37a5bgvoZy8) on how to assemble your 01 +# Chip (ESP32) -### M5 Atom Prep - -The recommended hardware for the developer kit is based around the M5 Atom. It contains an ESP32 board, as well as a microphone and speaker. - -The ESP32 needs to be setup to work with the 01OS server. We've created the firmware needed, just follow this guide. - -## ESP32 +The ESP32 needs to be setup to work with 01. We've created the firmware needed, just follow this guide. To set up audio recording + playback on the ESP32 (M5 Atom), do the following: 1. Download the [Arduino IDE](https://www.arduino.cc/en/software) -2. Get the firmware by copying the contents of `software/source/clients/esp32/src/client/client.ino` from the 01 repository. +2. Get the firmware by copying the contents of [client.ino](https://github.com/OpenInterpreter/01/blob/main/software/source/clients/esp32/src/client/client.ino) from the 01 repository. 3. Open Arduino IDE and paste the client.ino contents 4. Go to Tools -> Board -> Boards Manager, search "esp32", then install the boards by Arduino and Espressif 5. Go to Tools -> Manage Libraries, then install the following: @@ -43,6 +25,28 @@ To set up audio recording + playback on the ESP32 (M5 Atom), do the following: 6. The board needs to connect to WiFi. Once you flash, connect to ESP32 wifi "captive" which will get wifi details. Once it connects, it will ask you to enter 01OS server address in the format "domain.com:port" or "ip:port". Once its able to connect you can use the device. 7. To flash the .ino to the board, connect the board to the USB port, select the port from the dropdown on the IDE, then select the M5Atom board (or M5Stack-ATOM if you have that). Click on upload to flash the board. -### Suppliementary files +Check out [this video from Thomas](https://www.youtube.com/watch?v=Y76zed8nEE8) for flashing the ESP32 and connecting the 01. + +# Case -For CAD files, wiring diagram, and images, please visit the [01 Light hardware repository](https://github.com/OpenInterpreter/01/tree/main/hardware/light). +This case of the 01 can be 3d printed at home. It is recommended to use a resin printer for improved quality. + +Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=BjoO0Kt-IWM) for a deep dive on his design. + +The stl files can be found [here](https://github.com/OpenInterpreter/01/tree/main/hardware/light/bodies) + +# Assembly + +Check out [this video from James at CAD9 Design](https://www.youtube.com/watch?v=37a5bgvoZy8) on how to assemble your 01 + +# Connect + +### Captive portal + +To connect your 01, you will use the captive portal. + +1. Turn on your computer or laptop and connect to the '01 light' Wi-Fi network. +2. Enter your Wi-Fi/hotspot name and password in the captive portal page. +3. Enter the server URL generated on their computer and hit 'Connect'. + +Now you're connected and ready to go! From b32484e1299384a8c73791492c6da66fc45f57ab Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 15:57:09 -0400 Subject: [PATCH 28/30] Added instructions on how to use custom hardware with the 01 server in hardware documentation. --- docs/hardware/01-light.mdx | 2 +- docs/hardware/custom_hardware.mdx | 12 ++++++++++++ docs/hardware/desktop.mdx | 8 ++++++++ docs/hardware/mobile.mdx | 6 ++++++ docs/mint.json | 2 +- 5 files changed, 28 insertions(+), 2 deletions(-) diff --git a/docs/hardware/01-light.mdx b/docs/hardware/01-light.mdx index 7465ff3..ea4c5ea 100644 --- a/docs/hardware/01-light.mdx +++ b/docs/hardware/01-light.mdx @@ -9,7 +9,7 @@ The Bill of Materials for the 01 Light can be found [here](https://github.com/Op # Chip (ESP32) -The ESP32 needs to be setup to work with 01. We've created the firmware needed, just follow this guide. +To setup the ESP32 to work with 01, follow this guide to install the firmware: To set up audio recording + playback on the ESP32 (M5 Atom), do the following: diff --git a/docs/hardware/custom_hardware.mdx b/docs/hardware/custom_hardware.mdx index 4430d29..d3c00b7 100644 --- a/docs/hardware/custom_hardware.mdx +++ b/docs/hardware/custom_hardware.mdx @@ -2,3 +2,15 @@ title: "Custom Hardware" description: "Control 01 from your own device" --- + +You can build your own custom hardware that uses the 01 server. + +To use 01 with your custom hardware, run the server: + +```bash +poetry run 01 --server +``` + +You may need to set additional parameters via [flags](/software/flags) depending on your setup. + +To transmit audio commands to 01, send LMC audio chunks to the websocket defined by your server. diff --git a/docs/hardware/desktop.mdx b/docs/hardware/desktop.mdx index e27a389..782e006 100644 --- a/docs/hardware/desktop.mdx +++ b/docs/hardware/desktop.mdx @@ -2,3 +2,11 @@ title: "Desktop" description: "Control 01 from your computer" --- + + Make sure that you have navigated to the `software` directory. + +To run 01 with your computer's microphone and speaker, run: + +```bash +poetry run 01 +``` diff --git a/docs/hardware/mobile.mdx b/docs/hardware/mobile.mdx index 816a404..e83bcf7 100644 --- a/docs/hardware/mobile.mdx +++ b/docs/hardware/mobile.mdx @@ -2,3 +2,9 @@ title: "iOS & Android" description: "Control 01 from your mobile phone" --- + +1. our react-native WIP app, and mostly is similar to its README that explains how to use it with Expo, and + https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile + +2. the community native iOS app by eladekkal. + https://github.com/OpenInterpreter/01/tree/main/software/source/clients/ios diff --git a/docs/mint.json b/docs/mint.json index 5c116d4..3bdf7f4 100644 --- a/docs/mint.json +++ b/docs/mint.json @@ -52,7 +52,7 @@ "group": "Hardware Setup", "pages": [ "hardware/01-light", - "hardware/custom-hardware", + "hardware/custom_hardware", "hardware/desktop", "hardware/mobile" ] From 596e4e15a75e2648cdd23ac9bb11639ae95b7e97 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 16:17:29 -0400 Subject: [PATCH 29/30] "Update mobile control documentation: React Native app and community iOS app" --- docs/hardware/mobile.mdx | 71 +++++++++++++++++++++++++++++++++++++--- 1 file changed, 67 insertions(+), 4 deletions(-) diff --git a/docs/hardware/mobile.mdx b/docs/hardware/mobile.mdx index e83bcf7..acf42a6 100644 --- a/docs/hardware/mobile.mdx +++ b/docs/hardware/mobile.mdx @@ -3,8 +3,71 @@ title: "iOS & Android" description: "Control 01 from your mobile phone" --- -1. our react-native WIP app, and mostly is similar to its README that explains how to use it with Expo, and - https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile +Using your phone is a great way to control 01. There are multiple options available. -2. the community native iOS app by eladekkal. - https://github.com/OpenInterpreter/01/tree/main/software/source/clients/ios +## [React Native app](https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile) (iOS & Android) + +Work in progress, we will continue to improve this application. + +If you want to run it on your device, you will need to install [Expo Go](https://expo.dev/go) on your mobile device. + +### Setup Instructions + +- [Install 01 software](/software/installation) on your machine + +- Run the Expo server: + +```shell +cd software/source/clients/mobile/react-native +npm install # install dependencies +npx expo start # start local expo development server +``` + +This will produce a QR code that you can scan with Expo Go on your mobile device. + +Open **Expo Go** on your mobile device and select _Scan QR code_ to scan the QR code produced by the `npx expo start` command. + +- Run 01: + +```shell +cd software # cd into `software` +poetry run 01 --mobile # exposes QR code for 01 Light server +``` + +### Using the App + +In the 01 mobile app, select _Scan Code_ to scan the QR code produced by the `poetry run 01 --mobile` command + +Press and hold the button to speak, release to make the request. To rescan the QR code, swipe left on the screen to go back. + +## [Native iOS app](https://github.com/OpenInterpreter/01/tree/main/software/source/clients/ios) by [eladekkal](https://github.com/eladdekel). + +A community contibution ❤️ + +To run it on your device, you can either install the app directly through the current TestFlight [here](https://testflight.apple.com/join/v8SyuzMT), or build from the source code files in Xcode on your Mac. + +### Instructions + +- [Install 01 software](/software/installation) on your machine + +- In Xcode, open the 'zerooone-app' project file in the project folder, change the Signing Team and Bundle Identifier, and build. + +### Using the App + +To use the app there are four features: + +1. The speak "Button" + +Made to emulate the button on the hardware models of 01, the big, yellow circle in the middle of the screen is what you hold when you want to speak to the model, and let go when you're finished speaking. + +2. The settings button + +Tapping the settings button will allow you to input your websocket address so that the app can properly connect to your computer. + +3. The reconnect button + +The arrow will be RED when the websocket connection is not live, and GREEN when it is. If you're making some changes you can easily reconnect by simply tapping the arrow button (or you can just start holding the speak button, too!). + +4. The terminal button + +The terminal button allows you to see all response text coming in from the server side of the 01. You can toggle it by tapping on the button, and each toggle clears the on-device cache of text. From c5bb1a910e4eb2903d0bf84c8a0e6eed59762ed0 Mon Sep 17 00:00:00 2001 From: Mike Bird Date: Thu, 11 Jul 2024 16:21:00 -0400 Subject: [PATCH 30/30] Update FAQ: Add support for standalone devices & hosted servers, mobile apps. --- docs/troubleshooting/faq.mdx | 29 +++++++++++++++++++++-------- 1 file changed, 21 insertions(+), 8 deletions(-) diff --git a/docs/troubleshooting/faq.mdx b/docs/troubleshooting/faq.mdx index 8fd7925..8088ddf 100644 --- a/docs/troubleshooting/faq.mdx +++ b/docs/troubleshooting/faq.mdx @@ -55,13 +55,26 @@ description: "Frequently Asked Questions" The 01 defaults to `gpt-4-turbo`. -(from help email templates) + + We are exploring a few options about how to best provide a stand-alone device + connected to a virtual computer in the cloud, provided by Open Interpreter. + There will be an announcement once we have figured out the right way to do it. + But the idea is that it functions with the same capabilities as the demo, just + controlling a computer in the cloud, not the one on your desk at home. + -Standalone Device/Hosted Server -We are exploring a few options about how to best provide a stand-alone device connected to a virtual computer in the cloud, provided by Open Interpreter. There will be an announcement once we have figured out the right way to do it. But the idea is that it functions with the same capabilities as the demo, just controlling a computer in the cloud, not the one on your desk at home. + + We are figuring out the best way to activate the community to build the next + phase. For now, you can read over the Repository + https://github.com/OpenInterpreter/01 and join the Discord + https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing + to the open-source 01 Project! + -How Do I Get Involved -We are figuring out the best way to activate the community to build the next phase. For now, you can read over the Repository https://github.com/OpenInterpreter/01 and join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! - -Mobile App -The official app is being developed, and you can find instructions for how to set it up and contribute to development here: https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss ways to start contributing to the open-source 01 Project! + + The official app is being developed, and you can find instructions for how to + set it up and contribute to development here: + https://github.com/OpenInterpreter/01/tree/main/software/source/clients/mobile + Please also join the Discord https://discord.gg/Hvz9Axh84z to find and discuss + ways to start contributing to the open-source 01 Project! +