Building a voice-controlled AI assistant on a microcontroller sounds futuristic, but with the ESP32-S3 and cloud AI APIs, it's surprisingly achievable. This guide walks through the complete pipeline we use in the ProjectOSAI Companion Watch — from capturing audio to generating a spoken AI response.

The Voice Pipeline Architecture

The system works in three stages, all orchestrated by the ESP32-S3:

  1. Capture & Transcribe — Record audio from a microphone, send it to OpenAI's Whisper API for speech-to-text
  2. Think — Send the transcribed text to GPT-4o (or another LLM) and receive a text response
  3. Speak — Convert the AI's text response to audio using a TTS API and play it through a speaker

The ESP32-S3 acts as the orchestrator — it doesn't run the AI models locally (the chip doesn't have the memory for that), but it manages the entire flow over Wi-Fi.

What You'll Need

Hardware

Software & Accounts

Step 1: Audio Capture with I2S

The ESP32-S3's I2S peripheral can capture audio directly from a digital microphone. Configure it for 16kHz mono, which is what Whisper expects:

#include <driver/i2s.h>

#define I2S_PORT       I2S_NUM_0
#define SAMPLE_RATE    16000
#define SAMPLE_BITS    16
#define RECORD_SECONDS 5

void setupMicrophone() {
  i2s_config_t i2s_config = {
    .mode = (i2s_mode_t)(I2S_MODE_MASTER | I2S_MODE_RX),
    .sample_rate = SAMPLE_RATE,
    .bits_per_sample = I2S_BITS_PER_SAMPLE_16BIT,
    .channel_format = I2S_CHANNEL_FMT_ONLY_LEFT,
    .communication_format = I2S_COMM_FORMAT_STAND_I2S,
    .intr_alloc_flags = ESP_INTR_FLAG_LEVEL1,
    .dma_buf_count = 8,
    .dma_buf_len = 1024,
    .use_apll = false
  };

  i2s_pin_config_t pin_config = {
    .bck_io_num = 26,
    .ws_io_num = 25,
    .data_out_num = I2S_PIN_NO_CHANGE,
    .data_in_num = 33
  };

  i2s_driver_install(I2S_PORT, &i2s_config, 0, NULL);
  i2s_set_pin(I2S_PORT, &pin_config);
}

To capture audio, read from the I2S buffer into a byte array. For a 5-second recording at 16kHz/16-bit mono, you need 160,000 bytes — well within the ESP32-S3's PSRAM capacity.

Step 2: Send Audio to Whisper for Transcription

Whisper's API accepts audio as a multipart form upload. The ESP32's HTTPClient handles this, but you need to construct the multipart body manually:

String transcribeAudio(uint8_t* audioData, size_t audioLen) {
  HTTPClient http;
  http.begin("https://api.openai.com/v1/audio/transcriptions");
  http.addHeader("Authorization", "Bearer " + String(OPENAI_KEY));

  // Build multipart form data
  String boundary = "----ESP32Boundary";
  http.addHeader("Content-Type",
    "multipart/form-data; boundary=" + boundary);

  // Construct body with WAV header + audio data
  // ... (WAV header generation + multipart encoding)

  int code = http.POST(body);
  if (code == 200) {
    // Parse JSON response for "text" field
    return extractText(http.getString());
  }
  return "";
}

Pro tip: Whisper works best with WAV format. You'll need to prepend a valid WAV header (44 bytes) to your raw PCM data before uploading. The header encodes sample rate, bit depth, and data length.

Step 3: Send Text to GPT-4o

With the transcribed text in hand, send it to GPT-4o's chat completions endpoint. This is a standard JSON POST request:

String askGPT(String userMessage) {
  HTTPClient http;
  http.begin("https://api.openai.com/v1/chat/completions");
  http.addHeader("Authorization", "Bearer " + String(OPENAI_KEY));
  http.addHeader("Content-Type", "application/json");

  // Build the request JSON
  StaticJsonDocument<2048> doc;
  doc["model"] = "gpt-4o";
  doc["max_tokens"] = 150;  // Keep responses short for TTS

  JsonArray messages = doc.createNestedArray("messages");
  JsonObject sysMsg = messages.createNestedObject();
  sysMsg["role"] = "system";
  sysMsg["content"] = "You are a helpful AI assistant on a "
    "smartwatch. Keep responses brief and conversational.";

  JsonObject userMsg = messages.createNestedObject();
  userMsg["role"] = "user";
  userMsg["content"] = userMessage;

  String body;
  serializeJson(doc, body);

  int code = http.POST(body);
  if (code == 200) {
    // Parse response for choices[0].message.content
    return extractResponse(http.getString());
  }
  return "Sorry, I couldn't process that.";
}

Keep max_tokens low (100-200) for wearable use. Long responses take too long to generate and too long to speak — nobody wants a 2-minute monologue from their wrist.

Advertisement

Step 4: Text-to-Speech Playback

OpenAI's TTS API returns audio data that you can stream directly to the I2S speaker output. The tts-1 model is faster (lower latency) while tts-1-hd sounds better:

void speakText(String text) {
  HTTPClient http;
  http.begin("https://api.openai.com/v1/audio/speech");
  http.addHeader("Authorization", "Bearer " + String(OPENAI_KEY));
  http.addHeader("Content-Type", "application/json");

  String body = "{\"model\":\"tts-1\","
    "\"input\":\"" + text + "\","
    "\"voice\":\"nova\","
    "\"response_format\":\"pcm\"}";

  int code = http.POST(body);
  if (code == 200) {
    // Stream response directly to I2S output
    WiFiClient* stream = http.getStreamPtr();
    uint8_t buf[1024];
    while (stream->available()) {
      int len = stream->readBytes(buf, sizeof(buf));
      size_t written;
      i2s_write(I2S_NUM_1, buf, len, &written, portMAX_DELAY);
    }
  }
  http.end();
}

Request pcm format (raw audio) instead of MP3 — it saves you from needing an MP3 decoder library on the ESP32 and can be piped directly to I2S.

Step 5: Putting It All Together

The main loop ties everything together with a simple push-to-talk flow:

void loop() {
  if (digitalRead(BUTTON_PIN) == LOW) {
    // 1. Record audio
    uint8_t* audio = recordAudio(RECORD_SECONDS);

    // 2. Transcribe with Whisper
    String transcript = transcribeAudio(audio, audioLength);
    free(audio);

    if (transcript.length() > 0) {
      // 3. Get AI response
      String response = askGPT(transcript);

      // 4. Speak the response
      speakText(response);
    }
  }
  delay(50);
}

Optimizations for Wearable Use

If you're building this into a wearable (like we did with the ProjectOSAI watch), there are several things to consider:

Cost Considerations

Running this pipeline costs roughly $0.01–0.03 per conversation (Whisper + GPT-4o + TTS). At 20 conversations per day, that's about $6–18/month. Use our AI API Cost Calculator to estimate your specific usage.

What's Next

This guide covers the core pipeline, but there's much more you can add — wake word detection, display animations, persistent memory, reminder systems, and companion personalities. The ProjectOSAI Companion Watch implements all of these on the same ESP32-S3 platform.

If you want to skip the build and get a ready-made AI companion on your wrist, check out the watch.