GiobGeab Lipsync API

A REST API that converts Irish language text or audio into lip sync animation data for use with 3D character models. Built as part of the ABAIR project.

---

What It Does

Given a piece of Irish language text or an audio file, the API returns:

• A viseme train: a timestamped sequence of mouth shapes that drive character lip sync animation. Mouth shapes are described as per the ABAIR Viseme Inventory.
• An audio URL (for text input the URL is synthesised via ABAIR TTS, else the URL contains the same audio as inputted)
• Resolved animation cues: actions such as nods and smiles triggered at precise word boundaries or timestamps

This data is consumed by client applications to animate 3D character models in real time, without the API needing to know anything about the renderer.

---

Architecture Overview

Client
  │
  ├── POST /lipsync/sync     → Fast synchronous path (text input only)
  ├── POST /jobs             → Async path for audio URLs
  ├── POST /jobs/audio       → Async path for file uploads
  └── GET  /jobs/{id}        → Poll for result
         │
         ▼
    FastAPI (api/)
         │
         ├── Text input  → ABAIR TTS API → phoneme parse → viseme train
         └── Audio input → Rhubarb (local binary) → viseme train
                                   │
                              Celery Worker (worker/)
                                   │
                              Redis (job queue + results)

Services

Service	Role
FastAPI	Web server: receives requests, validates input, returns responses
Redis	Message broker and result store between FastAPI and Celery
Celery Worker	Processes audio jobs in the background (Rhubarb, merge)
Celery Beat	Scheduler: runs periodic cleanup of uploaded files
ABAIR	External Irish TTS API: text → audio + phoneme timing
Rhubarb	Local binary: audio → mouth shape timing
ffmpeg	Audio file converter: converts MP3 uploads to WAVs

---

Endpoints

POST /lipsync/sync

Synchronous. Text input only. Returns immediately.

{
  "text": "Dia duit! Conas atá tú?",
  "animation_cues": [
    { "trigger_phrase": "Dia duit", "instance": 1, "action": "nod" },
    { "trigger_phrase": "Conas atá tú", "instance": 1, "action": "smile" }
  ]
}

POST /jobs

Async. Audio input via external URL. Returns a job_id to poll.

{
  "audio_url": "https://example.com/audio.mp3",
  "animation_cues": [
    { "trigger_time": 1.5, "action": "nod" }
  ]
}

POST /jobs/audio

Async. Audio input via file upload. Single step: no separate upload needed. MP3 and WAV are both supported, however WAV will respond slightly quicker as MP3s need to first be converted to WAV.

multipart/form-data:
  file: <wav or mp3 file>
  animation_cues: [{"trigger_time": 1.5, "action": "nod"}]

GET /jobs/{job_id}

Poll for the result of an async job.

POST /upload

Upload an audio file and receive a hosted URL for use in /jobs.

---

Response Format

All endpoints return a JobResponse:

{
  "job_id": "uuid",
  "status": "done",
  "result": {
    "audio_url": "https://...",
    "visemes": [
      { "symbol": "dj", "end": 0.15, "viseme": "TD_S" }
    ],
    "actions": [
      { "action": "nod", "start": 0.023 }
    ]
  },
  "error": null
}

Status values: pending → processing → done / failed

---

Running Locally

Requirements

• Python 3.12+
• Redis
• Rhubarb Lip Sync binary
• Docker (optional, but running with docker means you dont need to install all services yourself)

With Docker (recommended)

docker compose up --build

Environment Variables

See .env file. Key variables:

REDIS_URL=redis://localhost:6379/0
RHUBARB_PATH=/path/to/rhubarb
UPLOAD_DIR=/tmp/giolipsync/uploads
UPLOAD_BASE_URL=http://localhost:8000/uploads
CLEANUP_INTERVAL_SECONDS=1800
FILE_MAX_AGE_SECONDS=1800

---

Interactive Docs

FastAPI generates interactive API documentation automatically:

http://localhost:8000/docs        (local)
https://giob-geab.abair.ie/docs   (production)

---

Production

The API is deployed at https://giob-geab.abair.ie.