Selected work

Audio tools and systems for making music.

Doctoral research · Ongoing

Composer-controlled music

Game music needs to respond to changing play without losing the composer’s intended style and structure. My research asks how AI can support that adaptation while leaving creative decisions with the composer.

In the current offline prototype, a composer sets rules for instrument changes. The system checks AI proposals against those rules and generates approved parts, allowing the resulting mixes to be reviewed before pursuing a real-time game system.

  • Python
  • LLM proposals
  • ACE-Step music generation
View research
How it works

The prototype applies composer-defined rules to LLM proposals and generates approved instrument parts with ACE-Step. I compared decisions across 10 scripted scenarios and assessed 36 mixes in a single-composer study; these are descriptive results from a limited offline evaluation.

Interactive audio · Research prototype

DSP graph generator

Turning a sound idea into a synthesizer patch means choosing audio nodes, connecting them and adjusting parameters. This prototype explores whether language prompts can make that process easier to experiment with.

Describe a sound or change you want; the browser prototype proposes an audio graph you can inspect, edit and hear. Proposed changes are checked before they are applied, keeping the signal path visible and editable.

  • Node.js
  • Fastify
  • Zod
  • React Flow
  • Tone.js
Contact about this prototype
How it works

Fastify and Zod check model-generated actions before graph updates. React Flow shows the signal path, while Tone.js playback and parameter controls let users inspect and hear the resulting sound.

Audio software · Public source

Speaker matching

This started during my local voice conversion experiments with RVC (Retrieval-based Voice Conversion): I needed to quickly collect a character’s speech from videos with multiple speakers. Listening through every recording and cutting out each line by hand was repetitive.

Provide a few short reference clips for each target speaker, then batch-process the recordings’ audio to find and export matching speech, grouped by speaker and labelled with source timestamps and similarity scores. Set your own matching threshold to favour stronger matches or include more candidates for review.

  • Python
  • PyTorch
  • Silero VAD / ONNX
  • NVIDIA TitaNet / NeMo
View code
How it works

Silero VAD detects speech and TitaNet compares it with the reference voices. GPU batching and threaded loading support batch processing. WAV exports preserve the source sample rate and channels; overlapping voices may remain.

Other projects

Game coaching · Agent skill · Manual handoff prototype

VALORANT VOD Coach

A scoreboard shows the result but not why a peek, rotation or team call went wrong. Replays can also make an in-game choice look obvious in hindsight.

I built an Agent skill and local workflow that prepare first-person recordings, optional replays and team audio for an AI assistant. It explains key decisions with video timestamps and suggests practice tasks, keeping what the player knew separate from what the replay reveals.

Python · JavaScript · FFmpeg · faster-whisper · pyannote.audio · PyTorch · Agent workflows

View skill
Setup and current scope

The main workflow freezes materials and annotations into a handoff prompt for an external AI assistant with local file, image and media-tool access. Full-match reviews are started manually, and their reports are opened from the assistant’s output paths. Optional API adapters (OpenAI, Gemini, Anthropic or Ollama) support limited image-and-text steps or a single-case experiment, not automatic full-match audio/video review. Audio preparation requires optional local models; check model judgments against the recordings.

From recordings to coaching

  1. Prepare the match

    Select a local first-person recording and optional replay, preview them, choose audio tracks, and confirm the map, agents and your player. Save a draft or skip audio when it is unavailable.

    Tools: Python standard-library HTTP server · HTML / CSS / JavaScript · ffprobe for media metadata.

  2. Organise speech locally

    With the optional audio setup installed, the preparation tool extracts selected tracks, transcribes speech and groups anonymous speakers. These provide review candidates; a transcript or speaker group alone does not establish who spoke.

    Tools: FFmpeg for audio extraction · faster-whisper with Whisper large-v3 / CTranslate2 for transcription · pyannote.audio Community-1 / PyTorch for speaker diarization.

  3. Listen, select and label

    Play a short clip and select the teammate you recognise. Mark game voices, overlapping speakers or unknown sounds, and correct transcript text separately. Clean confirmed samples help find similar clips for review; suggested identities remain distinct from human confirmation.

    Tools: Community-1 speaker embeddings · cosine similarity with score and competing-candidate margin thresholds · JavaScript labeling interface.

  4. Check sound against the picture

    Open the clip alongside a muted source-video preview. Play, pause and seek them together to check the timing, then record a match or mismatch. Saved labels can be revisited and corrected; each timing check applies to that clip.

    Tools: HTML5 audio / video · JavaScript playback and seek events · original source timestamps.

  5. Inspect the match and key moments

    After manual handoff, the AI assistant follows the Skill to review a whole-match frame index, then inspect fights, rotations and communication windows more closely. When a replay is available, it aligns round events and distinguishes player knowledge from later replay information.

    Tools: FFmpeg / ffprobe for timestamped frames · 1 FPS navigation and 4 / 8 FPS key windows · a compatible external AI assistant for visual interpretation.

  6. Turn evidence into practice

    Connect the reviewed footage and available voice evidence to timecoded explanations and concrete drills with observable goals. Paired reviews can produce individual and team reports; uncertain speaker identities or missing evidence remain explicit.

    Tools: Agent Skill instructions · structured JSON evidence records · Markdown and offline HTML reports.

How the speaker tools differ

Speaker matching starts with reference voices and uses Silero VAD (ONNX) and NVIDIA TitaNet via NeMo to export matching speech. This coach first groups anonymous speakers with Community-1, then uses human labels and Community-1 embeddings to propose teammate matches. Both compare voice similarity; the coach also adds transcription and video context.

Post-match review only. Unofficial and unaffiliated with Riot Games.

Project source on GitHub

Desktop software · In development

SiftFrame

Reviewing a photo collection means repeatedly comparing images and deciding what to keep. SiftFrame brings previews, keep/reject choices and undo/redo together in a local Windows workflow.

Python · PySide6 / Qt Widgets

View project

Photo selections use three states: undecided, keep and reject, with session undo and redo. A standalone Windows release has not yet been validated.

SiftFrame’s native Qt welcome screen with an empty project list.

Creative web project · Ongoing

MyPick

Favourite songs are easy to list but harder to arrange into a board you can share. MyPick lets fans build a personal Top 10, compare boards and save or export their selections.

Next.js · TypeScript · React

Open MyPick
About the project

Adapted an open-source song picker into a shared codebase for three groups, adding local board backup, comparison tools and export templates.

Adapted from rurimegu/MyPickHasunosora (MIT), an unofficial fan project.

≒JOY · ≠ME · Source code

More on GitHub