3.1. State-of-the-Art in LLMs for Music Generation and Control

Large Language Models have demonstrated remarkable capabilities in processing and generating sequential data, leading to their increasing application in the music domain.[1] Their use spans symbolic music generation, direct audio synthesis, and the control of musical parameters.[1, 7]

In symbolic music generation, LLMs like ChatMusician have shown proficiency in working with text-compatible representations such as ABC notation, which inherently encodes rhythmic information and musical structure.[1, 8] Similarly, text2midi demonstrates LLMs generating MIDI files from textual descriptions, with controllability over music theory terms including tempo.[1, 9] NotaGen further applies LLM training paradigms (pre-training, fine-tuning, and reinforcement learning with CLaMP-DPO) to classical sheet music generation in ABC notation, emphasizing musicality and structural coherence.[1, 10] These examples illustrate LLMs' capacity to handle symbolic formats that explicitly contain timing and rhythmic data, a crucial aspect for beat-synchronized SVS and relevant to RQ1 (LLM rhythmic interpretation) and the design of the RCG (RQ3). The MetaScore dataset leverages LLMs to generate natural language captions from metadata, enabling text-conditioned symbolic music generation and offering insights for dataset strategies and LLM fine-tuning.[11, 12, 13] Recent work also explores LLMs for symbolic music editing via textual instructions, such as modifying drum grooves represented in a "drumroll" notation.[14, 15, 16]

LLMs are also being explored for their ability to model and generate expressive musical performances. For instance, the CrossMuSim framework utilizes LLMs to generate rich textual descriptions for music similarity modeling [1], and the CFE+P model, a transformer-based system, predicts notewise dynamics and micro-timing adjustments from inexpressive MIDI input.[1] The Seed-Music framework employs auto-regressive LMs and diffusion for vocal music generation with performance controls from multi-modal inputs like lyrics and style descriptions.[17]

Despite these advancements, significant challenges remain in applying LLMs to real-time music generation and control. Chief among these are the inference speed of large models and the difficulty in achieving fine-grained, temporally precise control necessary for tasks like beat synchronization.[1]

A recurring theme in successful LLM-music systems, particularly those dealing with complex structures and precise timing (e.g., NotaGen [10], ChatMusician [8], drum groove editing [14]), is the utilization of well-defined, text-compatible symbolic representations. These formats, such as ABC notation or custom textual "drumroll" formats, explicitly encode rhythmic information in a manner that aligns with the sequential processing strengths of LLMs. This observation strongly suggests that for the LLM in the proposed system to effectively interpret beat data and generate the precise rhythmic control signals required by RQ1 and Objective O2, the RCG (targeted by RQ3 and Objective O3) should ideally be a symbolic or structured-text format. Such a representation is more amenable to LLM generation and manipulation compared to direct acoustic feature parameters or complex binary formats, which are less natural for LLM outputs. This consideration has significant implications for the design choices related to the RCG.

4. Achieving Real-Time Performance and Musical Nuance

The successful realization of an LLM-controlled beat-synchronized SVS system hinges on two intertwined aspects: achieving stringent real-time performance and imbuing the synthesized vocals with musical nuance and expressivity. This section details the strategies for advanced beat tracking, LLM-driven rhythmic interpretation, and comprehensive latency management.

4.1. Advanced Beat Tracking and Synchronization Strategies

The Live Audio Input and Advanced Beat Tracking Module is the system's perceptual front-end, responsible for capturing a live audio stream and performing real-time beat detection.[1] State-of-the-art approaches include traditional signal processing methods like onset detection, spectral flux analysis, and comb filtering, as well as machine learning models, often implemented using libraries such as Madmom or Librosa.[1] For this project, beyond basic BPM and beat timestamps, the aim is to extract richer rhythmic information, such as downbeat positions, inferred time signatures, and potentially characteristics of the musical "feel" (e.g., swing ratio, rhythmic intensity).[1] Recent research into beat tracking in singing voices, utilizing Self-Supervised Learning (SSL) front-ends like WavLM and DistilHuBERT followed by transformers [1, 55], is particularly relevant if the live audio input is melodic or contains vocals, as these methods are designed for less percussive cues.

Key challenges in live beat tracking include handling tempo drift, managing complex or ambiguous rhythmic patterns (syncopation, rubato), and minimizing the inherent latency of the detection process itself.[1] An inaccurate or delayed beat track will inevitably lead to poor synchronization of the SVS output.

Once beats are detected, robust synchronization logic is required to align the SVS output temporally with the detected beats.[1] This involves managing data flow and control signals between modules. Strategies include predictive processing (anticipating upcoming beats if the rhythm is stable), intelligent buffering and jitter management (smoothing processing time variations), and potentially advanced closed-loop control mechanisms where the AI monitors its own output and makes dynamic corrections.[1] Recent advancements in live music synchronization with AI, such as the ReaLJam system which uses anticipation and a specific client-server protocol for human-AI jamming [56, 57], offer valuable architectural insights for managing synchronization in interactive contexts.

The richness of the information extracted by the beat tracking module is crucial not only for achieving basic synchronization (Objective O2 [1]) but also for enabling the LLM to generate nuanced expressive timing (Objective O4, RQ5 [1]). If the beat tracker only provides rudimentary BPM and beat event timestamps, the LLM has limited information from which to infer musical groove, style, or feel. Conversely, if the beat tracking module can output richer data—such as identified downbeats, inferred metrical structure, or even quantitative measures of swing or rhythmic intensity, as suggested in the research plan [1]—the LLM is provided with a more detailed canvas upon which to paint its expressive interpretations. Therefore, the "advanced" nature of the beat tracking is a direct prerequisite for the "advanced" expressive capabilities sought from the LLM.

4.2. LLM-driven Rhythmic Interpretation and Expressive Timing

Achieving musically compelling beat-synchronized singing requires the LLM to transcend mere metronomic alignment and imbue the performance with expressive timing and interpretation that reflects the character of the live beat and the musical style.[1] This involves several strategies:

• LLM Training and Fine-tuning for Musical Expressivity: Fine-tuning pre-trained foundation models on specialized datasets that include expressive singing performances aligned with detailed beat and rhythmic annotations (micro-timing, dynamics, articulation) is key.[1] Recent LLM fine-tuning strategies, such as instruction merging (e.g., MergeIT [58]) or federated fine-tuning [59], could be explored for adapting LLMs to these specific musical tasks.
• Advanced Prompt Engineering for Rhythmic Control: Designing effective prompting strategies will allow users or other AI modules to guide the LLM's rhythmic interpretation at a high level.[1] Research on prompt engineering for LLM reliability and task-specific alignment (e.g., PROMPTEVALS [36, 37]) can inform this. Natural language cues like "sing this phrase with a laid-back bluesy feel" or "make the rhythm very tight and punchy" could be learned by the LLM to translate into specific modifications of the RCG or SVS parameters. Systems like VersBand demonstrate prompt-based control over singing and music styles [31, 50], and the "Not that Groove" paper shows LLM-based editing of drum grooves from textual instructions.[14, 16] ChatMusician also uses prompts with ABC notation for musical control.[8]
• Modeling Explicit Expressive Parameters: The LLM will be trained to output explicit parameters controlling expressive elements such as micro-timing deviations, dynamic accents, and articulation control (staccato, legato).[1] This is inspired by models like CFE+P [1] and requires the SVS module to be responsive to these fine-grained inputs. Systems like LLFM-Voice aim for fine-grained emotional generation including vocal techniques, tension, and pitch [1, 3, 4, 41], and S2Cap focuses on captioning singing style attributes including volume and mood [60, 61, 62], indicating growing interest in detailed expressive control. TCSinger 2 also allows multi-level style control via diverse prompts.[1, 18, 26]
• Incorporating Musical Knowledge and Style Awareness: Methods to imbue the LLM with deeper understanding of musical styles and their rhythmic conventions (e.g., specific swing ratios for jazz) will be explored, potentially through style-specific datasets or style embeddings.[1]

For the LLM to effectively generate expressive timing (Objective O4, RQ5 [1]), it requires a well-defined "vocabulary" of expressive parameters that it can output, typically via the RCG. This vocabulary, encompassing elements like micro-timing shifts in milliseconds, relative accent strengths, or specific articulation types (e.g., staccato, legato), must be co-designed with the capabilities of the SVS module. If the chosen SVS engine cannot faithfully render these subtle variations, the LLM's expressive intent will be lost. This underscores the importance of addressing Gap G2 (SVS Responsiveness) in conjunction with developing the LLM's expressive capabilities. The RCG (Objective O3 [1]) must be able to encode these shared parameters, ensuring that the LLM's high-level musical intelligence and expressive direction are accurately translated into precise, low-latency execution by the micro-timing module and SVS engine.

4.3. Latency Management in Integrated LLM-SVS Systems

Minimizing end-to-end latency is paramount for any real-time interactive system, and it is a core target (Objective O1: <100ms) of this research.[1, 1] Psychoacoustic research indicates that while practiced musicians can detect discrepancies as low as 10ms [63], and latencies above 30-50ms can make performances feel "sloppy" or "sluggish" [63], tolerances can extend up to 50-65ms in some musical duo contexts, especially with slower tempos or continuous sounds.[64] The Just Noticeable Difference (JND) for rhythmic asynchrony can be very small, sometimes less than 10ms for certain stimuli [65, 66], though perceived audiovisual synchrony can tolerate asynchronies exceeding 200ms in some complex scenarios.[67] For interactive music systems, values between 128ms and 158ms have been reported as mean thresholds in some spatial audio contexts [1, 56], but sub-100ms is generally desirable for responsive interaction.[1, 33] The 100ms target for this project represents an upper bound for achieving basic interactivity.

Several strategies will be employed for latency mitigation [1, 1]:

• Model Optimization: Techniques such as quantization, pruning, and knowledge distillation for both LLM and SVS models.
• Efficient Architectures: Utilizing LLMs and SVS models designed for speed. This includes Multi-Token Prediction (MTP) as seen in VocalNet [1, 1, 45], and exploring emerging MatMul-free LLM architectures [68] or specialized speech interaction models like LLaMA-Omni [51, 69, 70, 71] which report response latencies as low as 226-236ms for simultaneous text and speech generation.
• Streaming Synthesis: Processing input and outputting audio in small chunks. This is supported by systems like XTTS-v2 [1, 1], community APIs for GPT-SoVITS [1, 1], CSSinger [1, 1, 5, 48], and Tungnaá.[1, 7, 33]
• Predictive Processing: Anticipating upcoming beats if the rhythm is stable.
• Hardware Acceleration: Leveraging GPUs/TPUs.
• Intelligent Buffering and Jitter Management: Smoothing out processing time variations.
• Recent LLM Inference Optimizations: Techniques such as optimizing Time To First Token (TTFT) and Time Between Tokens (TBT), advanced KV cache management (PagedAttention, vLLM, PQCache), and sophisticated model placement and request scheduling strategies (Skip-Join MLFQ, dynamic SplitFuse) are critical for reducing LLM inference latency.[72] While some network-focused low-latency research like OCDMA over PON [73] is less directly applicable, it highlights the broader engineering efforts towards minimizing delay in real-time systems.

The <100ms target for *end-to-end* latency necessitates a meticulous "latency budget" allocation for each stage of the pipeline: beat tracking, LLM processing (including RCG generation), RCG transmission and parsing, SVS rendering, and audio output buffering. The proposed hybrid architecture [1] implicitly addresses this by minimizing the LLM's direct involvement in per-beat, time-critical computations, relegating it to macro-control. However, even these macro-control signals must arrive in time to influence upcoming musical sections. Therefore, rigorous profiling of each module's latency contribution, as specified in Objective O1 and the evaluation plan [1], will be essential to identify and mitigate bottlenecks, ensuring the cumulative delay remains within the acceptable perceptual threshold for interactive musical performance.

5.3. Dataset Strategy: Creation, Augmentation, and Few-Shot Learning

The availability of appropriate datasets is crucial for training the LLM, fine-tuning the SVS, and evaluating the system.[1] A multi-pronged strategy is proposed:

• Leveraging Existing Datasets [1]:
  • SVS Model Training/Fine-tuning: Public SVS datasets like M4Singer [1, 81, 82], Opencpop [1, 83, 84], PopBuTFy, and VCTK.
  • Beat Tracking Module Evaluation: Standard beat tracking datasets (GTZAN, Hainsworth, SMC MIDI, etc. [1, 55]).
  • LLM Pre-training/Fine-tuning (General Musicality): Large-scale MIDI collections like the Lakh MIDI Dataset [1, 85, 86] or the GigaMIDI dataset.[1, 9, 54, 87] GigaMIDI is particularly relevant due to its heuristics (DNVR, DNODR, NOMML) for identifying expressive MIDI performances based on note velocity variations, onset deviations, and metric positions [9, 54, 87], which can inform LLM training for expressive timing.
• Creating/Annotating Specialized Datasets for Beat-Aware Expressive Singing [1]: Addressing Gap G5, a core effort will be creating a dataset containing sung vocal performances meticulously aligned with lyrics, phoneme/note timings, expressive vocal parameters (detailed pitch contours, dynamics, articulation), and high-resolution beat/tempo information.
  • Annotation Process: Use robust beat tracking (verified in Phase 1) on accompaniment, followed by manual verification (e.g., Sonic Visualiser, ELAN [1]). Employ ASR and forced alignment (e.g., Montreal Forced Aligner [1, 88]) for vocal annotation. Extract pitch contours using methods like PYin with smoothing (e.g., Smart-Median smoother [1, 89]). Crucially, precisely synchronize vocal and beat annotations.
  • Expressive Parameter Extraction: Develop methods to quantify expressive vocal parameters relative to beat structure (e.g., micro-timing deviations from metronomic grid, vocal intensity variations aligned with beat strength, articulation patterns).
• Data Augmentation Techniques [1]:
  • For SVS Robustness: Pitch augmentation, mix-up augmentation, cycle-consistent training (inspired by SingAug [1, 90]).
  • For LLM Rhythmic Training: Rhythmic variation synthesis (programmatically altering timing of existing performances), noise injection.
• Few-Shot Learning Approaches [1]:
  • For LLM Adaptation: Fine-tune large pre-trained music/language LLMs on a small set of high-quality beat-aware expressive singing examples.
  • For SVS Model Adaptation: Adapt SVS models known for few-shot capabilities (e.g., GPT-SoVITS [1, 1], MiniMax-Speech [1], or general FSL techniques [18, 25, 26, 44, 91]) using limited examples of specific rhythmic styles or beat-aligned expressivity.

The quality of expressive data is paramount for training the LLM to generate nuanced rhythmic expression (RQ5, O4). Simply measuring note onset deviations from a grid [92, 93, 94, 95, 96] may not capture the full essence of musical "feel" like swing or groove. Research in microrhythm indicates that expressiveness involves more than just onset timing, incorporating dynamic envelope and timbre.[97] Similarly, the analysis of dynamic accents in vocal performance reveals complex interactions of features like intensity, pitch, and spectral characteristics.[94, 98, 99, 100, 101, 102, 103, 104, 105] Therefore, the annotation and feature extraction process for the new specialized dataset must be sophisticated enough to capture these multifaceted expressive elements if the LLM is to learn to reproduce them convincingly. This may involve incorporating methodologies for analyzing micro-timing deviations [40, 41, 92, 93, 94, 95, 96] and dynamic accents [94, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111] from singing performances.