Synthesizer V Studio 1.10.0b1 Update: Enhancing Pitch Generation with User Feedback

Dreamtonics releases Synthesizer V Studio 1.10.0b1 update.

This beta update incorporates essential improvements derived from our latest research progress — the application of Reinforcement Learning from Human Feedback (RLHF) for singing voice synthesis. RLHF has been proven to significantly enhance pitch generation while providing a mechanism to integrate user feedback back into the voice databases.

Specifically, this beta release includes updates to the voice databases and introduces new features in Synthesizer V Studio Pro that enable users to provide feedback on voice quality to Dreamtonics, with their consent. Models trained with this feedback data will be incorporated in future updates to the voice databases.

Improving Pitch Generation with RLHF

AIs can be trained to emulate a singer’s characteristics as observed in the training data. However, capturing data for AI training can be challenging. Conventional algorithms learn both the strengths and weaknesses from the recordings, and in some cases, this may result in the generation of less desirable singing voices.

Our latest update to Synthesizer V Studio addresses this issue by utilizing Reinforcement Learning from Human Feedback, a method previously employed in text generation AIs to steer the system towards generating more preferred responses. With RLHF, we can “instruct” Synthesizer V AI to generate pitch curves that users find appealing. This often translates to fewer off-key errors, better use of vibratos, and more expressive transitions between notes.

A new “Enhancement” slider, available in the AI Retakes panel after updating voice databases to the beta versions, controls the level of changes introduced by RLHF. By default, it is fully enabled to optimize listener preference. The Enhancement function can also be turned off for generating more diverse outputs, which can be beneficial when working with AI Retakes.

Submitting Feedback via AI Retakes

Beta voice databases in this update have been trained using feedback data collected from a month of rigorous internal testing. While the current beta version already features enhanced pitch generation with RLHF, we anticipate even better results with more comprehensive feedback covering a vast range of music contexts and a wider user base.

In our continued efforts to improve pitch generation and extend RLHF to the modeling of voice timbre, we are enabling feedback submission for all Synthesizer V Studio Pro users. Within the AI Retakes panel, the heart icon on each take, initially introduced as a bookmarking feature, now also serves as a button for users to share their preferred takes with Dreamtonics. Models trained with this feedback data will be deployed in future voice database updates.

This feedback submission feature requires user consent. We wish to clarify that alongside the take information, a snippet of the project data surrounding the selected note will also be shared. This data remains confined within Dreamtonics, strictly for research and development. If a user opts out of feedback collection, the heart icon will return to its original function as a bookmark for takes, and no project data will be transmitted even when the icon is clicked.

Usability Improvements

The Synthesizer V Studio 1.10.0b1 update incorporates several usability enhancements. A track color selector is now integrated into the right-click menu on track headers, providing an efficient method for track differentiation and organization.

Additionally, based on user feedback, we’ve added a menu command that enables the shifting of lyrics for selected notes. This feature facilitates quick adjustments following the use of the ‘Insert Lyrics…’ dialog.

Project Compatibility Notice

Given the design alterations in AI Retakes feature, project files saved using version 1.10.0b1 and subsequent versions will not be compatible with prior versions of Synthesizer V Studio (1.9.0 and earlier). For user convenience, an option has been introduced in the “File” menu, allowing projects to be saved in a format compatible with Synthesizer V Studio 1.9.0.

How to Access the New Release

Separate beta versions will be released for both Synthesizer V Studio and the voice databases.

Synthesizer V Studio 1.10.0b1

1.10.0b1 is a beta version. Beta releases of Synthesizer V Studio are offered before the final release for early preview of new features. Before installing and using a beta version, please understand that:

• The beta version is still a work in progress.
• The beta version may not function properly as described.
• Unexpected behaviors may happen while using the beta version.

You can access the beta versions for Synthesizer V Studio through the download page.

Voices Updates for Synthesizer V Studio 1.10.0b1

Given the extensive changes in this beta release, we are initially offering beta updates for the following products:

You can access the beta versions for these voice databases through the download page.

See below for detailed release notes.

Synthesizer V Studio 1.10.0b1 (Aug 2, 2023)

 
New Features

• Auto-Process: added support for RLHF-enhanced pitch models with the addition of “Enhancement” parameter on pitch takes (Voice Database update required).
• Arrangement: added a track color selector to the right-click menu on track headers.
• Piano Roll: added a menu item for shifting forward/backward the lyrics of the selected notes.
• AI Retakes Panel: users may opt-in retake feedback collection that sends the “liked” takes along with surrounding notes and other project data to Dreamtonics for improving the sound quality.

 
Enhancements

• Synthesizer V Engine: improved the generation of phoneme duration and diversified the effect of timbre retakes (Voice Database update required).
• AI Retakes Panel: redesigned the panel to always display the global (track or group-level) values for Expressiveness and Enhancement.
• Plugin: AU and VST3 plugins now support drag-and-drop importing of MIDI files.
• Misc: improved activation server connectivity from Mainland China.

 
Behavior Changes

• AI Retakes Panel: Expressiveness and Enhancement sliders in the take list now display absolute values (after summing with the global Expressiveness/Enhancement) instead of the offset.
• File: disabled forward-compatibility for projects saved in Synthesizer V Studio 1.10.0 and onwards due to a breaking change in the value range of take Expressiveness. Users may use “File/Save As (1.9.0-Compatible)…” to export the project to an earlier version of Synthesizer V Studio.

 
Bug Fixes

• Synthesizer V Engine: fixed crashes and quality degradation on standard voice databases due to some dictionary changes introduced in the last update.
• GUI: fixed missing fonts on some keyboard shortcuts on macOS.
• Live Render: fixed a bug causing the renderer to skip some parts of a project when loaded immediately after startup.