Adobe’s experimental AI tool can edit entire videos using one frame

Adobe demonstrated some of the experimental AI tools it’s working on at its Max conference that provide new ways to intuitively edit photos, videos, and audio. These experiments, called “sneaks,” include tools that instantly apply any changes you make to one frame across an entire video, easily manipulate light in images, and correct mispronunciations in audio recordings.

Project Frame Forward is one of the more visually impressive sneaks, allowing video editors to add or remove anything from footage without using masks — a time-consuming process for selecting objects or people. Instead, Adobe’s demonstration shows Frame Forward identifying, selecting, and removing a woman in the first frame of a video, and then replacing her with a natural-looking background-similar to Photoshop tools like Context-aware Fill or Remove Background. This removal is automatically applied across the entire video in a few clicks.

Users can also insert objects into the video frame by drawing where they want to place it and describing what to add with AI prompts. These changes will similarly be applied across the whole video. The demonstration shows that these inserted objects can also be contextually aware, showing a generated puddle that reflects the movement of a cat that was already in the video.

Another tool is Project Light Touch, which uses generative AI to reshape light sources in photos. It can change the direction of lighting, make rooms look as if they were illuminated by lamps that weren’t switched on in the original image, and allows users to control the diffusion of light and shadow. It can also insert dynamic lighting that can be dragged across the editing canvas, bending light around and behind people and objects in real time, such as illuminating a pumpkin from within, and turning the surrounding environment from day to night. The color of these manipulated light sources can also be adjusted, letting you tweak warmth or create vibrant RGB-like effects.

Project Clean Take is a new editing tool that can change how speech is enunciated using AI prompts, removing the need to re-record video or audio clips. Users can change the delivery or emotion behind someone’s voice — making them sound happier or inquisitive, for example — or replace words entirely while preserving the identifying characteristics of the original speaker’s voice. It can also automatically separate background noises into individual sources so that users can selectively adjust or mute specific sounds, helping to preserve the overall audio while improving voice clarity.

These are just a handful of sneaks that were showcased at Adobe’s Max event. Other notable mentions include Project Surface Swap, which lets you instantly change the material or texture of objects and surfaces, Project Turn Style for editing objects in images by rotating them like a 3D image, and Project New Depths, which lets you edit photographs as if it were a 3D space that identifies when inserted objects should be partially obscured by the surrounding environment. You can read more about each sneak preview in detail over on Adobe’s blog.

Sneaks aren’t publicly available to use, and they’re not guaranteed to become official features in Adobe’s Creative Cloud software or Firefly apps. Many features, like Photoshop’s Distraction Removal and Harmonize tools, initially started out as sneaks projects, however, so there’s a good chance that some version of these experimental capabilities will be available to creatives in the future.