Genie 4: Real-time world simulation, audio and dialogue generation

1. Worlds that generate scenes, sound and dialogue together

Genie 4 goes beyond visuals, generating ambient sound and character speech alongside scenes. It adds sounds suited to each setting, such as wind in a forest or waves on a beach. Characters carry on conversations in response to user input or events in the world. Google DeepMind describes it as a world model with native real-time audio generation.

Users control movement and the camera with a standard keyboard and direct scene changes through text. They can change the weather or introduce people and objects. The model manages the flow so scenes keep unfolding between inputs. One published prompt reads:

Prompt: A macro close up of a maple seed falling from the tree into the grass

According to Google DeepMind, Gemini Omni uses the real world's appearance and behavior as a reference for simulations. Other Gemini models contribute reasoning to maintain scene consistency and continuity. The company highlights real-time control, realistic physics and human movement, sound and dialogue, and simulations of events unfolding across multiple stages.

2. Controlling physics and human movement in real time

The company says the model handles physical situations such as collisions and aerodynamics, the study of airflow and forces. It also depicts natural human movement that responds to the environment. Users can move the camera or add people and objects while viewing a scene. Published examples include a crowded Japanese train station in the 1980s and a maple seed falling from a tree.

For complex goals, Gemini models serve as control agents, software that carries out goals on the user's behalf. The company says Gemini 3.8 Flash observes the scene being generated, breaks the goal into steps and checks incoming frames to track progress at each step. It revises its plan in real time when users intervene or the world changes.

In robotics experiments, Gemini 4 Argon breaks user task instructions into steps and passes them to Genie 4 as text descriptions. Google DeepMind showed a simulation of Apptronik's Apollo humanoid, a robot with a humanlike form, playing checkers. This aligns with efforts to build environments where robots and agents can practice varied situations before real-world deployment.

3. Speeding up early creative and development trials

Trusted testers include participants from creative work, journalism, education, gaming and engineering. Filmmakers can experiment with camera angles, lighting, dialogue and movement. Photojournalists are testing ways to expand documentary photographs into environmental simulations. Game developers can turn ideas into explorable prototypes before writing engine code, the programs underpinning a game's operation. Corporate partners are exploring potential uses in scenario generation.

This approach can reduce scene creation and revision cycles, turning ideas into simulations people can experience. However, visual details and object persistence may change during longer simulations or repeated, complex prompt revisions. Complex physical interactions, real-time audio accuracy and legible text remain research areas. So do spatial accuracy, including precise positions and distances, and long-term consistency, or how well scenes hold up over time.

4. A preview starting with trusted testers

Google DeepMind released Genie 4 as a research preview for trusted testers and plans to gather feedback before expanding access to consumers and developers. It named Google AI Ultra subscribers in Project Genie and early-access partners using the Gemini API and Google AI Studio as the next groups to gain access. It also runs a Project Genie waiting list.

The company says it conducted internal safety evaluations in advance and applied safeguards aligned with its safety policies. All generated video and audio carry SynthID watermarks, identifying marks that help recognize AI-generated content and establish its provenance. For future development, it outlined hybrid world models combining logical reasoning from language models and code with the sensory representation of pixel-level simulation.

Genie 4's core value is letting users test visuals, sound and interaction together, while improving consistency in long-running simulations remains the next challenge.