Introduce real-time edit preview with CompositionPlayer and add new video effects #139
Conversation
Added CompositionPlayer to provide real-time preview of the video edits being applied. Now, removing audio and rendering text overlays will be shown dynamically on the edit screen.
Introduced a new video effects that adjusts the colors of the video to create a warmer color tone.
Added a new SlShaderProgram effect that darkens the video frames towards the edges, creating a vignette effect. The inner radius of the vignette oscillates smoothly over time, making the effect dynamic.
This commit upgrades the Media3 library from version 1.4.1-rc01 to 1.6.1.
Added a new effect that applies artistic style transfer to video frames using a TensorFlow Lite model. The effect takes a style image as input and modifies the video frames to adopt the visual style of the provided image.
There was a problem hiding this comment.
Hello @shahdDaghash, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
This pull request introduces real-time video editing previews using CompositionPlayer and adds new video effects. It also upgrades the Media3 library to version 1.6.1. The new video effects include RGB adjustment, periodic vignette, and style transfer. The changes involve modifications to build configurations, addition of shader files for the vignette effect, updates to the PreloadManagerWrapper, and the implementation of new classes for the video effects.
Highlights
- Real-time Edit Preview: Integrates
CompositionPlayerfor real-time preview of video edits, providing immediate visual feedback. - New Video Effects: Adds RGB adjustment, periodic vignette, and style transfer effects to enhance video editing capabilities.
- Library Upgrade: Upgrades the Media3 library to version 1.6.1 to support the style transfer video effect and other improvements.
- Gradle Configuration: Updates
build.gradle.ktsto include new dependencies forlitertand to disableRestrictedApilint checks.
Changelog
Click here to see the changelog
- app/build.gradle.kts
- Added lintOptions to disable RestrictedApi checks.
- Added dependencies for litert, litert-gpu, and litert-support-api.
- Upgraded Media3 library to version 1.6.1.
- app/src/main/assets/fragment_shader_vignette_es2.glsl
- Added a new fragment shader for implementing the vignette effect.
- app/src/main/assets/vertex_shader_copy_es2.glsl
- Added a new vertex shader for copying the frame, used by the vignette effect.
- app/src/main/java/com/google/android/samples/socialite/ui/player/preloadmanager/PreloadManagerWrapper.kt
- Changed
STAGE_LOADED_TO_POSITION_MStoSTAGE_LOADED_FOR_DURATION_MSinPreloadStatusControlto align with Media3 API changes.
- Changed
- app/src/main/java/com/google/android/samples/socialite/ui/videoedit/PeriodicVignetteShaderProgram.kt
- Implemented a
GlShaderProgramthat periodically dims the frames to create a vignette effect.
- Implemented a
- app/src/main/java/com/google/android/samples/socialite/ui/videoedit/StyleTransferEffect.kt
- Implemented a
ByteBufferGlEffect.Processorto apply style transfer to video frames using TensorFlow Lite.
- Implemented a
- app/src/main/java/com/google/android/samples/socialite/ui/videoedit/VideoEditScreen.kt
- Modified
VideoEditScreento integrateCompositionPlayerfor real-time preview. - Added toggles for RGB adjustment, periodic vignette, and style transfer effects.
- Updated
VideoMessagePreviewto useCompositionPlayerfor previewing video edits. - Added
CenteredCircularProgressIndicatorto show processing state. - Added
VideoPreviewConfigdata class to manage video preview configurations.
- Modified
- app/src/main/java/com/google/android/samples/socialite/ui/videoedit/VideoEditScreenViewModel.kt
- Modified
applyVideoTransformationandprepareCompositionto include RGB adjustment, periodic vignette, and style transfer effects. - Added
buildVideoEffectsListto construct the list of video effects based on user selections.
- Modified
- app/src/main/res/values/strings.xml
- Added strings for RGB adjustment, periodic vignette, and style transfer effects.
- gradle/libs.versions.toml
- Added versions for litert, litert-gpu, and litert-support-api.
- Updated Media3 library version to 1.6.1.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customiz
8000
e Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
A shader's soft vignette,
A style transferred, unique,
Media made anew.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces real-time video editing preview and adds new video effects, which are valuable enhancements. The code appears well-structured, but it's crucial to verify the performance and compatibility of the new features across different devices and video formats. The upgrade to Media3 1.6.1 seems necessary for the style transfer effect, but it's important to ensure that this upgrade doesn't introduce any regressions. Remember that I am unable to approve this pull request, and that users should have others review and approve this code before merging.
Summary of Findings
- Real-time preview implementation: The introduction of CompositionPlayer for real-time preview is a significant improvement to the editing workflow. However, ensure that the implementation handles various video formats and resolutions efficiently to avoid performance issues.
- New video effects: The addition of new video effects like RGB adjustment, periodic vignette, and style transfer enhances the application's capabilities. Verify that these effects are well-integrated and do not introduce any compatibility issues or performance bottlenecks.
- Media3 library upgrade: Upgrading the Media3 library to version 1.6.1 is necessary for implementing the style transfer video effect. Ensure that the upgrade is seamless and does not introduce any regressions or breaking changes.
Merge Readiness
The pull request introduces significant new features and an upgrade to a core library. Before merging, it's essential to thoroughly test the new video effects, ensure compatibility across devices, and verify that the Media3 upgrade doesn't introduce regressions. Given the scope of the changes, I recommend addressing the review comments and conducting thorough testing before merging. I am unable to approve this pull request, and that users should have others review and approve this code before merging.
app/src/main/java/com/google/android/samples/socialite/ui/videoedit/VideoEditScreenViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/videoedit/VideoEditScreen.kt
Show resolved
Hide resolved
|
These additional effects are looking great! I just have a few notes.
|
|
Currently, Style Transfer effect can be only applied to SDR frames only. Since the camera will attempt to record an HDR video, tone-mapping using OpenGL is performed on the HDR video if style transfer effect is selected.
Added documentation for the changes that include adding CompoisitionPlayer for editing preview, and introducing the new effects.
To accommodate the new style transfer effect, HDR video now requires tone-mapping to SDR when this effect is applied. Since tone-mapping can only be performed on the final composition, the applyVideoTransformation method has been updated to call prepareComposition(), which now handles the necessary tone-mapping in both preview and export workflows.
Added download_model.gradle that is responsible for downloading the needed tflite files for the style transfer video effect.
Done!
Sorry about that, we needed to add tone-mapping to SDR when using this effect. I added the fix to for both preview and export operations. |
The ktlint check was failing due to a comment within the value arguments of the PeriodicVignetteShaderProgram constructor.
Integrates the CompositionPlayer to provide a real-time preview of video edits directly on the screen, significantly improving the editing workflow by allowing immediate visual feedback.
Additionally, it introduces new video effects including RGB adjustment, periodic vignette, and style transfer, and upgrades the Media3 library to version 1.6.1 to be able to implement the style transfer video effect.