Add effects to Camera screen. #137
Conversation
A new dropdown appears in the upper right of the Camera screen, allowing users to switch between no effect, a black and white filter, a green screen effect, and night mode.
There was a problem hiding this comment.
Hello @donovanfm, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
Summary of Changes
This pull request introduces camera effects to the Socialite application. A new dropdown menu has been added to the Camera screen, allowing users to select from various effects such as 'No Effect', 'Black and White', 'Green Screen', and 'Night Mode'. The implementation involves adding new dependencies, modifying the Camera composable to include the effect selection UI, and updating the CameraViewModel to handle the different effects. The Green Screen effect uses selfie segmentation to mask the background, while the Black and White effect uses Media3's RgbFilter. Concurrent camera usage is implemented for the green screen effect.
Highlights
- UI Enhancement: A dropdown menu is added to the Camera screen for selecting camera effects.
- New Dependencies: Added dependencies for camera effects, vision common, and selfie segmentation.
- Effect Implementation: Implemented 'Black and White', 'Green Screen', and 'Night Mode' effects using CameraX and Media3 libraries.
- Concurrent Camera: Implemented concurrent camera usage for the green screen effect.
Changelog
Click here to see the changelog
- app/build.gradle.kts
- Added dependencies for camera media3 effect, vision common, and selfie segmentation.
- Added dependency for camera effects.
- Updated AGP from 8.7.3 to 8.8.0
- Updated Gradle from 8.9 to 8.10.2
- app/src/main/java/com/google/android/samples/socialite/ui/camera/Camera.kt
- Added imports for icons and dropdown menu components.
- Added state variables for effect mode and menu expansion.
- Modified the Camera composable to include a dropdown menu for effect selection.
- Added a function to set the effect mode and restart the preview with the new effect.
- app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraProviderManager.kt
- Replaced
ProcessCameraProvider.getInstance(context).await()withProcessCameraProvider.awaitInstance(context)for camera provider retrieval.
- Replaced
- app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
- Added imports for camera effects, ML Kit components, and Media3 effects.
- Added logic to apply different camera effects based on the selected effect mode.
- Implemented green screen effect using selfie segmentation and overlay effect.
- Implemented black and white effect using Media3's RgbFilter.
- Implemented concurrent camera setup for green screen effect.
- Added SelfieSegmentationAnalyzer inner class to handle selfie segmentation and background removal.
- Added YuvToRgbConverter class to convert from YUV to RGB for the selfie segmentation.
- app/src/main/java/com/google/android/samples/socialite/ui/camera/YuvToRgbConverter.kt
- Added new file to convert from YUV to RGB for the selfie segmentation.
- app/src/main/java/com/google/android/samples/socialite/ui/player/preloadmanager/PreloadManagerWrapper.kt
- Replaced
STAGE_LOADED_TO_POSITION_MSwithSTAGE_LOADED_FOR_DURATION_MS.
- Replaced
- gradle/libs.versions.toml
- Updated versions for AGP, camera, and media3 libraries.
- Added versions for camera media effect, camera effects, vision common, and selfie segmentation.
- Added libraries for camera media3 effect, camera effects, vision common, and selfie segmentation.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
A filter's gentle touch,
Transforms the world we see so much,
From monochrome's grace,
To green screen's embrace,
A new reality, just for us.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
There was a problem hiding this comment.
Code Review
This pull request introduces camera effects, including black and white, green screen, and night mode, enhancing the camera screen's functionality. The implementation involves adding necessary dependencies, UI elements for effect selection, and integrating these effects into the camera preview. Overall, the changes seem well-structured and add significant value to the application.
Summary of Findings
- Unstable API Usage: The
@UnstableApiannotation is used, indicating reliance on potentially unstable features. While necessary, it's crucial to monitor these APIs for changes and ensure compatibility as the libraries evolve. - Green Screen Effect Implementation: The green screen effect implementation involves complex image analysis and manipulation. Ensuring its performance and reliability across different devices and conditions is essential.
- HDR Video Capture: The code checks for HDR video capture support but disables it when the green screen effect is active due to concurrent camera limitations. This trade-off should be clearly documented and considered for future improvements.
Merge Readiness
The pull request introduces significant new features and is generally well-structured. However, due to the use of @UnstableApi and the complexity of the green screen effect, thorough testing and monitoring are crucial. I am unable to directly approve this pull request, and recommend that others review and approve this code before merging. I recommend addressing the high severity issue before merging, and carefully consider the medium severity issues as well.
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/Camera.kt
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
This replaces the previous manual approach of having a manual YUV to RGB conversion class.
Conflicts arose between the Green Screen effect changes and the CompositionPlayer changes.
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
app/src/main/java/com/google/android/samples/socialite/ui/camera/CameraViewModel.kt
Outdated
Show resolved
Hide resolved
Also for now, disable HDR Video recording due to a dependency conflict. A future version of CameraX will resolve teh conflict, and then HDR Video can be re-enabled.
- Uses the raw size mask from selfie segmentation. This is generally much smaller than the image, and it can be scaled when drawn. - Stores the mask in an A8 image. This uses less memory and can be drawn with a ColorMatrixColorFilter to restore it to RGBA - Uses line buffer to read from the selfie segmentation mask buffer to keep memory localized while performing conversion from floats to bytes. - Reduces allocations by reusing buffers used for mask conversion These optimizations reduced the selfie segmentation mask logic from ~24ms to < 8ms on a Pixel 9. Further optimizations are possible, but will require more drastic changes to the drawing logic.
There was a problem hiding this comment.
Looks good and really fun features :)
I've also created donovanfm#1 which has some proposed optimizations that don't change the structure of the code too much.
|
|
||
| lateinit var mask: Bitmap | ||
| lateinit var bitmap: Bitmap | ||
| var isGreenScreenProcessing = false |
There was a problem hiding this comment.
It looks like this is only set to false. Maybe it isn't needed?
| .build() | ||
| val selfieSegmenter = Segmentation.getClient(options) | ||
|
|
||
| @ExperimentalGetImage |
There was a problem hiding this comment.
This should be @androidx.annotation.OptIn(ExperimentalGetImage::class)
Optimize selfie segmentation mask for green screen
A new dropdown appears in the upper right of the Camera screen, allowing users to switch between no effect, a black and white filter, a green screen effect, and night mode.