Releases: j20001970/GDMP
v0.5
GDMP v0.5 is here. All vision-related tasks are included in this release, as well as improvements of existing framework classes. Initial support for macOS and Web are also included. There are both newly introduced and deprecated classes in this release, please refer to the changelog below.
What's changed in this release
General
- godot-cpp update to v4.3
- MediaPipe update to v0.10.20
build.pynow build release libraries by default.
Framework improvements
MediaPipeProtonow handles nested set/get.MediaPipeGraphConfigis now the subclass ofMediaPipeProtoMediaPipePacketis more simplified.MediaPipeGraphreworked to match originalmediapipe::CalculatorGraphusage more.
New classes
- Two new classes
MediaPipeGraphBuilderandMediaPipeGraphNodeare introduced using MediaPipe'sapi2, which allows buildingMediaPipeGraphConfigdirectly from code. MediaPipeTaskRunneris the higher-level class for running calculator graphs, you can use it to run graphs that do not have task API yet (e.g. HolisticLandmarkerGraph).MediaPipeExternalFilesis the class autoload for downloading MediaPipe assets and models from Google Cloud Storage via Godot'sHTTPRequest, developers can use it to download additional model assets on demand instead of packaging them on export.
New vision tasks
The following new vision tasks and graphs has been added, marking the completion of all currently available vision tasks:
- HolisticLandmarkerGraph
- MediaPipeFaceStylizer
- MediaPipeInteractiveSegmenter
Deprecated classes
MediaPipeCameraHelper is considered deprecated since v0.5, and will be removed in future releases. 3.x (GDNative) versions are not affected by this change unless platforms support for CameraServer are backported to Godot 3.x in future. No new features will be developed for camera helper from now on.
Android
- CI now also build x86_64 architecture
- Camera permissions are removed from AAR manifest, you will need to enable camera permission upon exporting Godot projects if accessing cameras is required.
iOS
- CI now builds both x86_64 and arm64 architectures.
Linux
- CI build linux libraries with Freedesktop SDK.
- OpenCV is now built from source to avoid linking entire libraries.
macOS support
macOS CPU build has been added, available in both arm64 and x86_64 architectures, please help us test if it works!
Web support
GDMP for web support has been added via WebAssembly module. While the library can be loaded, most tasks are still not working as expected, and some tasks are not included at all due to OpenCV disabled. GDMP for web will be excluded from version release libraries until most tasks start working, you can still build web library with build.py if you would like to try anyway.
Q&A
Q: Audio/Text tasks when?
A: Audio and text tasks are planned to be added in next (v0.6) release.
Q: Why deprecate MediaPipeCameraHelper?
A:
If I had a nickel for every time I included a computer vision-related Godot plugin that also does camera I/O in the same project, forget the nickels because those plugins alone will be as heavy as node_modules first.
While the statement above is definitely for the joke, including camera I/O functionalities for GDMP does certainly increase project complexity, and hinder distribution due to libraries size, and more importantly, the lack of standardized ways to access camera frames on Godot Engine leads to the possibility of camera code duplication if multiple CV libraries are used. Therefore camera helper is planned to be removed in future releases, so that we can focus on providing MediaPipe framework and solutions to Godot.
Your next line is..."How do I access camera frames after camera helper is removed?"
Q: How do I access camera frames after camera helper is removed...?!
A: CameraServer-based approach is recommended as its usage is common across Godot Engine, while the platform support for CameraServer is incomplete (only macOS and Linux in v4.4), there has been effort to extend the support to more platforms via GDExtension.
CameraServerExtension is a Godot v4.4+ plugin (also made by me) that aim to bring CameraServer support to more platforms, while only Android and Linux is available at the time of writing, support for more platforms will come along the way. We'll try to cover all officially-supported platforms by the time MediaPipeCameraHelper is completely removed.
Q: Where is MediaPipeHolisticLandmarker?
A: As of MediaPipe v0.10.20, there are no official task API available for holistic landmarker, you can instead use MediaPipeGraphBuilder to create a calculator graph with mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph node, then use MediaPipeTaskRunner to run it. MediaPipeHolisticLandmarker will be added after MediaPipe getting the task API for holistic landmarker.
Q: Generative AI tasks (e.g. Image Generator, LLM Inference) when?
A: Generative AI tasks are planned after the support of audio and text tasks, it may or may not added in the same release of audio/text tasks.
Full Changelog: v0.4...v0.5
v0.4
It may be late, but it is never absent. GDMP v0.4, now available on Godot 4!
With Godot 4 major release, and GDNative superseded by GDExtension, GDMP has also completed the porting process to GDExtension. From now on, the master branch will track the development of GDMP for Godot 4.x, while 3.x branch keeps the original GDNative version, receiving feature backports where applicable. This release also completes most MediaPipe vision tasks that replace legacy pbtxt-based solutions.
What's changed in this release
General
- godot-cpp update to v4.2.2
- MediaPipe update to v0.10.13
- Setup process is now more streamlined (Thank @you-win for helping on windows-specific parts)
build.pyis now more configurable.- CI build workflow is created to help delivering pre-built libraries and testing more rapidly (Thank @Malcolmnixon for initial bringing up)
MediaPipeCameraHelper::set_graphis removed, instead aMediaPipeImageobject is generated for each camera frame, connectnew_framesignal to receive camera frames instead.- GDMP now register a custom resource provider to MediaPipe upon library initialization. That means when any calculators calling
mediapipe::GetResourceContentsto load resource such as models or assets, Godot's filesystem will be searched first, then fallback to platform-specific methods.
New classes
- The following vision tasks have been added:
MediaPipeFaceDetector
MediaPipeFaceLandmarker
MediaPipeGestureRecognizer
MediaPipeHandLandmarker
MediaPipeImageClassifier
MediaPipeImageEmbedder
MediaPipeImageSegmenter
MediaPipeObjectDetector
MediaPipePoseLandmarker - MediaPipeImage: an intermediate class between godot::Image and mediapipe::Image (which wraps ImageFrame and GpuBuffer), and is used in vision tasks primarily.
- MediaPipeProto: The number of proto classes to keep up with is too damn high! Therefore we introduced one class to rule them all, and should (probably) works for most cases.
Android
- Dependencies updated to match Godot v4.2.2
- GDMP for Android are now bundled as a Android Archive (aar) with native libraries.
iOS
- Building instruction for iOS have been updated, removing deprecated Tulsi method.
Li E4C6 nux
- Add script to enable Flatpak environment.
Misc
- We have added a icon for the project! But it isn't used anywhere for now, so I'll put it here.
Full Changelog: v0.3...v0.4
Contributors
Assets 3
v0.3
Behold, GDMP v0.2: Episode One!
This release mainly focus on development QoL changes, a couple of new classes are also added to improve the plugin's overall modularity. Enjoy!
What's changed in this release
General
- Old autoload-based methods have been removed.
- MediaPipe updated to v0.9.3.0
- Introducing Hedron's Bazel Compile Commands Extractor in mediapipe setup patch for generating a more complete
compile_commands.json. You can runextract_compile_commands.pyto do it. - Setup and build scripts improvements.
- Autoload
GDMPhas been renamed toMediaPipein order to approximate usage syntax in future GDExtension releases. - CameraHelper permission signals are now merged into one
permission_resultwith a boolean value indicating whether camera permission is granted. - Platform emoji is removed from release notes because it is not funny.
New Classes
-
GraphConfig: a Godot resource that represents mediapipe::CalculatorGraphConfig.
Graphs on Godot are now initialized with GraphConfig instead of graph config file path. -
GPUResources: represents mediapipe::GpuResources.
Graph, GPUHelper and CameraHelper on Godot are now required to set GPU resources manually to enable GPU support.
Android
- Gradle and AGP are synced with upstream Godot version.
- godot-lib detection is now more flexible.
- CameraHelper permission is now handled by godot::OS instead PermissionHelper from JNI.
Windows
- Use junction directory to symlink GDMP source to mediapipe workspace.
build.pywill now supplyingPYTHON_BIN_PATHenvironment variable to Bazel using current python interpreter, so you don't have to set it manually.
Godot 4 (GDExtension)
- GDMP for Godot 4 is still being worked on, you can try it on
4.0branch at the time of writing.
In Next Release
As you may already noticed, MediaPipe upstream is deprecating legacy solutions (i.e. the way we load a predefined graph config file to run for model inference pipeline using framework components) in favor of task API. While MediaPipe framework itself is not deprecated by any means, depending on deprecated solutions for real world applications will become more unreliable over time.
Therefore, in the next GDMP v0.2: Episode Two GDMP v0.4 release, the main focus will be bringing up several model task APIs from official MediaPipe solutions while maintain compatibility with existing framework classes, as well as official GDExtension support, GDMP for web platform is also planned. It may take some time, but we will get there.
Full Changelog: v0.2...v0.3
v0.2
GDMP v0.2 has been released. In addition to support for iOS and Windows platform, there are also quite a few minor improvements here and there, it ain't much but it's honest work.
What's changed in this release
General
- MediaPipe updated to v0.8.10.2
- Godot C++ bindings are now built in Bazel.
- Errors are now logged with Godot ERR_* macros for easier debugging.
- Add several methods to Graph for checking graph status.
- Graph config is now kept after graph stopped so that it can be reused.
- Add permission related methods and signals to CameraHelper.
- CameraHelper can mirror video frames on non-mobile platforms.
- GPUHelper now requires manual instantiation and initialization with existing graph instead of calling Graph.get_gpu_helper()
Android 🤖
- The build process of MediaPipe and GDMP aar have been merged.
- GDMP aar is now versioned.
iOS 🍎
- Initial iOS support has been added.
Linux 🐧
- Clangd compilation database has been added for easier development.
Windows 🪟
- Support for Windows has been added with GPU disabled.
For full changelog please refer to: v0.1...v0.2