A powerful command-line tool for macOS UI automation via accessibility APIs, enabling automated UI testing, scripting, and system control.
macOS UI CLI provides a comprehensive command-line interface to interact with macOS applications through the accessibility API. Built by Claude, an AI assistant from Anthropic, you should absolutely not trust it.
It probably doesn't work
The tool allows you to:
- Discover applications, windows, and UI elements
- Interact with UI elements (buttons, text fields, etc.)
- Manipulate windows (move, resize, focus, etc.)
- Simulate keyboard input
- Perform automated UI workflows
- Robust Error Handling: Detailed error messages with recovery suggestions
- Timeout & Retry Mechanisms: Reliable automation of flaky UI operations
- Flexible Output Formats: Support for plain text, JSON, and XML output
- Easy Discoverability: Find applications and UI elements by name, PID, or path
- Comprehensive Documentation: Detailed help and examples for all commands
- macOS 11.0 or later
- Xcode 13.0 or later (for development)
- Swift 5.5 or later
-
Clone the repository with submodules:
git clone --recurse-submodules https://github.com/yourusername/macos-ui-cli.git cd macos-ui-cliIf you already cloned the repository without
--recurse-submodules, run:git submodule update --init -
Set up the Haxcessibility library:
./setup-haxcessibility.shThis creates necessary symbolic links to make the Haxcessibility submodule work with Swift Package Manager without modifying its source files.
-
Build the project:
swift build -
Create the app wrapper for accessibility permissions:
./.build/debug/macos-ui-cli permissions --create-wrapper -
Run the executable through the app wrapper:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli -
Grant accessibility permissions when prompted by macOS
USAGE: macos-ui-cli [--version] [--help] [--format FORMAT] [--verbosity LEVEL] [--color] [<subcommand>]
OPTIONS:
--version Show the version.
-h, --help Show help information.
--format FORMAT Output format: text, json, or xml (default: text)
--verbosity LEVEL Verbosity level 0-3 (default: 1)
--color Enable colorized output
SUBCOMMANDS:
discover Commands for discovering UI elements
apps List and find applications
windows List and find windows
elements Find, inspect, and interact with UI elements
interact Commands for interacting with UI elements
button Interact with buttons
text Read from or write to text fields
window Manipulate windows (move, resize, focus, etc.)
keyboard Simulate keyboard input
util Utility commands for the tool
permissions Check and request accessibility permissions
version Show detailed version information
info Display system information
format Configure output formatting
# Original commands for backward compatibility
permissions Check and request accessibility permissions
apps List and find applications
windows List and find windows
elements Find, inspect, and interact with UI elements
button Interact with buttons
text Read from or write to text fields
window Manipulate windows (move, resize, focus, etc.)
keyboard Simulate keyboard input
The examples below assume you've created and granted permissions to the app wrapper. Replace /Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli with your path if different.
To see the version information:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --version
To display help information:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --help
For convenience, you can create an alias in your shell profile:
alias macos-ui-cli="/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli"
Check current accessibility permissions status:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli permissions
Request accessibility permissions (shows prompt to user):
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli permissions --request
Open System Settings to Accessibility settings:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli permissions --open
Get information about the currently focused application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli apps --focused
Find an application by name:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli apps --name "Calculator"
Find an application by PID:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli apps --pid 12345
List all accessible applications:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli apps
List windows of the focused application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli windows
List windows of a specific application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli windows --app "Calculator"
List windows of an application by PID:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli windows --pid 12345
Find the currently focused UI element:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli elements --focused
Find all buttons in the focused application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli elements --role button
Find elements with a specific title:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli elements --title "OK"
Find elements by path:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli elements --path "window[Calculator]/button[=]"
Search within a specific application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli elements --app "Calculator" --role button
Press a button in an application:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli button --app "Calculator" --title "=" --press
Read text from a text field:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli text --app "Notes" --field "Text Area" --read
Write text to a text field:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli text --app "Notes" --field "Text Area" --value "Hello, World!"
Manipulate a window (move and resize):
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli window --app "Calculator" --position "100,100" --size "400,300"
Focus a window:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli window --app "Calculator" --focus
Minimize a window:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli window --app "Calculator" --minimize
Toggle fullscreen mode:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli window --app "Calculator" --fullscreen
Simulate keyboard input:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli keyboard --text "Hello, World!"
Simulate key combinations:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli keyboard --combo "cmd+c"
-
Clone the repository with submodules:
git clone --recurse-submodules https://github.com/yourusername/macos-ui-cli.git cd macos-ui-cliIf you already cloned the repository without
--recurse-submodules, run:git submodule update --init -
Set up the Haxcessibility library:
./setup-haxcessibility.sh -
Open the package in Xcode (optional):
open Package.swift
To run the test suite:
swift test
This tool requires macOS Accessibility permissions to function properly. Command-line tools cannot directly receive accessibility permissions in macOS, so you must create and use an app wrapper.
To create the app wrapper:
macos-ui-cli permissions --create-wrapper
This creates a macOS application bundle at /Applications/macos-ui-cli.app that contains your command-line tool. You must then run the tool through this wrapper:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli
After running the wrapped app for the first time, macOS will prompt you to grant accessibility permissions:
- System Settings (or System Preferences) will open to the Accessibility section
- Click the lock icon to make changes (if needed)
- Check the box next to "macos-ui-cli"
- Close System Settings
You can check your current permission status:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli permissions
To manually open the accessibility settings:
macos-ui-cli permissions --open
macOS restricts accessibility API access for security reasons. Pure command-line tools cannot appear in the Accessibility permissions list in System Settings because:
- They don't have proper application bundles with Info.plist files
- They don't have the required
NSAccessibilityUsageDescriptionentry - They aren't launched through Launch Services which tracks permission grants
The app wrapper technique solves these issues by:
- Creating a proper application bundle structure
- Including the necessary Info.plist entries
- Providing a launchable application that can appear in System Settings
Without this wrapper, the accessibility API calls will always fail, even if you try to grant permissions manually.
The tool supports different output formats to match your needs:
Test different output formats:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli util format --format json --sample apps
Configure output format for any command:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --format json apps
Adjust verbosity level (0-3):
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --verbosity 2 elements --app "Calculator"
Enable colorized output:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --color apps
-
Text (default): Human-readable text output
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli apps -
JSON: Structured JSON for machine processing
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --format json apps -
XML: XML format for integration with XML-based tools
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --format xml apps
- 0 (minimal): Basic information only
- 1 (normal): Standard level of detail (default)
- 2 (detailed): Additional details and attributes
- 3 (debug): All available information
Example with detailed verbosity:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli --verbosity 2 elements --focused
Visualize element hierarchies in different formats:
/Applications/macos-ui-cli.app/Contents/MacOS/macos-ui-cli util format --sample hierarchy
The tool implements comprehensive error handling with detailed error information, error codes, and recovery suggestions.
- Accessibility Errors: Issues with accessibility permissions or API usage
- UI Element Errors: Problems finding or interacting with UI elements
- Operation Errors: Timeouts, failures, or unsupported operations
- Application Manager Errors: Issues accessing or controlling applications
- Window Errors: Problems with window management
- Input Errors: Keyboard or mouse input failures
- Validation Errors: Command argument validation issues
Error messages include recovery suggestions to help resolve issues:
Error: Permission denied for accessibility access
Error Code: 101
Recovery Suggestion: Grant permission for this application in System Preferences → Security & Privacy → Privacy → Accessibility
UI operations can be flaky due to application responsiveness issues. The tool implements:
- Timeouts: Prevent operations from hanging indefinitely
- Automatic Retry: Retry flaky operations with configurable attempts and delay
- Combined Approach: Both timeout and retry for maximum reliability
UI operations use these mechanisms automatically, but you can adjust them in scripting mode:
# Example of timeout and retry configuration in a future scripting mode
macos-ui-cli script --execute "
set timeout 5.0
set retry-count 3
set retry-delay 1.0
press button 'OK' in 'Dialog'
"
- Accessibility permission management with app wrapper support
- Application discovery and access
- PID and name-based application lookup
- Application property inspection
- Window enumeration and inspection
- UI element discovery and traversal
- Element property inspection
- Element filtering by role or title
- Path-based element lookup
- Button pressing and text field interaction
- Window manipulation (resize, move, focus, minimize, fullscreen)
- Element action discovery and execution
- Keyboard input simulation
- Comprehensive error handling with recovery suggestions
- Input validation for commands and arguments
- Timeout and retry mechanisms for UI operations
- Flexible output formatting (plain text, JSON, XML)
- Adjustable verbosity levels
- Colorized output
- Element hierarchy visualization
- Scripting support
- Recording and replaying UI interactions
- Advanced element selection strategies
- Interactive mode
- Integration with common development workflows
This project is licensed under the MIT License - see the LICENSE file for details.
MIT License
Copyright (c) 2023-2025 Anthropic, PBC
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
This project was created by Claude, an AI assistant from Anthropic. It demonstrates how AI can assist in software development tasks including:
- Architecture design
- Code implementation
- Error handling
- Test creation
- Documentation
The project was developed as a demonstration of Claude's coding capabilities and is released under the MIT license to allow others to learn from, modify, and extend it.
- Haxcessibility - Numist's port of the Objective-C library providing access to macOS accessibility APIs, integrated as an unmodified git submodule
- Swift Argument Parser - Command-line interface parsing framework
This project uses Haxcessibility as an unmodified git submodule in the vendor/Haxcessibility directory. We've taken special care to integrate it without modifying any of its source files:
-
Why a submodule? Using a git submodule allows us to track the exact version of Haxcessibility we're using, make it easy to update to newer versions, and respect the original codebase without modifications.
-
Symbolic Link Approach: The
setup-haxcessibility.shscript creates symbolic links to handle the angle bracket imports (#import <Haxcessibility/Header.h>) without modifying the original source files. This is why you'll see untracked files in the submodule after running the setup script - these are generated files needed by our build process but not part of the original repository. -
Build Configuration: The Package.swift file is configured to build directly from the vendor directory, with proper header search paths to make the imports work correctly.
This approach ensures that we can:
- Use the original, unmodified Haxcessibility code
- Easily update from upstream when needed
- Maintain a clean separation between our code and the library code