GoMFT is a web-based managed file transfer application built with Go, leveraging rclone for robust file transfer capabilities. It provides a user-friendly interface for configuring, scheduling, and monitoring file transfers across various storage providers.
Warning
This application is actively under development. As such, any aspect of the application—including configurations, data structures, and database fields—may change rapidly and without prior notice. Please review all release notes thoroughly before updating.
Dashboard showing active transfers |
Dashboard dark mode |
- Multiple Storage Support: Leverage rclone's extensive support for cloud storage providers:
- Google Drive
- Google Photos
- Amazon S3
- MinIO
- NextCloud
- WebDAV
- SFTP
- FTP
- SMB/CIFS shares
- Local filesystem
- And more via rclone
- Webhook Notifications: Receive real-time notifications of job events:
- Configurable webhook URLs
- HMAC-SHA256 authentication with secrets
- Custom HTTP headers
- Selectable events (job success, job failure)
- Detailed JSON payload with job information
- Multiple Notification Services: Get job status updates through various notification channels:
- Email notifications with configurable SMTP settings
- Webhooks with authentication for custom integrations
- Pushbullet notifications with optional device targeting
- Ntfy.sh (both public and self-hosted) for simple push notifications
- Gotify server integration for self-hosted notifications
- Pushover notifications with customizable sounds and priorities
- Configurable message templates for all notification types
- Event-based triggers (job start, completion, errors)
- Scheduled Transfers: Configure transfers using cron expressions with flexible scheduling options
- Transfer Monitoring: Real-time status updates and detailed transfer logs with bytes and files transferred statistics
- File Metadata Tracking: Complete history and status of all transferred files with detailed information:
- Process status (processed, archived, deleted)
- File size and hash information
- Advanced search and filtering capabilities
- Metadata retention for compliance and auditing
- Detailed file view with processing timestamps and job association
- Powerful filtering by status, filename, job, and date ranges
- Advanced search interface with multiple criteria
- Bulk management and record deletion capabilities
- Responsive design with mobile-friendly interface
- Multi-threaded File Transfers: Significantly improve performance with concurrent file processing:
- Configurable number of concurrent transfers (1-32) per job
- Automatic queue management to prevent system overload
- Independent configuration for each transfer job
- Optimized for both high-volume small files and large file transfers
- Maximizes bandwidth utilization for cloud storage providers
- Web Interface: User-friendly interface for managing transfers, built with Templ components
- File Pattern Matching: Support for file patterns to filter files during transfers
- File Output Patterns: Dynamic naming of destination files using patterns with date variables
- Archive Function: Option to archive transferred files for backup and compliance
- Transfer Configurations: Full control over source and destination connection parameters
- Job Management: Create, edit, and monitor transfer jobs with scheduling
- Security: Role-based access control with admin-managed user accounts and secure password management
- Authentication Providers: Flexible authentication options:
- Built-in email/password authentication
- Authentik integration for enterprise SSO
- OpenID Connect (OIDC) support for standard identity providers
- OAuth2 integration for popular providers (Google, GitHub, etc.)
- Multiple provider support with fallback options
- Automatic user provisioning from external providers
- Role mapping from external identity providers
- Password Recovery: Self-service password reset via email with secure token-based authentication
- User Profile Management: Personal settings including theme preferences
- Modern UI: Built with Templ, HTMX and Tailwind CSS for a responsive experience
- Docker Support: Easy deployment with Docker images and Docker Compose support
- Portable Deployment: Run on any platform that supports Docker or Go
- Go 1.21 or later
- rclone installed and configured
- SQLite 3
- Clone the repository:
git clone https://github.com/starfleetcptn/gomft.git
cd gomft
- Install dependencies:
go mod download
- Build the application:
go build -o gomft
GoMFT is available as a Docker image for quick and easy deployment.
- Pull the latest image from Docker Hub:
docker pull starfleetcptn/gomft:latest
- Run the container:
docker run -d \
--name gomft \
-p 8080:8080 \
-v /path/to/data:/app/data \
-v /path/to/backups:/app/backups \
starfleetcptn/gomft:latest
docker run -d \
--name gomft \
-p 8080:8080 \
-v /path/to/data:/app/data \
-v /path/to/backups:/app/backups \
-e PUID=$(id -u) \
-e PGID=$(id -g) \
starfleetcptn/gomft:latest
docker run -d \
--name gomft \
-p 8080:8080 \
-v /path/to/data:/app/data \
-v /path/to/backups:/app/backups \
-e PUID=1001 \
-e PGID=1001 \
starfleetcptn/gomft:latest
docker run -d \
--name gomft \
-p 8080:8080 \
-v /path/to/data:/app/data \
-v /path/to/backups:/app/backups \
-v /path/to/.env:/app/.env \
-e PUID=$(id -u) \
-e PGID=$(id -g) \
starfleetcptn/gomft:latest
- Access the web interface at
http://localhost:8080
For production deployments, you can use Docker Compose with environment variables:
version: '3'
services:
gomft:
image: starfleetcptn/gomft:latest
container_name: gomft
restart: unless-stopped
ports:
- "8080:8080"
volumes:
- ./data:/app/data
- ./backups:/app/backups
- ./.env:/app/.env
environment:
- PUID=1000
- PGID=1000
- TZ=UTC
- SERVER_ADDRESS=:8080
- DATA_DIR=/app/data
- BACKUP_DIR=/app/backups
- JWT_SECRET=change_this_to_a_secure_random_string
- BASE_URL=http://localhost:8080
- GOOGLE_CLIENT_ID=your_google_client_id
- GOOGLE_CLIENT_SECRET=your_google_client_secret
- TOTP_ENCRYPTION_KEY=your_32_byte_encryption_key_here
- EMAIL_ENABLED=true
- EMAIL_HOST=smtp.example.com
- EMAIL_PORT=587
- EMAIL_FROM_EMAIL=gomft@example.com
- EMAIL_FROM_NAME=GoMFT
- EMAIL_ENABLE_TLS=true
- EMAIL_REQUIRE_AUTH=true
- EMAIL_USERNAME=smtp_username
- EMAIL_PASSWORD=smtp_password
- LOGS_DIR=/app/data/logs
- LOG_MAX_SIZE=10
- LOG_MAX_BACKUPS=5
- LOG_MAX_AGE=30
- LOG_COMPRESS=true
- LOG_LEVEL=info
# The user directive is no longer needed when using PUID/PGID environment variables
Save this as docker-compose.yml
and run:
docker-compose up -d
For more information and available tags, visit the GoMFT Docker Hub page.
GoMFT uses an environment file located at .env
in the root directory of the application. On first run, a default configuration will be created:
# Basic configuration
SERVER_ADDRESS=:8080
DATA_DIR=/app/data
BACKUP_DIR=/app/backups
JWT_SECRET=change_this_to_a_secure_random_string
BASE_URL=http://localhost:8080
# Google OAuth configuration (optional, for built-in authentication)
GOOGLE_CLIENT_ID=your_google_client_id
GOOGLE_CLIENT_SECRET=your_google_client_secret
# Email configuration
EMAIL_ENABLED=true
EMAIL_HOST=smtp.example.com
EMAIL_PORT=587
EMAIL_FROM_EMAIL=gomft@example.com
EMAIL_FROM_NAME=GoMFT
EMAIL_REPLY_TO=
EMAIL_ENABLE_TLS=true
EMAIL_REQUIRE_AUTH=true
EMAIL_USERNAME=smtp_username
EMAIL_PASSWORD=smtp_password
# Two-Factor Authentication configuration
TOTP_ENCRYPTION_KEY=your_32_byte_encryption_key_here
# UserID and GroupID
PUID=1000
PGID=1000
-
SERVER_ADDRESS
: The address and port to run the server on -
DATA_DIR
: Directory for storing application data (database and configs) -
BACKUP_DIR
: Directory for storing database backups -
JWT_SECRET
: Secret key for JWT token generation -
BASE_URL
: Base URL for generating links in emails (e.g., password reset links) -
Google OAuth configuration for built-in authentication:
GOOGLE_CLIENT_ID
: Your Google OAuth client IDGOOGLE_CLIENT_SECRET
: Your Google OAuth client secret
-
Email configuration settings for system notifications and password resets:
EMAIL_ENABLED
: Set totrue
to enable email functionalityEMAIL_HOST
: SMTP server hostnameEMAIL_PORT
: SMTP server port (usually 587 for TLS, 465 for SSL, or 25 for non-secure)EMAIL_USERNAME
: Username for SMTP authenticationEMAIL_PASSWORD
: Password for SMTP authenticationEMAIL_FROM_EMAIL
: Email address used as senderEMAIL_FROM_NAME
: Name displayed as the senderEMAIL_REPLY_TO
: Optional reply-to email addressEMAIL_ENABLE_TLS
: Set totrue
to use TLS for secure email transmissionEMAIL_REQUIRE_AUTH
: Set totrue
to require authentication for SMTP connections, orfalse
for servers that don't need authentication
-
Two-Factor Authentication (2FA) configuration:
TOTP_ENCRYPTION_KEY
: Secret key used to encrypt/decrypt TOTP secrets (for 2FA)- Should be exactly 32 bytes (characters) for optimal security
- If not set, a default development key will be used (not secure for production)
- If shorter than 32 bytes, it will be automatically padded (less secure)
- If longer than 32 bytes, it will be truncated to 32 bytes
- Example:
TOTP_ENCRYPTION_KEY=abcdefghijklmnopqrstuvwxyz123456
GoMFT provides configurable logging with rotation support through the following environment variables:
LOGS_DIR
: Directory where log files are stored (default:./data/logs
)LOG_MAX_SIZE
: Maximum size in megabytes for each log file before rotation (default:10
)LOG_MAX_BACKUPS
: Number of old log files to retain (default:5
)LOG_MAX_AGE
: Maximum number of days to retain old log files (default:30
)LOG_COMPRESS
: Whether to compress rotated log files (default:true
)LOG_LEVEL
: Controls verbosity level of logging (values:error
,info
,debug
, default:info
)error
: Only show errors and critical issuesinfo
: Show errors and general operational information (default)debug
: Show all messages including detailed debugging information
Log files contain detailed information about file transfers, job execution, and system operations, which can be useful for troubleshooting and auditing.
- Start the server:
./gomft
-
Access the web interface at
http://localhost:8080
-
Log in with the default admin account:
- Email:
admin@example.com
- Password:
admin
- Important: Change this password immediately after first login
- Email:
-
Create transfer configurations:
- Navigate to "Transfer Configs" section
- Configure source and destination locations with connection details
- Set file patterns and archive options as needed
- Configure performance settings:
- Set "Concurrent Transfers" slider to optimize throughput
- Use higher values (8-16) for many small files or fast networks
- Use lower values (1-4) for large files or limited bandwidth
- Consider source/destination system capabilities when setting
-
Create jobs using your configurations:
- Navigate to "Jobs" section
- Select an existing transfer config
- Set up a schedule using cron expressions or run manually
- Enable/disable as needed
-
Monitor transfers:
- View active and completed transfers on the Dashboard
- Check detailed transfer history with performance metrics
- View job run details including any error messages
-
Configure webhook notifications:
- Enable webhooks in job settings to receive notifications
- Provide a valid webhook URL where notifications will be sent
- Optionally set a webhook secret for HMAC-SHA256 signature verification
- Configure custom HTTP headers in JSON format if needed
- Choose notification triggers (job success, job failure, or both)
- Test your webhook integration with manual job runs
-
Webhook Notifications:
- Webhook Integration: Send notifications to external systems when jobs complete
- Secure Authentication: HMAC-SHA256 signature for webhook verification
- Custom Headers: Add custom HTTP headers to webhook requests
- Flexible Configuration: Configure different webhooks for different jobs
- Event Selection: Choose to send notifications on success, failure, or both
- Detailed Payload: Rich JSON payload with complete job execution details
-
Multiple Notification Services:
- Pushbullet Integration: Send notifications to your devices through Pushbullet
- Device targeting support for specific device delivery
- Customizable title and message templates
- API key-based authentication
- Ntfy Integration: Use public ntfy.sh or self-hosted ntfy server
- Topic-based routing of notifications
- Priority levels for different job events
- Optional username/password authentication for private servers
- Customizable title and message templates
- Gotify Integration: Send notifications to self-hosted Gotify servers
- Application token-based authentication
- Priority levels (1-10) for different notification importance
- Customizable title and message templates
- Pushover Integration: Professional notification delivery service
- Application and user key authentication
- Device targeting for selective delivery
- Sound selection for different notification types
- Priority levels from lowest to emergency
- Customizable title and message templates
- Common Features:
- Variable substitution in notification templates
- Job data access in templates (status, files, bytes, times)
- Event-based filtering (job start, completion, errors)
- Success/failure tracking for diagnostic purposes
- Pushbullet Integration: Send notifications to your devices through Pushbullet
-
Manage file metadata:
- Navigate to the "Files" section to view all processed files
- Use filters to quickly find files by status, job ID, or filename
- Click on any file to view detailed metadata including timestamps, size, and hash
- Use the advanced search page for complex queries with multiple criteria
- Delete file metadata records when no longer needed
- View files associated with specific jobs by navigating from the job details
- Utilize admin tools (administrators only):
- Access the "Admin Tools" section from the navigation menu
- View system statistics and server information
- Create and manage database backups
- Browse and download system log files with the integrated log viewer
- Perform database maintenance and optimization tasks
- View webhook documentation and integration details
GoMFT uses a role-based access control system with flexible authentication options:
- Administrators: Can create and manage users, access all features
- Regular Users: Can manage transfers and view history
-
Built-in Authentication:
- Email/password login with secure password hashing
- JWT-based session management
- Password history tracking
- Account lockout protection
- Self-service password reset
-
External Authentication Providers:
-
Authentik Integration:
- Enterprise-grade SSO capabilities
- Automatic user provisioning
- Role synchronization
- Group mapping support
- Secure token exchange
-
OpenID Connect (OIDC):
- Standard-compliant identity provider support
- Automatic user creation and updates
- Role mapping from OIDC claims
- Multiple provider support
- Secure token validation
-
OAuth2 Providers:
- Google authentication
- GitHub integration
- Other OAuth2-compliant providers
- Custom provider configuration
- Automatic profile synchronization
-
-
Security Features:
- Secure password hashing with bcrypt
- JWT-based authentication with tokens
- Password history tracking prevents reuse
- Account lockout after failed attempts
- Two-factor authentication support
- Session management and timeout
- Secure token storage and handling
-
User Profile Management:
- Theme preferences (light/dark mode)
- Profile information updates
- Password change functionality
- Two-factor authentication setup
- External account linking
This implementation adds TOTP-based (Time-based One-Time Password) two-factor authentication support to the application, compatible with standard authenticator apps like Google Authenticator, Authy, and others.
- TOTP-based authentication (RFC 6238 compliant)
- QR code setup for easy enrollment
- Backup codes for account recovery
- Rate-limited verification attempts
- Secure secret storage
The following fields have been added to the users
table:
two_factor_secret
: Stores the TOTP secret keytwo_factor_enabled
: Boolean flag indicating if 2FA is enabledbackup_codes
: Stores recovery backup codes
- Navigate to
/profile/2fa/setup
- Scan the displayed QR code with your authenticator app
- Enter the verification code to confirm setup
- Save your backup codes in a secure location
- Enter email and password as usual
- If 2FA is enabled:
- Enter the 6-digit code from your authenticator app
- Alternatively, use a backup code if you can't access your authenticator
- The TOTP secrets are encrypted using AES-256-GCM
- You must set the
TOTP_ENCRYPTION_KEY
environment variable in production - This key should be 32 bytes (characters) long and kept confidential
- Changing this key after users have set up 2FA will invalidate their existing 2FA configurations
- For high-security deployments, store this key in a secure vault and inject it at runtime
-
Source/Destination Types:
- Google Drive
- Google Photos
- Local filesystem
- Amazon S3
- MinIO (S3-compatible storage)
- SFTP
- FTP
- SMB/CIFS shares
- And many more via rclone
-
Connection Options:
- Host/server addresses
- Authentication (username/password or key files)
- OAuth2 authentication for Google services
- Port configurations
- Cloud credentials (access keys, secret keys)
- Bucket and region settings
- Custom endpoints
- Custom rclone flags
-
Google Photos Specific Options:
- Read-only mode for safer operations
- Start year filter for historical photos
- Include/exclude archived media
- Album path configuration
- Built-in or custom OAuth authentication
-
Google Drive Specific Options:
- Folder ID for specific directory access
- Team/Shared Drive ID support
- Built-in or custom OAuth authentication
- Path-based navigation
-
File Options:
- File patterns for filtering (e.g.,
*.txt
,data_*.csv
) - Output patterns for dynamic naming
- Archive options for transferred files
- Skip already processed files to avoid duplicates
- Concurrent file transfers (configurable per job)
- File patterns for filtering (e.g.,
-
Performance Options:
- Multi-threaded File Transfers: Process multiple files simultaneously for higher throughput
- Configurable concurrency level (1-32 concurrent transfers)
- Per-job concurrency settings to optimize for different storage types
- Automatic transfer queue management to prevent overloading systems
- Adaptive processing based on source/destination capabilities
-
Schedule Options:
- Cron expressions for flexible scheduling
- Manual execution
- Enable/disable schedules
-
Notification Options:
- Email Notifications: Receive job status updates via email
- Webhook Notifications: Integration with external systems
- Pushbullet: Push notifications to your devices
- Ntfy: Simple push notifications via ntfy.sh
- Gotify: Self-hosted notification server integration
- Pushover: Professional notification service
- Configure event triggers (start, complete, error)
- Customize notification message templates
- Selective notification based on job status
GoMFT supports email notifications for various features:
- Password Reset: Users can request password reset links sent to their registered email
- Styled Emails: Professional HTML emails that match the application's design theme
- Secure Tokens: One-time use secure tokens with 15-minute expiration for enhanced security
- Flexible Configuration: Easily configure your SMTP server settings
- Authentication Options: Support for both authenticated and unauthenticated SMTP servers
- TLS Support: Secure communication with your SMTP server
- Development Mode: When emails are disabled, reset links are logged to the console
To configure email functionality:
- Edit the
.env
file and provide your SMTP server details - Set
EMAIL_ENABLED=true
in the email configuration section - Ensure the
BASE_URL
setting is configured correctly for your deployment
GoMFT can send webhook notifications to external systems when jobs complete. This allows integration with monitoring tools, chat applications, custom notification systems, or workflow automation platforms.
Webhook notifications are sent as HTTP POST requests with a JSON payload containing detailed information about the job execution:
{
"event_type": "job_execution",
"job_id": 123,
"job_name": "Daily Backup",
"config_id": 456,
"config_name": "S3 to Local Backup",
"status": "completed",
"start_time": "2023-07-14T15:30:00Z",
"end_time": "2023-07-14T15:35:42Z",
"duration_seconds": 342,
"bytes_transferred": 1048576,
"files_transferred": 25,
"history_id": 789,
"source": {
"type": "s3",
"path": "my-bucket/data"
},
"destination": {
"type": "local",
"path": "/backups/data"
}
}
For failed transfers, additional error information is included:
{
"status": "failed",
"error_message": "Permission denied accessing destination path"
}
When a webhook secret is configured, GoMFT signs the payload using HMAC-SHA256 and includes the signature in the X-Hub-Signature-256
header. To verify the webhook:
- Compute the HMAC-SHA256 of the raw request body using your shared secret
- Compare it with the value in the
X-Hub-Signature-256
header - Process the webhook only if the signatures match
This ensures that webhook requests are authentic and haven't been tampered with.
GoMFT provides a comprehensive set of administrative tools for system management and monitoring:
The Admin Tools panel includes an integrated log viewer with the following features:
- Log File Browser: View a list of all available log files in the system
- Real-time Log Viewing: View log file contents directly in the web interface
- Refresh Function: Update the log list and content with the latest information
- User-friendly Interface: Clean, readable presentation with custom scrolling
- Dark Mode Support: Consistent theming with the rest of the application
- Navigation: Easily switch between different log files
This log viewer allows administrators to:
- Monitor system activity and diagnose issues without requiring server access
- View application logs, scheduler logs, and transfer logs in one place
- Track down errors and warning messages in real-time
The Admin Tools interface also includes database management capabilities:
- Create and manage database backups
- Restore from previous backups
- Download backups for safekeeping
- View system statistics
- Optimize the database with maintenance tools
.
├── components/ # Templ components for UI
├── internal/
│ ├── api/ # REST API handlers
│ ├── auth/ # Authentication/authorization
│ ├── config/ # Configuration management
│ ├── db/ # Database models and operations
│ ├── email/ # Email service for notifications and password resets
│ ├── scheduler/ # Job scheduling and execution
│ └── web/ # Web interface handlers
├── static/ # Static assets
│ ├── css/
│ └── js/
└── main.go # Application entry point
- Backend: Go with Gin web framework
- Frontend: Templ for Go HTML components
- UI Enhancement: HTMX for dynamic interactions
- Styling: Tailwind CSS
- Authentication: JWT (JSON Web Tokens)
- Database: GORM with SQLite
- File Transfer: rclone
- Deployment: Docker containerization and traditional installation
- Install development dependencies:
go install github.com/cosmtrek/air@latest # Hot reload for development
go install github.com/a-h/templ/cmd/templ@latest # Templ template compiler
- Generate template code:
templ generate
- Run in development mode:
air
- Fork the repository
- Create a feature branch
- Commit your changes
- Push to the branch
- Create a Pull Request
GoMFT uses the following directory structure:
/app/data
: Main application data directory- Contains the SQLite database (
gomft.db
) - Contains rclone configurations in
/app/data/configs
- Contains log files in
/app/data/logs
- Contains the SQLite database (
/app/backups
: Database backup directory
When using Docker, you should mount volumes to these locations:
volumes:
- /host/path/data:/app/data # For all application data
- /host/path/backups:/app/backups # For database backups
These paths can be customized using the environment variables DATA_DIR
, BACKUP_DIR
, and LOGS_DIR
.
By default, Docker containers run as the root user, which can pose security risks. GoMFT supports running as a non-root user, which is recommended for production environments.
- Improved Security: Limits the potential damage if the container is compromised
- Better File Permissions: Files created by the container will match your host user permissions
- Compliance: Many security policies and best practices require containers to run as non-root
-
Using PUID/PGID environment variables (recommended):
# Using current user's ID docker run -e PUID=$(id -u) -e PGID=$(id -g) starfleetcptn/gomft:latest # Or in docker-compose.yml environment: - PUID=1000 - PGID=1000
This is the most flexible method as it allows changing the user at runtime without rebuilding the image.
-
Using the
--user
flag with Docker run:docker run --user $(id -u):$(id -g) starfleetcptn/gomft:latest
-
Using Docker Compose with environment variables for
user
directive:services: gomft: image: starfleetcptn/gomft:latest user: "${UID:-1000}:${GID:-1000}"
-
Building a custom image with specified UID/GID:
services: gomft: build: context: . args: UID: ${UID:-1000} GID: ${GID:-1000}
Variable | Description | Default |
---|---|---|
PUID |
User ID to run as | Built-in user ID (1000) |
PGID |
Group ID to run as | Built-in group ID (1000) |
USERNAME |
Username to use | gomft |
These environment variables allow you to change the user/group IDs at runtime without rebuilding the image.
When mounting volumes, ensure that the directories on the host have appropriate permissions for the container user:
# Create directories with correct ownership
mkdir -p data backups
chown -R $(id -u):$(id -g) data backups
# Or adjust permissions to allow the container user to write
mkdir -p data backups
chmod -R 777 data backups # Less secure, but easier for testing
MIT License - see the full license terms
The GoMFT logo is licensed under the Creative Commons Attribution 4.0 International Public License.
The gopher design is from https://github.com/egonelbre/gophers.
The original Go gopher was designed by Renee French (http://reneefrench.blogspot.com/).