The Omaha project provides a shared autoupdate and install system for Windows client products at Google that works on multiple Windows platforms, including Windows Vista. Goals include the following:
Non-goals include the following:
Google client products have the capability to update themselves to a newer version without end-user intervention. This capability is called "autoupdating."
Although our client products shared the autoupdate feature, they did not all share autoupdate code. Some had implemented their own autoupdate solutions, and others had taken advantage of the common autoupdater code and the common autoupdate server.
It was obviously desirable for products to avoid unnecessary duplication of code and for Google to avoid maintaining multiple servers that perform identical functionality. This reason alone was enough to consider unifying all client products under a single autoupdate solution. In addition, the evolution of Microsoft Windows made this unification almost mandatory. Windows Vista features a strict security model that limits the ability of most applications on a machine to perform system-changing activities, such as modifying the Windows registry, writing to the Program Files directory, or in some cases writing any persistent change to the system at all. Updating a program installation requires all these capabilities; thus, on Vista, most traditional autoupdating will fail.
Another problem caused by lack of shared code is that each autoupdate mechanism had its own subset of desired features. Most did not have a way to do canary experiments. Most did not have a way to have multiple update tracks. Most did not define a consistent versioning mechanism. Some of these features were not optional as user base size increased. By unifying the code we could deploy a rich set of autoupdate capabilities to all client applications.
Also, as the number of Google applications increased, it became more and more desirable to improve the overall install user experience. The browser typically prompted the user with a long series of techy, confusing and scary dialogs all trying to convince the user not to install. Then the user was prompted with a wizard filled with choices that they did not need to or know how to decide amongst. These factors combined to form a bad user experience and large drop-off during the app installation process.
To address the goals of Vista compatibility, reduction of duplicative effort at Google, improvement of deploy and autoupdate capabilities, and application install user experience improvement, we proposed to develop shared client infrastructure that handles all installation and autoupdating tasks for Google Windows client products. This client communicates with a single Google autoupdate server. Taken together, this server and client is named Google Update. The project code name is Omaha.
| Feature |
|---|
| Solve pushing server content and binary changes at the same time |
| Allow our partners to add their own versions of their product to Omaha without our intervention or a server push on our end |
| Allow our partners to be able to test new versions integrated with Omaha without making the URL live to the rest of the world |
| Be able to offer different versions of the same product, including beta software at different URLs |
| Continue to support at least as good one-click install experience as we have now |
| Offer better non-admin/limited user account support, i.e. downloading and running the product over offering a link |
| Be able to install product updates without the product running |
| Optionally be able to automatically run the product after the installation completes |
| Be able to show no UI vs. a small progress bar - e.g. Earth install vs. Offline Gmail |
| Provide decent proxy support, i.e. reliable downloads outside the browser |
| Optionally don't require interactive installers/uninstallers for Omaha so product teams can rely on our translations for them. |
| Offer a small stub executable |
|
Make the Omaha UI super fast to load |
|
Omaha to support localization in the top X languages |
| Be able to save the installer for later use |
User downloads a meta-installer from a Google website, going through standard browser file download steps. Once the user figures out how to launch the meta-installer, the meta-installer installs and runs the Omaha client. On Windows platforms, such as Windows Vista, with UAC, an elevation prompt is displayed when installing applications per-machine. The client then begins downloading and installing the app referenced by the micro reference.
Google website detects that Omaha client is installed, and instead of delivering a downloadable EXE or MSI, it vends an application micro-reference via an ActiveX (or equivalent) object. In this case, the user didn't have to go through any complicated download steps. They just clicked on a web link and the download starts. The rest of this scenario is the same as the initial install scenario.
The Omaha client is a Windows application, installed by a custom Windows installer. It performs several functions:
The client is able to handle multiple downloads in parallel, multiple sequential installs, and it is able to resume or restart partial downloads.
The client is not a separate application in the sense that it can be uninstalled independently of Google applications. Instead, it is considered a required feature of applications. It will uninstall itself when the last Google app using it is uninstalled by the user.
In addition to specific reporting related to installation and update, Omaha includes a generic crash-reporting facility for Google applications. The Breakpad project's network library is used, and the client will handle local minidump storage and queuing of reports.
The Omaha client consists of two major parts:
The runtime has the following functionality:
The runtime functionality is divided between a couple of processes, most notable a core process and many worker process. There are a few reasons for the separation: robustness, running code with the least privilege, executing Google installers in the context of the logged in users as much as possible, and avoiding resource leaks and resource consumption in long-lived processes.
The execution model for the runtime code depends on how Omaha is actually installed. There are two main scenarios here:
The Omaha core runs in a local system process for the machine, and one process for each logged in user. Typically, there will be two core processes running. There is potential for an optimization here, as the local system process can create a process that runs as the logged in user when it needs to.
The machine Omaha core process is started at boot time by a system service that terminates soon after. There is a reason for not running the Omaha core in a service: a bug in XP prevents us from further updating the service if the service has crashed. More investigation will have to be done here, especially since having a service is quite nice and eliminates the need for another process. The user Omaha core process is started by the shell from the user's run key. To eliminate a single point of failure, there are fallbacks implemented using the Windows system scheduler to attempt to start the core process if for some reason it is not running. It should be desirable to measure the overall availability of the core process. We strive to be as available as the Windows OS is in terms of uptime of the Omaha core.
The Omaha core is highly reliable and lightweight:
The Omaha worker processes do most of the actual work:
In order for an application to receive updates, the application must register with Omaha. The application registration mechanism is based on Windows registry. There are primarily two update policies supported by Omaha, which directly affect how an Omaha aware application registers with the Omaha runtime:
In the case of per-machine updates, each application installed per-machine creates a key HKLM/Software/Google/Update/Clients/{GUID}. Under this key, the application creates the following values:
In the case of per-user updates, the only difference is the location to where the key is created. In this case, the key is HKCU/Software/Google/Update/Clients/{GUID}.
When the update loop kicks in, Omaha runs the per-machine updates as SYSTEM and per-user updates as the logged in user. This assumes that the user has an active session, in other words, updates for per-user installed applications will not run if the user has not logged in. As long as there is anything under these keys, Omaha keeps pinging for updates and updating these applications. It is possible that the application is dead and its registration with Omaha still active, for example, in the case where the user just deleted the application files. Unfortunately, Omaha does not detect such conditions and it is going to download and apply updates for it until the application cleanly unregisters.
Exposing registry keys and low level implementation details to applications has disadvantages. It breaks the encapsulation of Omaha. An argument can be made that such implementation details would be better to be abstracted out in a code library that the application code links with. However, there is a great advantage for the current registration mechanism. Registry keys can be created easily directly from the installer script for applications that are using off the shelf installer technology thus making the integration with Omaha easier.
The service is set to auto run when Omaha is installed per-machine. The SCM failover mechanism is used to restart the service in case of a crash. When Omaha is installed per-user, the goopdate worker process is started by the Windows shell from HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Run.
The client runtime runs periodic tasks, such as the update check request to the server to discover updates. These tasks are kicked off by a scheduler. The scheduler code runs inside the Omaha core process. The main requirement for the scheduler is to execute a worker process once in a while.
A lot of consideration was given to using the built in Windows scheduler instead of building our own. The windows task scheduler on Vista seems to be greatly improved. On XP and Windows 2003 there are some limitations to the windows task scheduler:
The runtime follows a few constraints regarding resource usage and timing.
Limiting resource use:
Timing:
The web browser control exposes a high level interface to Google web pages and client applications, enabling features such as one-click web installs and rich update clients such as Pack. It needs to limit access so that only authorized Google web pages and applications have access to the services offered by the client run time. The interface is defined by very coarse-granularity functions. Rather than "download file X from server Y" it would instead offer "download application id X from Google". In the event that the client had a security hole, it's important that the interface limit the amount of damage that can be done.
There are two major parts to crash reporting on Windows: generating the minidump and reporting it to the server. With Omaha, each application would still be responsible for catching exceptions and signaling Omaha to generate and transmit the minidump to Google servers. The minidump is created in a secure directory which Omaha can write to. It has code to try to minimize possible accidental self-DDOS as a result of frequent crashes while still reporting important data, perhaps determining when a duplicate minidump was generated, or reporting the number of crashes that have been generated but not reported.
The Omaha meta-installer is a simple wrapper that includes the client as well as information necessary to install one or more Google applications. It performs the following functions:
Our goal is that the minimal installer will be less than 172KB (which takes less than one minute to download over a 28.8kbps link at 10 wire bits per 8 data bits, representing what's likely the slowest modem still in operation today). The current size of the matainstaller is larger than that. We are looking for ways to keep the download size down.
The Google Update server is not part of the Omaha open source project. Providing updates for applications requires a server that implements the Omaha Server Protocol.
Omaha has two server components: autoupdate and download. This separation of server responsibilities is desirable because the requirements are different: downloading large binaries takes high bandwidth and can tolerate relatively high latency, whereas small transactions such as update pings may be much more frequent but require extremely low bandwidth per transaction. These differences allow Google to provision the two servers appropriately.
The autoupdate server provides the following functionality:
The download server's primary job is to host binaries. In order to support private betas and other restricted software updates, the download server can restrict access to certain downloads.
To the greatest extent possible, teams will be able to reconfigure their area of the update server without affecting other teams.