8000 Data and Backups · overleaf/overleaf Wiki · GitHub
[go: up one dir, main page]
More Web Proxy on the site http://driver.im/
Skip to content

Data and Backups

Jump to bottom
Miguel Serrano edited this page Apr 7, 2025 · 14 revisions

Data Storage

Overleaf Community Edition and Overleaf Server Pro store their data in 3 separate places:

  • Mongo Database: user and project data lives here.
  • Redis: used as high-performant cache for in-flight data, it mainly stores data from project edition and collaboration.
  • Filesystem: stores project files (such as images), stores full project history data and it's also used as a temporary disk cache during project compilations.
    • Note: This might be ~/sharelatex_data or ~/overleaf_data, depending on when you're instance was setup.
    • Note: For project files and full project history data we also support S3 compatible storage backends.

See section "Folders in detail" for details on the folder layout on disk.

Performing a backup

There are 3 things which need to be backed up

  • Mongodb
  • Overleaf Filesystem data
  • Redis

In order to produce a consistent backup it is mandatory to stop users from producing new data while the backup process is running.

We advise setting up a maintenance window during which users are unable to edit their projects.

For Server Pro 3.4.0 and earlier:

  • The admin interface https://overleaf.example.com/admin (Close Editor tab) provides buttons for closing the editor and disconnecting users. Closing the editor stops users from opening new editor sessions, but keeps existing sessions open. Disconnecting users is happening with a 10 seconds grace period before the page reloads.
  • You will need to close the editor first, then disconnect all users.

Starting with Server Pro 3.5.0 the shutdown down process automates the closing of the site and user disconnections.

Stop the sharelatex container and start the backup process.

Once you are done with the backup you will need to restart the sharelatex container to restore editing capability.

Backup Tips

Backups need to be on a separate server to the one Overleaf running on, ideally in a different location entirely. Running multiple instances of mongodb is also not a backup as it will not prevent against corruption.

Mongodb

mongodb comes with a tool called mongodump which will export the dataset in a safe backup format.

Filesystem data

The path where your files are stored is specified in your settings file. This might be: /var/lib/overleaf (/var/lib/overleaf for versions 4.x and earlier), copying this directory recursively is needed. rsync is a good tool for the job.

Redis

Redis stores user sessions and pending document updates before they are flushed to Mongodb.

AOF (Append Only File) persistence is the recommended configuration for Redis persistence.

Toolkit users have AOF persistence enabled by default for new installs. Existing users are recommended to follow the instructions on the official documentation to switch to AOF:

https://redis.io/docs/latest/operate/oss_and_stack/management/persistence/#how-i-can-switch-to-aof-if-im-currently-using-dumprdb-snapshots

After switching to AOF toolkit users should set REDIS_AOF_PERSISTENCE=true in their config/overleaf.rc so redis starts with the --appendonly yes option.

If you decide to continue using RDB snapshots along with AOF persistence you can copy the RDB file to a secure location as a backup.

Migrating data between servers

At best you do not have any valuable data in the new instance yet. We do not have a process for merging the data of instances.

Assuming the new instance has not data yet, here are some steps you could follow. On a high level we produce a tar-ball of the mongo, redis and sharelatex volumes, copy it over to the new server and inflate it there again.

With the default docker-compose file that would be:

# Gracefully shutdown the old instance
# (See https://github.com/overleaf/overleaf/wiki/Data-and-Backups#performing-a-backup)
old-server$ docker stop sharelatex
old-server$ docker stop mongo redis

# Create the tar-ball
# Note that ~/sharelatex_data can also be ~/overleaf_data
old-server$ tar --create --file backup-old-server.tar ~/sharelatex_data ~/mongo_data ~/redis_data

# Copy the backup-old-server.tar file from the old-server to the new-server using any method that fits

# Gracefully shutdown new instance (if started yet)
new-server$ docker stop sharelatex
new-server$ docker stop mongo redis

# Move new data, you can delete it too
new-server$ mkdir backup-new-server
# Note that ~/sharelatex_data can also be ~/overleaf_data
new-server$ mv ~/sharelatex_data ~/mongo_data ~/redis_data backup-new-server/

# Populate data dirs again
new-server$ tar --extract --file backup-old-server.tar

# Start containers
new-server$ docker start mongo redis
new-server$ docker start sharelatex

Depending on your docker-compose config, you may need to adjust the paths of the mongo/redis/sharelatex volume.

With a toolkit setup that would be:

# Gracefully shutdown the old instance
# (See https://github.com/overleaf/overleaf/wiki/Data-and-Backups#performing-a-backup)
old-server$ bin/stop

# Create the tar-ball
old-server$ tar --create --file backup-old-server.tar config/ data/

# Copy the backup-old-server.tar file from the old-server to the new-server using any method that fits

# Gracefully shutdown new instance (if started yet)
new-server$ bin/stop

# Move new data, you can delete it too
new-server$ mkdir backup-new-server
new-server$ mv config/ data/ backup-new-server/

# Populate config/data dir again
new-server$ tar --extract --file backup-old-server.tar

# Start containers
new-server$ bin/up

Folders in detail

This section lists the folders used by Community Edition and Server Pro.

The type of each directory is shown with the following key:

  • (b) include in backups, best when the instance is stopped to ensure consistency
  • (d) can be deleted
  • (e) ephemeral files, can be deleted when the instance is stopped

Directories containing ephemeral files can be mounted on a local disk or tmpfs filesystem for better performance.

  1. ~/mongo_data (b)
    • mongodb datadir
  2. ~/redis_data (b)
    • redis db datadir
  3. ~/sharelatex_data (or ~/overleaf_data)
    1. bin/
      1. synctex (d)
        • unused in latest release, previously a custom synctex binary was used (synctex is used for source mapping between .tex files and the pdf)
    2. data/
      1. cache/ (e)
        • binary file cache for compiles
      2. compiles/ (e)
        • latex compilation happens here
      3. db.sqlite (d)
        • unused in latest release, previously stored clsi cache details (either moved to simple in-memory maps or we scan the disk)
      4. db.sqlite-wal (d)
        • unused in latest release, see db.sqlite
      5. output/ (e)
        • latex compilation output storage for serving to client
      6. template_files/ (b)
        • image previews of template system (Server Pro only)
      7. user_files/ (b)
        • binary files of projects
      8. history/ (b)
        • full project history files
    3. tmp/
      1. dumpFolder/ (e)
        • temporary files from handling zip files
      2. uploads/ (e)
        • buffering of file uploads (binary file/new-project-from-zip upload)
      3. projectHistories/ (e)
        • temporary files for full project history migrations

Exporting users' projects

A export-user-projects.mjs script was added in Overleaf CE and Server Pro 5.4.0:

# Overleaf Toolkit users:
$ bin/docker-compose exec sharelatex /bin/bash -ce "cd /overleaf/services/web && node modules/server-ce-scripts/scripts/export-user-projects.mjs --help"

# legacy docker-compose.yml users:
$ docker exec sharelatex /bin/bash -ce "cd /overleaf/services/web && node modules/server-ce-scripts/scripts/export-user-projects.mjs --help"

  --help, -h          Show help
  --user-id           The user ID (required unless using --export-all or --project-id)
  --project-id        Export a single project (cannot be used with --user-id or --export-all)
  --list              List user's projects (cannot be used with --output)
  --output            Output zip file (for single export operations)
  --export-all        Export all users' projects (requires --output-dir)
  --output-dir        Directory for storing all users' export files

Setup & Managing Overleaf

Server Pro

Legacy

Clone this wiki locally
0