Description
Problem
Task inputs at this time only operate with file paths, globs, and environment variables. For the most part, this works and is all most users require for their tasks. But once a task or project reaches a certain complexity, and more granularity is required, the current inputs are simply not enough.
This is even more true when determining affected status, as the current affected implementation is based on touched files.
Proposal
String and object formats
Moving forward, to support all the complexity that inputs will require, we will support both string and object based formats.
Besides current file/glob/env strings, the new string format will be URI based, with each input type having a custom protocol/scheme, and object fields defined as query params. For complex inputs, we suggest always using objects.
Files
Strings will work the same, but will also support the file:// protocol.
inputs:
- 'file.txt'
- 'file://file.txt'For objects, the file field is required, while all other fields are optional. This format does not support protocols.
inputs:
- file: 'file.txt'Globs
Strings will work the same, but will also support the glob:// protocol.
inputs:
- 'file.*'
- 'glob://file.*'For objects, the glob field is required, while all other fields are optional. This format does not support protocols.
inputs:
- glob: 'file.*'Env vars
There are no changes to env vars. They will not support a protocol nor the object format at this time.
inputs:
- '$VAR'
- '$VAR_*'New input types
To better support our users and their needs, we will be introducing 2 new input types.
Depends on project has changed files
When a project (A) depends on another project (B), there are situations where a task (in A) should re-run if there are any changes in the dependency project (B). Right now this can be achieved through no-op task relationships, but it's non-ideal.
To support this, we'll be introducing a new project:// input, which can be used to reference a project by ID from the owning project's dependsOn list. Any changes in this project will bust the task's cache.
# String
inputs:
- 'project://b'
# Object
inputs:
- project: 'b'By default this will use **/* to detect any file changes within the dependency project. When using the object form, a filter field can be provided to restrict what triggers the change using glob.
# String
inputs:
- 'project://b?filter=src/**/*&filter=data/**/*.json'
# Object
inputs:
- project: 'b'
filter:
- 'src/**/*'
- 'data/**/*.json'Or as an alternative, the group field can be used to reference a file group in the target project.
# String
inputs:
- 'project://b?group=sources'
# Object
inputs:
- project: 'b'
group: 'sources'Toolchain manifest dependency has changed
When a project contains a toolchain (node) manifest file (package.json), there are situations where a dependency version changes and the task should be marked affected. Currently this granularity is not possible. The alternative is to include the manifest/lockfile as an input, but this is too broad and invalidates the cache too often.
To support this, we'll be introducing a new manifest:// input, which requires a toolchain ID, and an optional list of deps. If no deps are provided, it will track all deps.
# String
inputs:
- 'manifest://node'
- 'manifest://node?deps=react'
- 'manifest://node?deps=react&deps=typescript'
# Object
inputs:
- manifest: 'node'
deps: ['react', 'typescript']New file group formats
The @token syntax can be quote confusing for new comers, and since the current implementation of tokens will be deprecated in the future, we'll be replacing most of the @ tokens with this new format and the group:// protocol. When no params are passed in the input, it defaults to "use group files as-is", similar to @group(name).
fileGroups:
sources:
- 'src/**/*'
# String
inputs:
- 'group://sources'
- 'group://sources?as=dirs'
- 'group://sources?as=root'
# Object
inputs:
- group: 'sources'
as: 'files'Other?
If you have other ideas for new inut types, be sure to leave some feedback!
Update existing input types
In addition to introducing 2 new input types, we'll also be expanding the existing inputs with new functionality.
Optional files
Currently all non-glob file inputs require the file to exist at runtime, otherwise a warning is logged. This warning can be silenced with settings, but it would be much easier to allow files to be marked as optional, and as such, we'll be adding an optional field to files.
# String
inputs:
- 'file://file.txt?optional'
# Object
inputs:
- file: 'file.txt'
optional: trueWhen using the string format, the file:// protocol is required when marking as optional.
File matching content
Currently when using a non-glob file input, any change to that file will mark it as affected, and re-run the task. However, sometimes you only want to mark it as affected if the contents of the file contains certain information.
To support this, we'll be adding a match field, which supports a regex pattern. If the content matches the pattern, the input will be affected and included.
# String
inputs:
- 'file://file.txt?match=(foo|bar)?' # Prefer an object
# Object
inputs:
- file: 'file.txt'
match: '(foo|bar)?'Caching glob lookups
Moving forward, we will start caching all glob lookups for increased performance. This may be problematic if files are created dynamically at runtime, causing inconsistencies with the cache. To mitigate this, a new cache field will be added for globs.
# String
inputs:
- 'glob://file.*?cache=false'
# Object
inputs:
- glob: 'file.txt'
cache: falseOther?
If you have other ideas for input enhancements, be sure to leave some feedback!