Release 1.6.2

What's New

System Certificate Authentication Improper Chain Detection
Authentication Events
Multi-underlay channel group secret
Flag to disable posture check functionality

System Certificate Authentication Improper Chain Detection

Previous versions of SDKs and controllers issued and stored client certificate chains with differing levels of fidelity.
Depending on when an identity was enrolled, it may or may not have a proper client certificate chain. In single
controller environments, the controller will automatically include its known intermediates to help create valid
x509 certificate chains back to the network's Root CA.

In HA environments, each controller does not know of all the intermediates in the system by default. In order to support
dynamically scaling controllers, clients must store and provide a proper client certificate chain (a leaf certificate
followed by all necessary intermediate CA certs). Identities enrolled with SDKs that did not store the chain
or controllers that did not provide the chain will encounter authentication issues in multi-controller environments.

To help determine which identities have issues, the system has been augmented to allow detection of problematic
certificate authenticators. Firstly, authenticators have flags on them that are set during authentication to detect
the current behavior of a specific client that owns and is invoking the certificate authenticator.

Detecting Improper Client Cert Chains As A Client SDK

The current API Sessions endpoint (https://<host>/edge/client/v1/current-api-session) now returns a value named
improperClientCertChain as a boolean value. If it is not present or false no action should be taken. If true
it means that the current API Session was authenticated with an internal PKI issued certificate where the client
did not provide a full chain to the root CA during authentication; indicating a problem with the certificate storage
mechanism in the application or due to the controller version used during enrollment/extension not providing a chain.
The SDK should proactively opt to begin certificate extension on its own to obtain a proper chain. Authentication
succeeded in this case because the controller relied upon a deprecated certificate pool that happen to include the necessary
intermediate CAs.

Detecting Clients Without Proper Chains

After a client has authenticated with the system via a network-issued certificate at least one time, a number of fields
are set depending on what the client provided. These values can be reviewed via the Edge Management API for
Authenticators (edge/management/v1/authenticators). Where an authenticator has isIssuedByNetwork set to true and
lastAuthResolvedToRoot set to false, indicates that the related identity/client is not providing a chain when it
should.

Additionally, if authenticator events are enabled and being processed, events will have a field
improper_client_cert_chain set to true (see Authentication Events below)

Fixing Clients Chains

Once an authenticator has been identified as problematic, an administrator should verify the client is using the newest
possible versions of its SDK and either re-enroll it or request the identity to extend the next time it
authenticates. Both scenarios result in a new certificate and chain being provided to the client.

Re-Enrollment removes the current authenticator, making authentication impossible until enrollment is completed.
Once completed, a new authenticator is created, and the new certificate generated from the process can be used to
authenticate. Humans or other external automation processes have to deliver and consume the new enrollment JWT.
Extension leaves the authenticator in place and authentication can still occur. When the extension process is
completed, the authenticator is updated and will use the new certificate generated from the process. SDKs will
handle the process by raising SDK specific events to drive the process.

While related and similar, using one over the other depends on the situation. Re-Enrollment is best for when
an client or its host are unrecoverable; either through damage, decommissioning, or suspected to be compromised.
Extension is useful when the client and host are in a known good status and one simply wants to issue new
certificates to said client. Extension is also "hands off" from the client as long as the client is using
SDKs that support enrollment request/key rolling.

Useful Authenticator Values

Authenticators are an entity within OpenZiti and can be queried via the CLI (ziti edge list authenticators) or
Edge Management API (GET https://<host>/edge/management/v1/authenticators). Below are properties that are
useful for determining clients with improper chains.

isIssuedByNetwork (boolean) indicates the authenticator is a certificate that was issued by the network
isExtendRequested (boolean) indicated the authenticator will report to the client to extend its certificate on next
authentication.
isKeyRollRequested (boolean) indicates if key rolling is requested with an outstanding isExtendRequested
extendRequestedAt (string, date time) indicates when an outstanding extension was requested, otherwise null
or not provided
lastAuthResolvedToRoot (boolean) indicates if the last time the client authenticated, it provided a certificate
chain that resolved to the root CA. Only valid if isIssuedByNetwork is true.

Re-Enrollment

Re-enrollment can be accomplished via the CLI or the Edge Management API. Re-enrollment removes the current authenticator,
stopping the client from authenticating. All of the underlying configuration for the related identity is preserved.
The newly created enrollment will have a new enrollment JWT that will be consumed during the enrollment process and
will result in a new client certificate and chain.

CLI:

ziti edge update authenticator cert <id> --re-enroll [--duration <duration>]

Edge Management API:

POST /edge/management/v1/authenticators/:id/re-enroll

{
    expiresAt: "2025-05-21T13:47:26Z"
}

Both of the above provide an enrollment id and or enrollment JWT token as output. These can be consumed in various
OpenZiti applications or CLI commands.

CLI Enrollment

 ziti edge enroll -h
enroll an identity

Usage:
  ziti edge enroll path/to/jwt [flags]

Flags:
      --ca string         Additional trusted certificates
  -c, --cert string       The certificate to present when establishing a connection.
  -h, --help              help for enroll
  -n, --idname string     Names the identity. Ignored if not 3rd party auto enrollment
  -j, --jwt string        Enrollment token (JWT file). Required
  -k, --key string        The key to use with the certificate. Optionally specify the engine to use. supported engines: [parsec]
  -a, --keyAlg RSA|EC     Crypto algorithm to use when generating private key (default RSA)
  -o, --out string        Output configuration file.
  -p, --password string   Password for updb enrollment, prompted if not provided and necessary
      --rm                Remove the JWT on success
  -u, --username string   Username for updb enrollment, prompted if not provided and necessary
  -v, --verbose           Enable verbose logging

OpenZiti Tunneler Enrollment

ziti-edge-tunnel add --jwt "$(< ./in-file.jwt)" --identity myIdentityName

Request Extension

Extension leaves the authenticator in place, but notifies SDKs that they should generate a CSR to request a new
certificate. Once the process is complete, the client will have a new certificate and chain that must be used
for later authentication requests. This can be initiated through the CLI or Edge Management API.

CLI:

ziti edge update authenticator cert <id> --request-extend [--request-key-roll]

Edge Management API:

POST /edge/management/v1/authenticators/:id/request-extend

{
    rollKeys: true,
}

These commands will cause the authenticator to be updated to have isExtendRequested, extendRequestedAt, and optionally isKeyRollRequested` updated on the authenticator. These values will be used to signal the client to
extend their certificate on the next successful authentication attempt.

Authentication Events

A new event names space authentication has been added for events with two types success and fail. These can be
enabled via the controller configuration files.

Event Documentation:

// An AuthenticationEvent is emitted when an authentication attempt is made
//
// Types of authentication events
//   - fail - authentication failed
//   - success - authentication succeeded
//
// Types of authentication methods
//   - updb - username password from the internal database
//   - cert - a certificate, either first party or 3rd party
//   - ext-jwt - an external JWT from an IDP

Example Controller Configuration:

events:
  jsonLogger:
    subscriptions:
      - type: authentication
#          - success
#          - fail
    handler:
      type: file
      format: json
      path: ${TMPDIR}/ziti-events.log

Example Output:

{
  "namespace": "authentication",
  "event_src_id": "ctrl_client",
  "timestamp": "2025-05-21T10:24:32.6532054-04:00",
  "event_type": "fail",
  "type": "password",
  "authenticator_id": "B0JrMDPGxd",
  "external_jwt_signer_id": "",
  "identity_id": "BGJabPP0K",
  "auth_policy_id": "default",
  "remote_address": "127.0.0.1:51587",
  "success": false,
  "reason": "could not authenticate, password does not match",
  "improper_client_cert_chain": false
}

Multi-underlay channel group secret

For additional security the experimental multi-underlay channel code now requires that
clients provide a shared secret. This ensures that channels are get the expected
underlays without requiring much larger group ids. On the client side this will requ...

Release 1.6.1

What's New

Bug fixes and library updates
Ability to request that SDKs extend and optionally roll their key
Address translations can now be specified in host.v1 service configuration

Ability to request that SDKs extend and optionally roll their key

It is now possible for administrators to flag specific certificate authenticators as needed to extend their current
certificate early and/or optionally roll the keypair that underpins the certificate. This capability only works for
certificates issued by the OpenZiti network. If '3rd party CAs' are in use, those certificate authenticators will not
work with this system.

SDKs must support this capability for it to have any effect, and the application utilizing the SDK must respond to the
certificate extension events to store certificate credentials.

This capability is located in the Management API at /edge/management/v1/authenticators/{id>/request-extend.
Its payload is currently and optional boolean value for rollKeys that can be set to true/false and defaults to
false if not provided.

This can also be issued via the CLI:

> ziti edge update authenticator cert -h
Request a specific certificate authenticator to --requestExtend or --requestKeyRoll, --requestKeyRoll implies --requestExtend

Usage:
  ziti edge update authenticator cert <authenticatorId> [--requestExtend] [--requestKeyRoll] [flags]

Flags:
  -h, --help             help for cert
  -e, --requestExtend    Specify the certificate authenticator should be flagged for extension
  -r, --requestKeyRoll   Specify the certificate authenticator should be flagged for key rolling, implies --requestExtend

Requesting an extension flags new fields on a certificate authenticator in the values isExtendRequest and
isKeyRollRequested. These values are set to false after the client performs a certificate extension. The CLI
has been updated to report these values on certificate authenticators via ziti edge list authenticators.

These values are also present on the /edge/client/v1/current-api-session endpoint when a client has use certificate
authentication to initiate an API Session using a certificate authenticator.

Additionally, a log of key rolling activity per authenticator will be available in a future release.

host.v1 Address Translation

The host.v1 service configuration type now includes a forwardAddressTranslations field that specifies
how a hosting tunneler should translate destination IPs from the client when connecting to the underlay
application.

Component Updates and Bug Fixes

github.com/openziti/edge-api: v0.26.42 -> v0.26.43
github.com/openziti/ziti: v1.6.0 -> v1.6.1
- Issue #2996 - Add ability to signal SDKs to extend cert authenticator
- Issue #2963 - support intercept.v1 --> host.v1 address translation

Release 1.6.0

What's New

Bug fixes and library updates
Change to cluster add peer
Experimental multi-underlay SDK support
Experimental SDK flow-control support

Cluster Add Peer Change

The ziti agent cluster add command no longer supports the --id argument
for providing the peer id. The add operation will now always connect to the
peer, verify the certs and get the peer advertise address and id from the
peer directly. This will ensure that the peer is reachable and valid before
it is added to the cluster.

Multi-underlay SDK support

For SDKs which support it, the edge router now supports a separate control channel
connection along side the data connection. If the SDK doesn't request separate
channels, the edge router will continue to work with a single connection. This
feature is still experimental and may have bugs, may be changed or be removed.

SDK Flow-control

For SDKs which support it, the edge router now supports delegating flow control
to the SDK. This feature is still experimental and may have bugs, may be changed
or be removed.

Component Updates and Bug Fixes

github.com/openziti/channel/v4: v3.0.39 -> v4.0.6
- Issue #182 - MultiListener can deadlock
- Issue #180 - Add GetUserData to Channel interface
- Issue #176 - Multi-channel need a mechanism to notify the txer that the underlay has closed
- Issue #172 - Support multi-underlay channels
github.com/openziti/identity: v1.0.100 -> v1.0.101
- Issue #64 - Support a way to check if a cert/serverCert can be saved
github.com/openziti/metrics: v1.3.0 -> v1.4.1
- Issue #53 - Add reporter useful for emitting metrics to stdout
github.com/openziti/sdk-golang: v0.25.1 -> v1.1.0
- Issue #702 - [Go SDK] Support xgress flow control from the SDK
- Issue #722 - Move xgress impl to SDK
- Issue #717 - ER connection race condition can leak connections
- Issue #689 - Concurrent map iteration and modification in getEdgeRouterConn causes panic
- Issue #701 - Support multi-underlay channels for edge router connections
github.com/openziti/transport/v2: v2.0.167 -> v2.0.168
github.com/openziti/xweb/v2: v2.3.0 -> v2.3.1
github.com/openziti/ziti: v1.5.4 -> v1.6.0
- Issue #3005 - Always check that a controller is reachable and valid before adding it to an HA controller cluster
- Issue #2986 - [Router] Support xgress flow control from the SDK
- Issue #2999 - OIDC JWT backed sessions cannot verify extended certs
- Issue #2997 - Add Authenticator Id to OIDC JWTs/return for current-api-session
- Issue #2904 - Support client certificate authorities in TLS handshake
- Issue #2973 - CLI: add a subcommand to retrieve network JWT
- Issue #2984 - Extend enrollments does not return a full chain
- Issue #2930 - Support multi-underlay channels for the edge SDK
- Issue #2978 - Create loop4 sim for testing circuit contention and scale
- Issue #2981 - Remove PayloadBufferForwarder API from xgress retransmitter
- Issue #2906 - Controller not removed from DB controller store when removed from controller
- Issue #2922 - Validate node address before adding to cluster
- Issue #2932 - Fix router data model 'create public key' related errors
- Issue #2919 - Make xgress pluggable, so it can be used from the SDK
- Issue #2955 - Extract xgress inspection types
- Issue #2954 - Encapsulate xgress metrics
- Issue #2952 - Remove global payload ingester
- Issue #2951 - Remove global xgress retransmitter
- Issue #2950 - Move router specific xgress code to a new xgress_router package
- Issue #2920 - Make xgress acker configurable

Release 1.5.4

What's new

Bug fixes

Component Updates and Bug Fixes

github.com/openziti/ziti: v1.5.3 -> v1.5.4
- Issue #2947 - Panic on router started up if edge/tunnel bindings not configured
- Issue #2948 - Allow ER/T to run without edge listener

Release 1.5.3

What's New

This release updates the Go version from 1.23 to 1.24

Release 1.1.18

What's New

Backport of exponential terminator cost crediting

Component Updates and Bug Fixes

github.com/openziti/ziti: v1.1.17 -> v1.1.18
- Issue #2851 - Change terminator failure cost crediting to be exponential based on time since last failure

Release 1.5.2

What's New

This release reverts a change refactoring some flow-control apis, as the change caused a panic

Release 1.5.1

What's New

Bug fixes and features
github.com/openziti/sdk-golang: v0.25.0 -> v0.25.1
- Issue #699 - SDK UPDB enrollment
github.com/openziti/ziti: v1.5.0 -> v1.5.1
- Issue #2931 - help ext-jwt-signer auth by logging incoming jwt audience
- Issue #2934 - API Session Certs in HA not connect to ERs in all scenarios
- Issue #2926 - Implement minimal Xgress SDK

Release 1.5.0

What's New

Bug fixes and features
Change to router endpoints file default name
Updated Cluster Defaults
Updates to terminator costing
Router metrics changes

Router Endpoints File

The router endpoints file used to have a default name of endpoints.
As it is a YAML file, the file now has a default name of endpoints.yml.
This may affect existing setups running the beta HA code. If this is
a concern, the endpoints file path can be configured:

ctrl:
  endpointsFile: /path/to/endpoints.file

Updated Cluster Defaults

The following defaults have changed.

cluster:
  # How many updates before creating a new snapshot. 
  #
  # New default: 500
  # Old default: 8192
  snapshotThreshold: 500 

  # How many old entries to keep around, so that a stream of
  # entries can be sent to sync peers, instead of sending an
  # entire snapshot
  #
  # New default: 500
  # Old default: 10240
  trailingLogs: 500

Terminator Costing Changes

When a terminator is selected for a service dial, and that dial results in a failure, a failure cost
is added to that terminator. This will bias future dials towards other terminators, if they are available.

The failure cost can be reduced by successful dials. Failure cost is also reduced over time. In previous
releases this was a fixed credit of 5, every minute. This is now changing to an exponential
amount, based on time since the last failure.

If X is minutes since last failure, the credit is now: min(10, 2 ^ (X/5)).

Router Metrics Changes

There are four new router metrics, focused on visibility into flow control.

xgress.blocked_by_local_window_rate - meter which ticks whenever an xgress becomes blocked by the local window being full
xgress.blocked_by_remote_window_rate - meter which ticks whenever an xgress becomes blocked by the remote receive buffer being full
xgress.blocked_time - timer which tracks how long xgresses are in a blocked state.
xgress_edge.long_data_queue_time - timer that tracks times to process incoming data payloads to xgress_edge.

The xgress_edge.long_data_queue_time will be controller by a router config file setting. It will default to disabled. The other metrics will always be enabled.

Router metrics has two new configuration setting:

metrics:
  # Number of usage events to be able to queue. Defaults to 256. If this queue backs up, it can
  # slow down dispatch of data from an SDK onto the fabric.
  eventQueueSize: 256

  # If set to true, enables the xgress_edge.long_data_queue_time metric. Defaults to false.
  enableDataDelayMetric: false

Component Updates and Bug Fixes

github.com/openziti/channel/v3: v3.0.37 -> v3.0.39
github.com/openziti/edge-api: v0.26.41 -> v0.26.42
github.com/openziti/foundation/v2: v2.0.58 -> v2.0.59
github.com/openziti/metrics: v1.2.69 -> v1.3.0
- Issue #49 - Make usage registry event queue size configurable
- Issue #50 - Do metrics message construction in msg sender goroutine rather than usage/interval event goroutine
github.com/openziti/runzmd: v1.0.65 -> v1.0.67
github.com/openziti/secretstream: v0.1.31 -> v0.1.32
github.com/openziti/storage: v0.4.5 -> v0.4.7
github.com/openziti/transport/v2: v2.0.165 -> v2.0.167
github.com/openziti/ziti: v1.4.3 -> v1.5.0
- Issue #2910 - Add additional metrics for visibility into flow control backpressure
- Issue #2608 - Verify latest Prometheus server
- Issue #2899 8000 - Allow configuring size of router metrics event queue size
- Issue #2896 - ziti router run --extend does not function
- Issue #2796 - Generated API client enrollment operations fail
- Issue #2889 - Ensure identity online/offline statuses work correctly for ER/Ts
- Issue #2891 - Restore can panic if using import from db
- Issue #2835 - Add mechanism for selecting CLI layout
- Issue #2836 - Add run subcommand
- Issue #2837 - Add enroll subcommand
- Issue #2851 - Change terminator failure cost crediting to be exponential based on time since last failure
- Issue #2854 - Fix controller online status
- Issue #2829 - Update Raft Configuration Defaults
- Issue #2849 - Router endpoints file should have .yml extension by default
- Issue #2875 - add --authenticate to verify ext-jwt-signer oidc
- Issue #2873 - updates to verify ext-jwt-signer oidc

Release 1.1.17

What's New

Update to latest metrics library to pick up usage processing fixes
Backport of router metrics changes

Router Metrics Changes

There are four new router metrics, focused on visibility into flow control.

xgress.blocked_by_local_window_rate - meter which ticks whenever an xgress becomes blocked by the local window being full
xgress.blocked_by_remote_window_rate - meter which ticks whenever an xgress becomes blocked by the remote receive buffer being full
xgress.blocked_time - timer which tracks how long xgresses are in a blocked state.
xgress_edge.long_data_queue_time - timer that tracks times to process incoming data payloads to xgress_edge.

The xgress_edge.long_data_queue_time will be controller by a router config file setting. It will default to enabled. The other metrics will always be enabled.

Router metrics has two new configuration setting:

metrics:
  # Number of usage events to be able to queue. Defaults to 256. If this queue backs up, it can
  # slow down dispatch of data from an SDK onto the fabric.
  eventQueueSize: 256

  # If set to false, disables the xgress_edge.long_data_queue_time metric. Defaults to true.
  enableDataDelayMetric: false

Component Updates and Bug Fixes

github.com/openziti/foundation/v2: v2.0.49 -> v2.0.58
github.com/openziti/metrics: v1.2.58 -> v1.3.0
- Issue #49 - Make usage registry event queue size configurable
- Issue #50 - Do metrics message construction in msg sender goroutine rather than usage/interval event goroutine
github.com/openziti/ziti: v1.1.16 -> v1.1.17
- Issue #2910 - Add additional metrics for visibility into flow control backpressure
- Issue #2899 - Allow configuring size of router metrics event queue size

Releases: openziti/ziti

v1.6.2

Release 1.6.2

What's New

System Certificate Authentication Improper Chain Detection

Detecting Improper Client Cert Chains As A Client SDK

Detecting Clients Without Proper Chains

Fixing Clients Chains

Useful Authenticator Values

Re-Enrollment

Request Extension

Authentication Events

Multi-underlay channel group secret

Uh oh!

v1.6.1

Release 1.6.1

What's New

Ability to request that SDKs extend and optionally roll their key

host.v1 Address Translation

Component Updates and Bug Fixes

Uh oh!

v1.6.0

Release 1.6.0

What's New

Cluster Add Peer Change

Multi-underlay SDK support

SDK Flow-control

Component Updates and Bug Fixes

Uh oh!

v1.5.4

Release 1.5.4

What's new

Component Updates and Bug Fixes

Uh oh!

v1.5.3

Release 1.5.3

What's New

Uh oh!

v1.1.18

Release 1.1.18

What's New

Component Updates and Bug Fixes

Uh oh!

v1.5.2

Release 1.5.2

What's New

Uh oh!

v1.5.1

Release 1.5.1

What's New

Uh oh!

v1.5.0

Release 1.5.0

What's New

Router Endpoints File

Updated Cluster Defaults

Terminator Costing Changes

Router Metrics Changes

Component Updates and Bug Fixes

Uh oh!

v1.1.17

Release 1.1.17

What's New

Router Metrics Changes

Component Updates and Bug Fixes

Uh oh!