Release 3.21.1

New Feature (GA)

Added Rebuild and Rebalance Throttle Controls, a cluster-wide setting that caps the bandwidth used by rebuild and rebalance traffic on each node, with separate transmit and receive limits, so operators can protect application performance during recovery or let recovery run faster when needed.

LBM1-49133

New Feature (GA)

Added Spare Capacity Reservation, a cluster-level setting (SpareCapacityReservationPercentage, 0-50%) that keeps part of each node's capacity free so surviving nodes always have room to rebuild after a node failure.

LBM1-49135

New Feature (GA)

Server eviction (evict data), introduced as a Technology Preview, is now generally available and officially supported for production use. Evicting a server migrates all of its volumes and snapshots to other servers in the cluster before planned maintenance or removal, keeping volumes fully protected throughout, and can also be used to move data off a server whose nodes have become unavailable without waiting for them to reach a permanently failed state.

LBM1-49137

New Feature (GA)

Lightbits CSI plugin: Added IP-ACL support. With the new opt-in ip-acl StorageClass parameter, the plugin manages each volume's IP ACL, adding a worker node's data-path addresses when the volume is staged on it and resetting the list when the last node detaches. Statically provisioned volumes opt in with the "|ipacl:enabled" volumeHandle suffix.

LBM1-40422

New Feature (Tech Preview)

Added Tech Preview support for installing the Lightbits storage cluster on Red Hat Enterprise Linux (RHEL) 10, extending the server OS options beyond the existing RHEL family and Ubuntu releases (RHEL 10 has been supported on the client side since earlier releases). This Tech Preview is intended for evaluation only: customers are encouraged to validate and test their workloads with Lightbits deployed on RHEL 10 in non-production environments. It is not supported for production use and is not generally available in this release.

LBM1-49136

Enhancement

When a container upgrade fails to refresh some configuration files, the event now shows the full path and reports image read errors as a warning

LBM1-45797

Enhancement

Fixed a packaging issue where an upgrade could overwrite a cluster's customized /etc/profile-generator/user-config.yaml; it is now correctly preserved across upgrades in both RPM and container mode.

LBM1-45583

Enhancement

Fixed a rare race condition that could cause a volume replica to serve outdated data. It required a volume with two or more replicas whose primary server had become read-only after running out of storage capacity or metadata space, while another replica finished rebuilding and took over as primary. If the first server's read-only state cleared within the same fraction of a second as that takeover, it could return as primary without the rebuild it needed.

LBM1-39499

Enhancement

Fixed an issue where a protection group could remain read-only for an extended period after a node migration, even after the affected node had become writable again.

LBM1-34456

Enhancement

OpenStack Cinder driver hardening: The driver no longer suppresses TLS certificate warnings for other backends in the same cinder-volume process, and the API JWT no longer appears in DEBUG logs or in a generated sample configuration.

LBM1-47383

Enhancement

Improved cluster rebalance performance on large clusters: the per-node selection pass used to do far more work than its answer needed, which could make a drain or rebalance take too long to finish on clusters with many nodes and protection groups. (Measured: roughly 3.4x fewer allocations for a 32-node, 480-group rebalance pass.)

LBM1-47484

Enhancement

Lightbits CSI plugin security enhancement: Added support for NVMe In-Band Authentication (DH-HMAC-CHAP). When the new inBandAuth Helm value is enabled and the Lightbits cluster has In-Band Authentication turned on, each node registers itself as a trusted host, its secrets are generated by the cluster and stored in the discovery-client configuration, and every NVMe/TCP connection is authenticated, securing access between Kubernetes worker nodes and the storage cluster. On clusters without In-Band Authentication, the plugin connects as before. This requires the discovery-client to run in a container; otherwise the Helm chart does not render.

LBM1-48471

Enhancement

lbcli list snapshots can now filter by volume; for example: lbcli list snapshots --project-name X --volume-uuid Y.

LBM1-47642

Enhancement

Log streaming: The API now returns a clear error when an rsyslog endpoint includes a port or scheme (for example "host.example.net:24712"). The endpoint must be a hostname or IP address, with the port in its own field. lbcli is not affected: its --endpoint flag still accepts host:port.

LBM1-48175

Enhancement

Protection groups left behind when nodes become permanently failed are now removed automatically, every 12 hours by default. On long-running clusters these leftover protection groups can build up and slow down switchovers. Upgrade note: after upgrading from 3.20.1, the cleanup is enabled on every cluster, including clusters where it had been turned off. To keep it off, set the cleanup interval back to 0 after the upgrade.

LBM1-47138

Enhancement

The installation TUI can now deploy Lightbits in container mode on Ubuntu and Debian. It detects the target operating system, signs in to the container registry and pulls the images, checks before deployment that the selected Lightbits version supports the target servers, and verifies the installation with lbcli when it finishes.

LBM1-45442

Enhancement

The installation TUI's Remote Targets step now supports bulk server entry: paste a list of addresses, including ranges and CIDR blocks, instead of adding hosts one at a time.

LBM1-46424

Enhancement

The NVMe Keep Alive Timeout (KATO) grace period, previously fixed at 10 seconds, is now configurable with nvme_kato_grace_secs. Setting it to 0 disables the grace period.

LBM1-47388

Enhancement

The Node dashboard now includes a panel showing metadata utilization as a percentage.

LBM1-46712

Enhancement

When a node's journal device fails, the node now automatically creates and uses a new journal device on restart, provided it can shut down gracefully and enough devices remain. This reduces node downtime after a journal device failure.

LBM1-42323

Enhancement

When installing on Ubuntu, the installer now automatically opens the ports that Lightbits requires in the ufw firewall.

LBM1-47495

Major

Fixed a rare issue where deleting a snapshot during a rebuild could raise data-integrity events and leave the affected data not rebuilt on the secondary replica.

LBM1-47343

Major

Fixed an edge case where the frontend could hang for up to 120 seconds due to a missing timeout on the replicator connection.

LBM1-46486

Major

Fixed an issue where a trim (deallocate) request that arrived during a replica configuration change, such as a node failure, could be reported to the host as successful even though a replica had not applied it, leaving the replicas inconsistent. Such requests now fail back to the host and are retried on the new primary through an ANA path change.

LBM1-48806

Major

Fixed an issue in VMware environments where nodes could briefly go down and come back up when the client aborted NVMe commands.

LBM1-45998

Major

Fixed an issue where, after a node was restarted or replaced, volume recovery on that node could stall indefinitely on clusters with many clone volumes created from a small number of shared snapshots. Clone volumes could be processed before the snapshots they depend on and use up all available recovery workers, so the source snapshots were never processed and the affected protection groups could not finish rebuilding. Source snapshots are now processed before their clones.

LBM1-48205

Major

Fixed node-manager failing to start when fewer healthy drives are present than initialDeviceCount (for example, after a drive failure). The node now starts with the missing drive marked as failed.

LBM1-48532

Major

PEERMON (connectivity report) now fails a node if any of the replication connections to it has more than 15s of down time with no live link (configurable in duroslight/conf.yaml as replicator/peer_dead_timeout_ms).

LBM1-45912

Minor

Fixed an issue where a volume migration kept retrying when its target node became permanently failed. The migration now stops.

LBM1-44464

Minor

Fixed a failure in cluster root key rotation on TPM-backed clusters, where the rotation could time out and report an internal error. TPM operations are now given a longer default timeout and are bounded so that a stalled helper process cannot block the rotation.

LBM1-46100

Minor

Fixed a false UnrecoverableDataIntegrityError event, with no volume ID, and a "Rebuild read detected a damaged object" message in the frontend log. This could happen during a rebuild when a volume with previously deleted snapshots was deleted.

LBM1-48646

Minor

Fixed a volume leak when retyping an OpenStack Cinder volume between projects.

LBM1-44442

Minor

Fixed an issue in the management services where an operation that kept failing was eventually retried with almost no delay. After enough consecutive failures (about 1.5 minutes of retries on 3.18.x, or about 16 hours on 3.19.x and 3.20.x), the delay between retries dropped to nearly zero. The continuous retries could keep a CPU core busy and flood the logs, pushing out the older log entries needed to investigate the original failure.

LBM1-48041

Minor

Fixed an issue where a node that had just restarted could briefly report its power-up as already 100% complete, based on a leftover value from before the restart; this could let a recovery step proceed before the node was actually ready. A node that cannot currently be queried now correctly reports unknown progress instead of a stale value.

LBM1-47398

Minor

Fixed an issue where a snapshot or volume created while a node was inactive, and deleted while that node was still powering up, could stay behind on the node. The leftover could block deletion of the snapshot it was created from, or prevent volumes on that node from rebuilding correctly.

LBM1-48012

Minor

Fixed an issue where a volume rebuild could fail if a snapshot was deleted during the rebuild.

LBM1-46869

Minor

Fixed an issue where the node manager completed its shutdown before the data-path service had stopped, which delayed the server's shutdown. The data-path service now has dedicated time to stop.

LBM1-44334

Minor

Fixed an RPM upgrade issue where the Lightbits package's etcd version requirement (>= v3.5.30) could be satisfied by version-string comparison even against an older installed etcd package (such as 3.5.18), so an upgrade from v3.18.3 or v3.20.1 could leave the cluster on an outdated etcd. Upgrades now pull in the correct etcd version.

LBM1-48605

Minor

Fixed an upgrade preflight check that could intermittently pin its etcd snapshot to an already-down etcd member and fail with a timeout even though cluster quorum was intact; the preflight now selects a healthy member so cluster upgrades start deterministically.

LBM1-46187

Minor

Fixed the Ansible node cleanup leaving journaling RAID (mdadm) devices behind, which could cause the next installation to fail because too few NVMe drives were available.

LBM1-45625

Minor

Fixed the Chrony and Cluster dashboards in the monitoring stack, which showed no data because of an incorrect data-source reference.

LBM1-45884

Minor

Fixed the TCP segments re-transmitted panel in the monitoring dashboard showing negative values after a counter reset.

LBM1-43755

Minor

Fixed the volume-state panel on the Cluster dashboard, which reported an incorrect number of volumes in the migrating state.

LBM1-46295

Minor

Fixed volumes configured with a specific IP-ACL (not allow-any) being invisible to an allowed host when the node is set to listen on all interfaces; an allowed IPv4 host address arriving as an IPv4-mapped IPv6 address no longer fails the ACL match.

LBM1-46207

Minor

OpenStack Cinder: retyping a volume to change only its QoS policy now updates the policy in place, without migrating the volume's data.

LBM1-47082

Minor

Reduced logging noise from lightbox-exporter: a repeated SMART or NVMe error-log collection failure is now logged as a warning once, and again only when the failure changes or clears.

LBM1-44531

Minor

Fixed profile-generator failing to start on some rare cases of installations with node-manager holding the shared local database lock.

LBM1-46174

Minor

SSD SMART alert reliability. Aligned the for: durations of the SSD SMART gauge-threshold alerts (SmartTemperatureCritical, SmartTemperatureExtreme, SmartCriticalWarning, SmartDeviceUnhealthy, SmartAvailableSpareLow, SmartCriticalWear) to 10 minutes so a single missed Prometheus scrape (short network blip, exporter restart, target-response jitter) no longer resets the alert's evaluation streak. Real incidents that briefly race a scrape now fire reliably instead of being suppressed until the next continuous window; the observable alert-firing latency for a persistent overheating or health event stays under 15 minutes end to end.

LBM1-44667

Minor

The API now returns a conflict (HTTP 409) instead of an internal server error (HTTP 500) when a request is rejected because another task is already running on the same resource.

LBM1-46814