Issues Fixed in Lightbits 3.21.1

ID

Description

48806

In v3.20.1, a trim (deallocate) request issued while a volume's replicas are changing, for example, during a node failure, can be reported to the host as successful even though one replica did not apply it. No data is lost, but the trimmed blocks can remain allocated on that replica, so its contents for those blocks can differ from the other replicas until the blocks are written again. This issue is fixed in v3.20.2 and v3.21.1.

48646

A false UnrecoverableDataIntegrityError event with no volume ID, and "Rebuild read detected a damaged object" in the frontend log, could be reported during a rebuild when a volume with previously deleted snapshots is deleted. This issue is fixed in v3.20.2 and v3.21.1.

48605

In v3.18.3 and v3.20.1, upgrading a cluster with RPM packages from a release that ships etcd 3.5.18 (for example, v3.18.2 or v3.19.1) does not upgrade etcd. The cluster keeps running etcd 3.5.18 instead of the etcd 3.5.30 included in the release. To check a server, run "rpm -q etcd". This issue is fixed in v3.20.2 and v3.21.1; upgrading to a fixed version also upgrades etcd.

48532

The node-manager service can fail to start if the number of drives present falls below the minimum required for the node's configured layout, for example, after a drive failure, power loss, or planned maintenance. In earlier releases, the node started with the missing drive marked as failed; in affected releases it does not start at all. Workaround: Set initialDeviceCount in system-profile.yaml to a lower value. This issue is fixed in v3.18.4, v3.20.2 and v3.21.1.

48205

When a large number of clone volumes are created from a small number of shared snapshots and a node is then restarted or replaced, some of those clone volumes can become stuck retrying volume creation, which can leave the node's protection groups unable to complete their rebuild. This issue is fixed in v3.20.2 and v3.21.1.

48041

In affected versions, when a management operation fails repeatedly, the delay between retries eventually drops to almost zero: after about 1.5 minutes of consecutive failures in v3.18.x, or about 16 hours in v3.19.x and v3.20.x. The continuous retries can keep a CPU core busy and flood the logs, pushing out older log entries needed to investigate the original failure. This issue is fixed in v3.18.4, v3.20.2, and v3.21.1.

48012

In rare cases, a volume or snapshot that is created while a server's node-manager service is down (or the server is powered off), and deleted just as node-manager recovers, can remain on that server. If the leftover is a clone, or a snapshot of a clone, deleting the snapshot it was created from never completes: that snapshot stays in Deleting state and blocks new operations on its source volume. A leftover object can also prevent a rebuild from completing. Data is not lost, and affected volumes remain accessible and continue to serve I/O. Workaround: Contact Lightbits Support to remove the leftover volume or snapshot. This issue is fixed in v3.18.4 and v3.21.1.

47398

After a node restarts, the node's reported power-up progress can show 100% before the node has finished powering up, because the value from before the restart is reported. Automation that waits for power-up progress to reach 100% may therefore continue while the node is still recovering. Workaround: Wait for the node to report an Active state instead of relying on power-up progress alone. This issue is fixed in v3.21.1.

47343

In rare cases, deleting a snapshot while a rebuild is in progress can raise DataIntegrityDuringRebuild events, and the affected data is not rebuilt to the secondary replica. The frontend log on the primary shows "Rebuild read detected a damaged object". This issue is fixed in v3.18.3, v3.20.2, and v3.21.1.

46869

If a snapshot is deleted while a rebuild is in progress, the rebuild can fail with log errors similar to "Exceeded maximum number of 50 write attempts". The failed rebuild is not retried automatically, so the affected protection group remains degraded. Workaround: Contact Lightbits Support to restart the node instance on which the rebuild failed. This issue is fixed in v3.18.3, v3.20.2, and v3.21.1.

46714

In rare cases, after the etcd service on a server is stopped and restarted, the node on that server may remain inactive and not rejoin the cluster within the expected time. This issue is fixed in v3.20.1, v3.20.2, and v3.21.1.

46207

In affected versions, when a server is deployed to listen on any IPv6 address ("::" or "[::]") and IP ACL enforcement is enabled, volumes that have a specific IPv4 ACL (rather than ALLOW_ANY) are not visible to hosts, even when the host's IPv4 address is in the ACL. The host connects to the subsystem, but no namespaces are listed. Workaround: Deploy the server to listen on a specific address, or on 0.0.0.0 instead of "::". This issue is fixed in v3.21.1.

46187

In affected versions, if one etcd member in the cluster is down, a cluster upgrade can fail to start with the error "Failed to take etcd snapshot: ... context deadline exceeded", depending on which member the pre-upgrade check selects. Workaround: Make sure all etcd members are running before starting a cluster upgrade. This issue is fixed in v3.20.2 and v3.21.1.

46100

On clusters that use TPM-based server-side encryption, rotating the cluster root key can fail on servers with slower physical TPM devices. The TPM operations needed for the rotation can take longer than the five-minute default timeout. When this happens, the rotation is rolled back and a ClusterRootKeyRotation event is raised with status Failed, cause FailedToComplete, and the description "cannot handle a cluster-kek-rotation task - internal error". Data and volumes are not affected. The cluster continues to use the previous cluster root key, and encryption remains enabled. Workaround: Retry the cluster root key rotation. Contact Lightbits Support if the failure persists. This issue is fixed in v3.21.1.

45998

In VMware environments, when the client kernel aborts NVMe commands sent to the cluster, nodes can briefly go down and come back up. This issue is fixed in v3.18.3, v3.20.2, and v3.21.1.

45912

If a storage node has some working and some stalled or failed replication connections to other nodes, clients may experience write I/O timeouts. Workaround: Restart the network services on the affected node. This issue is fixed in v3.21.1.

45625

After running the Ansible node cleanup, leftover journaling RAID (mdadm) devices can keep drives attached, so the next installation detects fewer NVMe drives than expected. Workaround: Stop the mdadm RAID device manually before reinstalling. This issue is fixed in v3.21.1.

44464

In affected versions, if the server selected as the new primary for an in-progress volume migration becomes permanently failed, the migration does not abort and instead retries indefinitely, repeatedly updating the cluster state. This issue is fixed in v3.21.1.

44334

In affected versions, stopping or restarting the node-manager service on a server (for example, during maintenance or an upgrade) can leave the data-path service hung while it waits for a configuration update that never arrives, which delays or blocks the server's shutdown. This issue is fixed in v3.18.3 and v3.21.1.