Known Issues in Lightbits 3.20.1

49019

In rare cases, after a node becomes active, the frontend service on this or another node could transiently fail a replication connection (with connection error: seastar::metrics::double_registration, automatically reestablished), thus leaving some stale stats under /mnt/stats. Subsequently, an access to the stats (manual or via lb_support) could cause a frontend segfault.

48930

In a rare situation where during a rebuild at least two snapshots are deleted, one containing data and a higher one containing a trim to the same blocks, the trim might be applied to the next existing version/snapshot - thus losing the valid data in it.

48806

In v3.20.1, a trim (deallocate) request issued while a volume's replicas are changing, for example, during a node failure, can be reported to the host as successful even though one replica did not apply it. No data is lost, but the trimmed blocks can remain allocated on that replica, so its contents for those blocks can differ from the other replicas until the blocks are written again. This issue is fixed in v3.20.2 and v3.21.1.

48646

A false UnrecoverableDataIntegrityError event with no volume ID, and "Rebuild read detected a damaged object" in the frontend log, could be reported during a rebuild when a volume with previously deleted snapshots is deleted. This issue is fixed in v3.20.2 and v3.21.1.

48621

In Lightbits v3.19.2 and later, when the cluster reaches its maximum number of stored events, automatic event cleanup can delete the wrong events: instead of the oldest events, it removes events by identifier order, which can silently erase whole periods of recent event history. Deleted events cannot be recovered.

48620

In container-based Lightbits deployments (v3.19.2 and later), support bundles collected on the host with the Ansible logs-collector role are incomplete: lbcli command output files are empty, the event and snapshot listings and the etcd JSON dump are missing, and no package list is collected. Workaround: Collect support logs with lbcli fetch logs (or the equivalent API call), which is not affected.

48605

In v3.18.3 and v3.20.1, upgrading a cluster with RPM packages from a release that ships etcd 3.5.18 (for example, v3.18.2 or v3.19.1) does not upgrade etcd. The cluster keeps running etcd 3.5.18 instead of the etcd 3.5.30 included in the release. To check a server, run "rpm -q etcd". This issue is fixed in v3.20.2 and v3.21.1; upgrading to a fixed version also upgrades etcd.

48546

When SSD Journaling is enabled and the journal devices are allocated from the same device pool as the data devices, the node-manager service can fail to start after a data drive is lost. No workaround is currently available.

48532

The node-manager service can fail to start if the number of drives present falls below the minimum required for the node's configured layout, for example, after a drive failure, power loss or planned maintenance. In earlier releases, the node started with the missing drive marked as failed; in affected releases it does not start at all. Workaround: Set initialDeviceCount in system-profile.yaml to a lower value. This issue is fixed in v3.18.4, v3.20.2, and v3.21.1.

48435

On some AMD EPYC servers, NVMe SSDs can report I/O timeouts. This can cause a storage node to be marked as failed, and while the node is out of service, volumes with reduced protection can become read-only. Workaround: Disable PCIe Relaxed Ordering in the server BIOS.

48205

When a large number of clone volumes are created from a small number of shared snapshots and a node is then restarted or replaced, some of those clone volumes can become stuck retrying volume creation, which can leave the node's protection groups unable to complete their rebuild. This issue is fixed in v3.20.2 and v3.21.1.

48171

When a server becomes read-only after running out of metadata capacity, the proactive rebalance feature does not move any volumes or snapshots off that server, even when other servers have free space. The server cannot recover on its own, and volumes with a replica on it remain in Degraded or ReadOnly protection state until capacity is freed manually. This affects all currently released versions. Workaround: Free capacity on the affected server, for example, by deleting unneeded volumes or snapshots that have a replica on it, or contact Lightbits Support.

48041

In affected versions, when a management operation fails repeatedly, the delay between retries eventually drops to almost zero: after about 1.5 minutes of consecutive failures in v3.18.x, or about 16 hours in v3.19.x and v3.20.x. The continuous retries can keep a CPU core busy and flood the logs, pushing out older log entries needed to investigate the original failure. This issue is fixed in v3.18.4, v3.20.2, and v3.21.1.

48012

In rare cases, a volume or snapshot that is created while a server's node-manager service is down (or the server is powered off), and deleted just as node-manager recovers, can remain on that server. If the leftover is a clone, or a snapshot of a clone, deleting the snapshot it was created from never completes: that snapshot stays in Deleting state and blocks new operations on its source volume. A leftover object can also prevent a rebuild from completing. Data is not lost, and affected volumes remain accessible and continue to serve I/O. Workaround: Contact Lightbits Support to remove the leftover volume or snapshot. This issue is fixed in v3.18.4 and v3.21.1.

47894

On clusters that use volume IP access control lists, if a client IP address is removed from a volume's IP ACL, or the volume is deleted, while the node-manager service on a server is stopped or restarting (for example, during a rolling upgrade), the client keeps network access to that server's NVMe and discovery ports after node-manager starts again. The access remains until the server is rebooted. Workaround: Avoid changing volume IP ACLs or deleting volumes while node-manager is stopped or an upgrade is in progress; if this has happened, reboot the affected server to remove the stale access rules.

47753

Changing a node's Permanent Failure (PF) threshold can behave unexpectedly if the node is already Inactive. Lowering the threshold to a value that has not yet elapsed does not take effect until the original threshold time passes, and raising the threshold can prevent the node from reaching Permanent Failure at all. Workaround: Change a node's PF threshold before maintenance that will make it inactive. To shorten the threshold of a node that is already inactive, use a value that has already elapsed.

47343

In rare cases, deleting a snapshot while a rebuild is in progress can raise DataIntegrityDuringRebuild events, and the affected data is not rebuilt to the secondary replica. The frontend log on the primary shows "Rebuild read detected a damaged object". This issue is fixed in v3.18.3, v3.20.2, and v3.21.1.

47300

If the cluster manager's connection to the cluster's internal coordination service is interrupted (for example, due to a leader change, network disruption, or a brief service restart), the cluster manager's view of cluster servers can stop updating entirely, with no automatic recovery. Workaround: Restart the cluster-manager service to restore normal operation.

46633

On large clusters (for example, clusters with more than 1,000 protection groups), evicting a server can fail with a timeout before any data has been moved, because the cluster manager takes a long time to prepare the eviction. Workaround: Increase the EvictionNoProgressTimeout cluster configuration parameter. Contact Lightbits Support for the recommended value for your cluster.

46207

In affected versions, when a server is deployed to listen on any IPv6 address ("::" or "[::]") and IP ACL enforcement is enabled, volumes that have a specific IPv4 ACL (rather than ALLOW_ANY) are not visible to hosts, even when the host's IPv4 address is in the ACL. The host connects to the subsystem, but no namespaces are listed. Workaround: Deploy the server to listen on a specific address, or on 0.0.0.0 instead of "::". This issue is fixed in v3.21.1.

46187

In affected versions, if one etcd member in the cluster is down, a cluster upgrade can fail to start with the error "Failed to take etcd snapshot: ... context deadline exceeded", depending on which member the pre-upgrade check selects. Workaround: Make sure all etcd members are running before starting a cluster upgrade. This issue is fixed in v3.20.2 and v3.21.1.

46100

On clusters that use TPM-based server-side encryption, rotating the cluster root key can fail on servers with slower physical TPM devices. The TPM operations needed for the rotation can take longer than the five-minute default timeout. When this happens, the rotation is rolled back and a ClusterRootKeyRotation event is raised with status Failed, cause FailedToComplete, and the description "cannot handle a cluster-kek-rotation task - internal error". Data and volumes are not affected. The cluster continues to use the previous cluster root key, and encryption remains enabled. Workaround: Retry the cluster root key rotation. Contact Lightbits Support if the failure persists. This issue is fixed in v3.21.1.

45729

When SSD Journaling is configured with multiple NVMe devices, cluster cleanup does not remove the RAID (mdadm) arrays created by Lightbits. Before re-deploying on the same servers, manually remove these arrays using mdadm.

45712

In rare cases, if the internal connection used to track volume or snapshot state changes is interrupted (for example, due to a leader change, network interruption, or a brief service restart), a volume or snapshot update can remain stuck showing as still in progress even though the underlying change has completed. Workaround: Restart node-manager on the affected server to resolve the stuck state.

45625

After running the Ansible node cleanup, leftover journaling RAID (mdadm) devices can keep drives attached, so the next installation detects fewer NVMe drives than expected. Workaround: Stop the mdadm RAID device manually before reinstalling. This issue is fixed in v3.21.1.

45202

When a volume is created with lbcli, any part of a label key before a hyphen is dropped. For example, --labels="pve-vmid=100" is saved with the key "vmid" instead of "pve-vmid". Workaround: Avoid hyphens in label keys, or create the volume through the REST API, which is not affected.

45157

During a node power-up after an abrupt shutdown, the node's reported power-up progress can reach 100%, drop back to 50-60%, and then climb to 100% again. This affects the reported percentage only. Recovery completes correctly, and the node returns to service only when it is actually done. No workaround is needed.

44642

Update volume commands issued from VCP V1.4.0 (or older) to a Lightbits cluster running Lightbits 3.19.1 release or newer will fail due to invalid arguments. Running with Lightbits releases 3.20.1/3.21.1 or newer will require updating restRequestCompatibility in api-service yaml to true and restarting the service.

44464

In affected versions, if the server selected as the new primary for an in-progress volume migration becomes permanently failed, the migration does not abort and instead retries indefinitely, repeatedly updating the cluster state. This issue is fixed in v3.21.1.

44386

Grafana dashboards may display a short spike of negative BW/Throughput rate, following the stoppage of a Lightbits service.

44383

When cluster-level encryption is enabled, deleting a server that hosts the active Cluster Manager (CM) could prevent subsequent server additions from completing successfully. After removing such a server from the cluster, stop the CM service on it or shut the server down.

44334

In affected versions, stopping or restarting the node-manager service on a server (for example, during maintenance or an upgrade) can leave the data-path service hung while it waits for a configuration update that never arrives, which delays or blocks the server's shutdown. This issue is fixed in v3.18.3 and v3.21.1.

43471

[Log Streaming] Log lines longer than the destination rsyslog server's maxMessageSize (default 8 KB) are truncated by the receiving rsyslog. To stream full-length lines (Lightbits logs can reach ~48 KB), increase maxMessageSize on the rsyslog server — e.g. global(maxMessageSize="64k") before any inputs — and restart rsyslog. This setting is on the rsyslog server and cannot be configured from Lightbits.

40883

When using the vCenter Plugin (VCP) to upgrade a cluster to Lightbits 3.17.1 or later, the upgrade fails because VCP cannot parse the updated version format. To complete the upgrade, use the Lightbits core CLI or REST API directly.

37505

In a rare combination of events, the physicalOwnedCapacity volume statistic may report an incorrect value when data at a specific LBA is overwritten with content that has a different compression ratio. This affects the reported statistic only; stored data and I/O are unaffected.