Incident Report: Invalid KeyPackage Handling & Group Membership Failures

Engineering updates
1

Date: March 28, 2025
Impact: Broken group experience for groups with members with old installations that had invalid KeyPackages
Status: Resolved

Summary

A race condition in key package validation logic led to inconsistent group membership states, including broken or forked groups.

The bug was triggered by improper mapping between installation IDs and their associated key packages, leading to non-deterministic behavior in group syncs and conversation creation across multiple partners. In addition, a related cleanup oversight in the soft-fail mechanism for key packages led to incorrect membership validation during user removals.

Root Causes

Problem 1: Unordered KeyPackage Mapping

  • The application fetched key packages for all installations linked to users being added to a conversation or group.

  • A mapping bug caused random associations between installation IDs and their key package status.

  • Valid key packages were marked as invalid or vice versa.

  • Since verification depended on these mappings, different users observed inconsistent group states—triggering validation errors and forked groups.

  • This was particularly problematic for users with older or expired key packages.

Problem 2: Incomplete Cleanup on Removal

After fixing the above, testing revealed a second issue:

  • When a user with invalid key packages was added and later removed, their failed installations were not excluded from the verification checks.

  • This led to false mismatches in expected vs. actual installations removed.

  • The system attempted to validate removal against installations that were never successfully added.

Fixes Deployed

  1. Deterministic Mapping Fix:

    • Ensured key package health is accurately mapped to the correct installation IDs in a deterministic order.

  2. Failed Installation Cleanup:

    • On member removal, the system now excludes failed installations from expected installation checks.

    • Also removes failed installations from group state to avoid residual noise.

  3. Test Coverage Expansion:

    • Added comprehensive test cases around keypackage validation and group modification flows, especially for edge cases.