- Add UpdateVersionCache method to JetStreamEventStore for cache synchronization - Add SubscribeToEventStored convenience helper to NATSEventBus - Create integration tests for cross-node broadcasting scenarios - Add example demonstrating NATSEventBus + JetStreamEventStore integration - Tests verify: - Single-node broadcasting works - Multi-node event flow works - Version cache consistency across nodes - Namespace isolation maintained - EventStored subscription works correctly
2.5 KiB
2.5 KiB
Issue: Implement Actor Migration Between Cluster Nodes
Problem
When nodes join or leave the cluster, actors need to be migrated to maintain even distribution. Currently:
handleRebalanceRequestincluster/manager.go:150is emptyhandleMigrationRequestincluster/manager.go:167is emptyRebalanceShardsincluster/shard.go:211returns unchanged mapSendMessageincluster/distributed.go:139ignores sharding
Required Implementation
1. Rebalance Algorithm (cluster/shard.go)
Implement ConsistentHashPlacement.RebalanceShards to:
- Calculate new shard assignments based on active nodes
- Identify actors needing migration
- Generate migration plan with source/dest nodes
2. Migration Coordinator (cluster/manager.go)
Implement handleRebalanceRequest to:
- Accept migration plan from leader
- For each actor in plan:
- Pause incoming messages
- Capture actor state (replay events up to current version)
- Serialize state
- Send migration request to destination node
- Wait for ack
- Delete actor from current node
- Track migration status via
ActorMigration.Status
3. Cross-Node Message Routing (cluster/distributed.go)
Implement proper routing in SendMessage:
- Use
GetActorNode(actorID)to determine target node - If remote: marshal message, send via NATS to target node
- If local: send to local runtime
- Route response back to caller if needed
Suggested Approach
- Define message types for actor migration requests/responses in
cluster/types.go - Implement state capture - replay events to get current state
- Implement state restore - deserialize and restore actor state
- Implement coordinator - manage migration phases
- Add error handling - handle failed migrations, retries, cleanup
- Add tests - test migration with mock NATS
Related Files
cluster/manager.go:150- handleRebalanceRequest (empty)cluster/manager.go:167- handleMigrationRequest (empty)cluster/shard.go:211- RebalanceShards (stub)cluster/distributed.go:139- SendMessage (simplified)cluster/types.go:108- ActorMigration struct
Acceptance Criteria
RebalanceShardsreturns new shard map with actor assignmentshandleRebalanceRequestprocesses migration planhandleMigrationRequestaccepts actor migrationsSendMessageroutes to correct node- Actors can be migrated with state preserved
- Failed migrations are handled gracefully
- Integration test with multi-node cluster