fix: prevent false DeadMaster and drain MariaDB relay logs (#106)#108
fix: prevent false DeadMaster and drain MariaDB relay logs (#106)#108renecannao wants to merge 4 commits into
Conversation
Spec for issue #106: IO-running-aware analysis, MariaDB-safe relay drain, skip undrainable candidates, validate before re-point, and functional CI. Signed-off-by: Rene Cannao <rene@proxysql.com>
Treat valid replicas with a running IO thread as UnreachableMaster rather than DeadMaster when the master check fails. Add MariaDB-safe DrainRelayLogs (SQL thread only), skip undrainable promotion candidates, and validate the candidate before re-pointing siblings. Include unit tests, MariaDB CI job, and docs updates. Signed-off-by: Rene Cannao <rene@proxysql.com>
|
Warning Review limit reached
Next review available in: 7 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (19)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![gemini-code-assist[bot]](https://cdn.github.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request addresses false DeadMaster detections by treating replicas with running IO threads as UnreachableMaster and introduces a safe relay log draining mechanism (DrainRelayLogs) for MariaDB to prevent data loss during failover. The review feedback highlights several critical issues, including potential nil pointer dereferences when reading topology instances, inconsistent error return values in DrainRelayLogs, a logic bypass in candidate selection when no candidate is chosen, and unused helper code in analysis_classification.go.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| instance, err = ReadTopologyInstance(instanceKey) | ||
| log.Infof("Drained relay logs on %+v, Self:%+v, Exec:%+v", *instanceKey, instance.SelfBinlogCoordinates, instance.ExecBinlogCoordinates) | ||
| return instance, err |
There was a problem hiding this comment.
If ReadTopologyInstance fails and returns a nil instance along with an error, accessing instance.SelfBinlogCoordinates and instance.ExecBinlogCoordinates in the log statement will cause a panic.
The error should be handled or checked before accessing the fields of instance.
instance, err = ReadTopologyInstance(instanceKey)
if err != nil {
return instance, log.Errore(err)
}
log.Infof("Drained relay logs on %+v, Self:%+v, Exec:%+v", *instanceKey, instance.SelfBinlogCoordinates, instance.ExecBinlogCoordinates)
return instance, nil| if _, err := ExecInstance(instanceKey, instance.QSP.start_slave_sql_thread()); err != nil { | ||
| return nil, log.Errorf("%+v: DrainRelayLogs: start sql thread failed: %+v", *instanceKey, err) | ||
| } |
There was a problem hiding this comment.
Returning nil on error is inconsistent with other error paths in this function (which return instance, err). Returning instance is safer and maintains consistency.
| if _, err := ExecInstance(instanceKey, instance.QSP.start_slave_sql_thread()); err != nil { | |
| return nil, log.Errorf("%+v: DrainRelayLogs: start sql thread failed: %+v", *instanceKey, err) | |
| } | |
| if _, err := ExecInstance(instanceKey, instance.QSP.start_slave_sql_thread()); err != nil { | |
| return instance, log.Errorf("%+v: DrainRelayLogs: start sql thread failed: %+v", *instanceKey, err) | |
| } |
| // classifyUnreachableMasterWhenIORunning returns UnreachableMaster when the | ||
| // master check failed but at least one replica still has IO running. | ||
| // Returns NoProblem when this rule does not apply (caller continues other branches). | ||
| func classifyUnreachableMasterWhenIORunning(isMaster bool, lastCheckValid bool, countValidIORunningReplicas uint) AnalysisCode { |
There was a problem hiding this comment.
The helper function classifyUnreachableMasterWhenIORunning is defined and tested but is never used in the actual classification logic in analysis_dao.go (which implements the same logic inline).
To improve maintainability and avoid dead code, either use this helper function in analysis_dao.go or remove it.
| if err != nil || candidateReplica == nil { | ||
| return candidateReplica, aheadReplicas, equalReplicas, laterReplicas, cannotReplicateReplicas, err | ||
| } |
There was a problem hiding this comment.
If chooseCandidateReplica returns candidateReplica == nil (with err == nil) on the first iteration, the function returns immediately. This bypasses the descriptive error at the end of the function for forRematchPurposes.
The check should handle candidateReplica == nil when forRematchPurposes is true by returning the appropriate error.
if err != nil {
return nil, aheadReplicas, equalReplicas, laterReplicas, cannotReplicateReplicas, err
}
if candidateReplica == nil {
if forRematchPurposes {
return nil, aheadReplicas, equalReplicas, laterReplicas, cannotReplicateReplicas, fmt.Errorf("GetCandidateReplica: no drainable candidate replica found for %+v", *masterKey)
}
return nil, aheadReplicas, equalReplicas, laterReplicas, cannotReplicateReplicas, nil
}| refreshed, readErr := inst.ReadTopologyInstance(&candidate.Key) | ||
| if readErr != nil { | ||
| return readErr | ||
| } | ||
| *candidate = *refreshed |
There was a problem hiding this comment.
If ReadTopologyInstance succeeds but returns a nil pointer for refreshed (along with a nil error), dereferencing it (*refreshed) will cause a panic.
A nil check on refreshed should be performed before dereferencing.
| refreshed, readErr := inst.ReadTopologyInstance(&candidate.Key) | |
| if readErr != nil { | |
| return readErr | |
| } | |
| *candidate = *refreshed | |
| refreshed, readErr := inst.ReadTopologyInstance(&candidate.Key) | |
| if readErr != nil { | |
| return readErr | |
| } | |
| if refreshed == nil { | |
| return fmt.Errorf("DelayMasterPromotionIfSQLThreadNotUpToDate: refreshed instance is nil for %+v", candidate.Key) | |
| } | |
| *candidate = *refreshed |
Signed-off-by: Rene Cannao <rene@proxysql.com>
Always re-read/drain candidates instead of trusting stale SQLThreadUpToDate. Lengthen drain wait, require up-to-date after drain, and fix functional test recovery API parsing plus wait-for-relay readiness before failover. Signed-off-by: Rene Cannao <rene@proxysql.com>
1 participant
Summary
Fixes #106
UnreachableMasterinstead ofDeadMasterwhen the master check fails (SQL stopped alone is not proof the master is dead).DrainRelayLogsstarts the SQL thread only (no IO stop/start), waits for catch-up, then stops replication. MariaDB nice-stop now uses this path so relay events are applied before promotion.GetCandidateReplicatries the next replica if drain fails (e.g. Error 1062); fails recovery if none can drain.RegroupReplicasGTIDWithValidation).test-mariadb-relay-drain.sh, MariaDB 10.6/10.11 CI job, docs updates.Design / plan
docs/superpowers/specs/2026-07-16-mariadb-deadmaster-relay-drain-design.mddocs/superpowers/plans/2026-07-16-mariadb-deadmaster-relay-drain.mdTest plan
go test ./go/inst/ -count=1go build ./go/cmd/orchestratorfunctional-mariadb(10.6, 10.11)