Page MenuHomePhabricator
Create Task
Maniphest T193668

Transaction should be in the callback stage (not 'cursory')
Closed, ResolvedPublicPRODUCTION ERROR
Actions

Description

Saw a flood of these errors in fatalmonitor following the rollout of 1.32.0-wmf.2 to group1

Stacktrace:

#0 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(267): Wikimedia\Rdbms\LoadBalancer->runMasterTransactionListenerCallbacks()
#1 [internal function]: Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2(Wikimedia\Rdbms\LoadBalancer)
#2 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactoryMulti.php(425): call_user_func_array(Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2;4828, array)
#3 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(269): Wikimedia\Rdbms\LBFactoryMulti->forEachLB(Closure$Wikimedia\Rdbms\LBFactory::commitMasterChanges#2;4828)
#4 /srv/mediawiki/php-1.32.0-wmf.2/includes/libs/rdbms/lbfactory/LBFactory.php(445): Wikimedia\Rdbms\LBFactory->commitMasterChanges(string)
#5 /srv/mediawiki/php-1.32.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(290): Wikimedia\Rdbms\LBFactory->commitAndWaitForReplication(string, integer)
#6 /srv/mediawiki/php-1.32.0-wmf.2/includes/jobqueue/jobs/RefreshLinksJob.php(122): RefreshLinksJob->runForTitle(Title)
#7 /srv/mediawiki/php-1.32.0-wmf.2/extensions/EventBus/includes/JobExecutor.php(60): RefreshLinksJob->run()
#8 /srv/mediawiki/rpc/RunSingleJob.php(79): JobExecutor->execute(array)
#9 {main}

Details

SubjectRepoBranchLines +/-
rdbms: fix finalization stage errors in LBFactory::commitMasterChangesmediawiki/coremaster+71 -12
Use AutoCommitUpdate in LinksUpdate::doUpdatemediawiki/coremaster+6 -5
Make DeferredUpdates avoid running during LBFactory::commitMasterChangesmediawiki/coremaster+65 -9
Revert "rdbms: enforce and improve LBFactory/LoadBalancer callback handling"mediawiki/corewmf/1.32.0-wmf.2+179 -409
Customize query in gerrit

Related Objects

Event Timeline

thcipriani created this task.May 2 2018, 7:28 PM
Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMay 2 2018, 7:28 PM
thcipriani triaged this task as Unbreak Now! priority.May 2 2018, 7:29 PM
thcipriani added a parent task: T191048: 1.32.0-wmf.2 deployment blockers.

Added as train blocker setting as UBN!

Restricted Application added subscribers: Liuxinyu970226, TerraCodes. · View Herald TranscriptMay 2 2018, 7:30 PM
thcipriani added a subscriber: aaron.May 2 2018, 7:44 PM

Adding @aaron since it looks like the messages were added in https://gerrit.wikimedia.org/r/#/c/422543/

Aklapper added a project: MediaWiki-libs-Rdbms.May 2 2018, 9:58 PM
Nikerabbit added a parent task: T41480: [DO NOT USE] Issues affecting translatewiki.net [superseded by #affects-translatewiki.net].May 3 2018, 12:07 PM
Nikerabbit subscribed.
gerritbot subscribed.May 3 2018, 8:47 PM

Change 430507 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@wmf/1.32.0-wmf.2] Revert "rdbms: enforce and improve LBFactory/LoadBalancer callback handling"

https://gerrit.wikimedia.org/r/430507

gerritbot added a project: Patch-For-Review.May 3 2018, 8:47 PM
gerritbot added a comment.May 3 2018, 8:59 PM

Change 430507 merged by jenkins-bot:
[mediawiki/core@wmf/1.32.0-wmf.2] Revert "rdbms: enforce and improve LBFactory/LoadBalancer callback handling"

https://gerrit.wikimedia.org/r/430507

ReleaseTaggerBot added a project: MW-1.32-notes (WMF-deploy-2018-05-01 (1.32.0-wmf.2)).May 3 2018, 9:00 PM
Stashbot subscribed.May 3 2018, 9:08 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-03T21:08:36Z] <aaron@tin> Started scap: Deploy rMWdb9acea7eb1c (bug T193668)

gerritbot added a comment.May 3 2018, 9:39 PM

Change 430532 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

gerritbot added a comment.May 3 2018, 10:11 PM

Change 430532 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

gerritbot added a comment.May 3 2018, 10:37 PM

Change 430532 merged by jenkins-bot:
[mediawiki/core@master] Make DeferredUpdates avoid running during LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/430532

Stashbot added a comment.May 3 2018, 10:52 PM

Mentioned in SAL (#wikimedia-operations) [2018-05-03T22:52:27Z] <aaron@tin> Finished scap: Deploy rMWdb9acea7eb1c (bug T193668) (duration: 103m 50s)

ReleaseTaggerBot edited projects, added MW-1.32-notes (WMF-deploy-2018-05-08 (1.32.0-wmf.3)); removed MW-1.32-notes (WMF-deploy-2018-05-01 (1.32.0-wmf.2)).May 3 2018, 11:00 PM
gerritbot added a comment.May 3 2018, 11:47 PM

Change 430273 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] Use AutoCommitUpdate in LinksUpdate::doUpdate

https://gerrit.wikimedia.org/r/430273

gerritbot added a comment.May 4 2018, 12:40 AM

Change 430273 merged by jenkins-bot:
[mediawiki/core@master] Use AutoCommitUpdate in LinksUpdate::doUpdate

https://gerrit.wikimedia.org/r/430273

gerritbot added a comment.May 5 2018, 10:50 PM

Change 431279 had a related patch set uploaded (by Aaron Schulz; owner: Aaron Schulz):
[mediawiki/core@master] rdbms: fix finalization stage errors in LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/431279

greg lowered the priority of this task from Unbreak Now! to High.May 7 2018, 4:14 PM
greg subscribed.

No longer UBN! as this was rollback. Removing as blocker (obviously as we're done with wmf.2 now).

greg removed a parent task: T191048: 1.32.0-wmf.2 deployment blockers.May 7 2018, 4:14 PM
gerritbot added a comment.May 7 2018, 7:39 PM

Change 431279 merged by jenkins-bot:
[mediawiki/core@master] rdbms: fix finalization stage errors in LBFactory::commitMasterChanges

https://gerrit.wikimedia.org/r/431279

Nikerabbit added a comment.Jun 14 2018, 9:36 AM

What's the plan now? Can this be closed?

aaron closed this task as Resolved.Jun 14 2018, 7:18 PM
aaron claimed this task.
Krinkle edited projects, added Performance-Team; removed Patch-For-Review.Jun 14 2018, 9:23 PM
Liuxinyu970226 unsubscribed.Jun 15 2018, 3:34 PM
• Vvjjkkii renamed this task from Transaction should be in the callback stage (not 'cursory') to nrdaaaaaaa.Jul 1 2018, 1:12 AM
• Vvjjkkii reopened this task as Open.
• Vvjjkkii removed aaron as the assignee of this task.
• Vvjjkkii added projects: CheckUser, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), Tamil-Sites, Gamepress, Hashtags, Jade, KartoEditor, Language-2018-Apr-June, New-Editor-Experiences, Mail, TCB-Team (now WMDE-TechWish).
• Vvjjkkii updated the task description. (Show Details)
• Vvjjkkii removed subscribers: gerritbot, Aklapper.
CommunityTechBot renamed this task from nrdaaaaaaa to Transaction should be in the callback stage (not 'cursory').Jul 2 2018, 5:21 AM
CommunityTechBot closed this task as Resolved.
CommunityTechBot assigned this task to aaron.
CommunityTechBot updated the task description. (Show Details)
CommunityTechBot removed projects: TCB-Team (now WMDE-TechWish), Mail, New-Editor-Experiences, Language-2018-Apr-June, KartoEditor, Jade, Hashtags, Gamepress, Tamil-Sites, Connected-Open-Heritage-Batch-uploads (RAÄ-KMB_1_2017-02), CheckUser.
CommunityTechBot added subscribers: gerritbot, Aklapper.
• mmodell changed the subtype of this task from "Task" to "Production Error".Aug 28 2019, 11:09 PM
• Phabricator_maintenance added a project: affects-translatewiki.net.Sep 26 2019, 1:53 PM
• Phabricator_maintenance removed a parent task: T41480: [DO NOT USE] Issues affecting translatewiki.net [superseded by #affects-translatewiki.net].Sep 26 2019, 1:55 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits