Maniphest T191056

1.32.0-wmf.10 deployment blockers
Closed, ResolvedPublicRelease
Actions

Assigned To

Authored By

	greg
	Mar 29 2018, 6:08 PM

Details

Release Version: 1.32.0-wmf.10
Release Date: Jun 25 2018, 12:00 AM

2018 week 26 1.32-wmf.10 Changes wmf/1.32.0-wmf.10

This MediaWiki Train Deployment is scheduled for the week of Monday, June 25th:

Monday June 25th	Tuesday, June 26th	Wednesday, June 27th	Thursday, June 28th	Friday
Backports only.	Branch `wmf.10` and deploy to Group 0 Wikis.	Deploy `wmf.10` to Group 1 Wikis.	Deploy `wmf.10` to all Wikis.	No deployments on fridays

See https://wikitech.wikimedia.org/wiki/Deployments for full schedule.

How this works

Any serious bugs affecting wmf.10 should be added as subtasks beneath this one.
- Use this form to create one.
Any open subtask(s) block the train from moving forward. This means no further deployments until the blockers are resolved.
If something is serious enough to warrant a rollback then you should bring it to the attention of deployers on the #wikimedia-operations IRC channel.
If you have a risky change in this week's train add a comment to this task using the Risky patch template
For more info about deployment blockers, see Holding the train.

Other Deployments

Previous: 1.32.0-wmf.9

Next: 1.32.0-wmf.11

	Subject	Repo	Branch	Lines +/-
	Be able to over-ride wmgUseMwEmbedSupport from LocalSettings.php	operations/mediawiki-config	master	+1 -1

Customize query in gerrit

Related Objects
Search...

Status	Subtype	Assigned	Task
Resolved	Release	dduvall	T191056 1.32.0-wmf.10 deployment blockers
Resolved		greg	T198422 Running scap sync-dir php-1.32.0-wmf.10 fails due to syntax error
Resolved		aaron	T198483 Save Timing increased 50% since 2018-06-28 20:53
Resolved		matej_suchanek	T198449 Rollback from autopatrolled user was marked as not patrolled

Event Timeline

greg created this task.Mar 29 2018, 6:08 PM

Restricted Application added a subscriber: Aklapper. · View Herald TranscriptMar 29 2018, 6:08 PM

greg triaged this task as Medium priority.Mar 29 2018, 6:16 PM

greg assigned this task to dduvall.Jun 11 2018, 10:45 PM

Restricted Application added a project: Release-Engineering-Team (Kanban). · View Herald TranscriptJun 11 2018, 10:45 PM

dduvall moved this task from Backlog to In-progress on the Release-Engineering-Team (Kanban) board.Jun 26 2018, 3:47 PM

Trace of l10update failure (see related task and SAL message):

17:59:05 Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
17:59:06 Copying to deploy1001.eqiad.wmnet from deploy1001.eqiad.wmnet
17:59:06 Started rsync common
cannot delete non-empty directory: php-1.31.0-wmf.28/cache/l10n
cannot delete non-empty directory: php-1.31.0-wmf.28/cache/l10n
cannot delete non-empty directory: php-1.31.0-wmf.28/cache
cannot delete non-empty directory: php-1.31.0-wmf.28/cache
cannot delete non-empty directory: php-1.31.0-wmf.28
17:59:28 Finished rsync common (duration: 00m 22s)
17:59:28 Started cache_git_info
18:00:03 Finished cache_git_info (duration: 00m 35s)
18:00:04 Started l10n-update
18:00:04 Updating ExtensionMessages-1.32.0-wmf.8.php
18:00:05 Updating LocalisationCache for 1.32.0-wmf.8 using 30 thread(s)
18:01:27 Generating JSON versions and md5 files
18:01:27 Creating empty /srv/mediawiki-staging/wmf-config/ExtensionMessages-1.32.0-wmf.10.php
18:01:27 Bootstrapping l10n cache for 1.32.0-wmf.10
18:01:28 Last output:
[Tue Jun 26 18:01:28 2018] [hphp] [433:7f2d762fa3c0:0:000001] []
Fatal error: Uncaught exception 'Exception' with message '/srv/mediawiki-staging/php-1.32.0-wmf.10/extensions/MwEmbedSupport/extension.json
 does not exist!' in /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/registration/ExtensionRegistry.php:105
Stack trace:
#0 /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/GlobalFunctions.php(50): ExtensionRegistry->queue()
#1 /srv/mediawiki/wmf-config/CommonSettings.php(904): wfLoadExtension()
#2 /srv/mediawiki-staging/php-1.32.0-wmf.10/LocalSettings.php(4): include()
#3 /srv/mediawiki-staging/php-1.32.0-wmf.10/includes/Setup.php(100): include()
#4 /srv/mediawiki-staging/php-1.32.0-wmf.10/maintenance/doMaintenance.php(81): include()
#5 /srv/mediawiki-staging/php-1.32.0-wmf.10/maintenance/rebuildLocalisationCache.php(181): include()
#6 /srv/mediawiki-staging/multiversion/MWScript.php(100): include()
#7 {main}

Jdforrester-WMF subscribed.Jun 26 2018, 6:27 PM

Mentioned in SAL (#wikimedia-operations) [2018-06-26T18:39:41Z] <marxarelli> setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056)

Change 442163 had a related patch set uploaded (by Dduvall; owner: Jforrester):
[operations/mediawiki-config@master] Be able to over-ride wmgUseMwEmbedSupport from LocalSettings.php

https://gerrit.wikimedia.org/r/442163

gerritbot added a project: Patch-For-Review.Jun 26 2018, 6:50 PM

Change 442163 merged by jenkins-bot:
[operations/mediawiki-config@master] Be able to over-ride wmgUseMwEmbedSupport from LocalSettings.php

https://gerrit.wikimedia.org/r/442163

Now cdb-rebuild failed on mwdebug2002 due to lack of disk space.

19:40:01 Failure processing (u'/srv/mediawiki/php-1.32.0-wmf.10/cache/l10n', u'l10n_cache-mt.cdb', True)
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 501, in update_l10n_cdb_wrapper
    return update_l10n_cdb(*args)
  File "/usr/lib/python2.7/dist-packages/scap/utils.py", line 402, in context_wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/scap/tasks.py", line 477, in update_l10n_cdb
    writer.put(key.encode('utf-8'), value.encode('utf-8'))
  File "/usr/lib/python2.7/dist-packages/scap/cdblib.py", line 128, in put
    self.fp.write(value)
IOError: [Errno 28] No space left on device

Since the sync succeeded other than cdb update on mwdeploy2002, I'm going to move ahead with group0 today.

Manually cleaned up old deployments (wmf.3, wmf.4, wmf.5) to ensure enough disk space. I'm re-syncing now.

MarcoAurelio subscribed.Jun 26 2018, 10:04 PM

Rolling back group1 due to T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment blocker.

Jdforrester-WMF removed a subtask: T197918: Archive the MwEmbedSupport extension.Jun 27 2018, 9:24 PM

I think commons data loss occurring from Wednesday could be related to train? T198177

Current plan for today's train rollout is mentioned in {T198350#4322794}. It'll be a graduated rollout of (group1 - commonswiki), then commonswiki, then time for thorough verification that the issue is resolved, then group2 time permitting.

MarcoAurelio unsubscribed.Jun 28 2018, 6:05 PM

dduvall closed this task as Resolved.Jun 28 2018, 9:14 PM

dduvall removed a subtask: T198350: Rising lock wait timeout SQL errors upon 1.32.0-wmf.10 group1 deployment.

Krinkle added a subtask: T198422: Running scap sync-dir php-1.32.0-wmf.10 fails due to syntax error.Jun 28 2018, 9:22 PM

• matmarex added a subtask: T198449: Rollback from autopatrolled user was marked as not patrolled.Jun 29 2018, 8:10 PM

• matmarex closed subtask T198449: Rollback from autopatrolled user was marked as not patrolled as Resolved.Jul 2 2018, 7:37 PM

aaron closed subtask T198483: Save Timing increased 50% since 2018-06-28 20:53 as Resolved.Jul 10 2018, 9:17 AM

greg closed subtask T198422: Running scap sync-dir php-1.32.0-wmf.10 fails due to syntax error as Resolved.Jan 23 2019, 4:25 PM