Jump to content
Server Admin Log/Archive 35
23:35 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/Html.php: I67ceb34eabf2f T200506 (duration: 00m 50s)
23:00 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/WikidataPageBanner/: I9dc9a4 T199855 (duration: 00m 51s)
21:14 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/deferred/: Id2852d73d00
21:11 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/jobqueue/jobs/: Id2852d73d00
19:46 mutante: right when it was fixed on ms-be2043 it also broke on ms-be2040. following the same instructions to fix xfs in a root screen (T199198 )
19:12 mutante: ms-be2043 - following instructions at https://wikitech.wikimedia.org/wiki/Graphite#Repair_xfs_misreporting_free_space to repair xfs misreporting free space (T199198 ), fixing docs, icinga-downtime doesn't want fqdn but short name
18:10 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/includes/Title.php: Hot-deploy of I05eea553c58 Copyright again (duration: 00m 50s)
17:57 mutante: depooled wtp2020 because icinga reported memory errors
17:39 SMalyshev: repooled wdqs1005
16:30 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/VisualEditor/modules/ve-mw/init/ve.init.mw.Target.js: Hot-deploy of I38eda4aac48 T203213 (duration: 00m 54s)
10:49 moritzm: installing libgd2 security updates on trusty
09:46 moritzm: installing libx11 security updates on trusty
08:57 gehel: elasticsearch data directory migration on all logstash nodes
08:16 godog: repair sde1 on ms-be2041 - T199198
05:54 elukey: resumed the Hadoop workers reboots for kernel upgrades
05:33 elukey: restart pdfrender on scb1003
00:44 mutante: netmon1002 - restarted smokeping, removed radon as target (unblock decome of former dns server), added cobalt instead as a target also in C4
23:42 mutante: vega - removed rsync config and let puppet regenerate it
23:38 mutante: bromine - remove outdated rsync conf fragment for static-bugzilla, stopping rsync, running puppet
23:36 mutante: releases1001 - killing outdated rsync processes for releases from bromine
22:57 jforrester@deploy1001: Finished scap: Full sync for i18n re-build following Ieaded578ffd
22:25 jforrester@deploy1001: Started scap: Full sync for i18n re-build following Ieaded578ffd
22:13 volans: this was a test for the logging to IRC, please ignore ^^^^
22:12 END: (NOTRUN) - Cookbook sre.switchdc.mediawiki.00-disable-puppet (exit_code=0) (switchdc/volans@sarin)
22:12 START: - Cookbook sre.switchdc.mediawiki.00-disable-puppet (switchdc/volans@sarin)
21:57 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/: Hot-deploy Ieaded578ffd T200968 due to bugs (duration: 00m 51s)
20:23 mutante: cp1080 - powercycled - lots of RECOVERY from Icinga for IPsec connections - leaving depooled so far (T201174 )
20:17 mutante: powercycling cp1080
20:06 mutante: dzahn@neodymium conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet| reason: Strongswan CRITICALs fom Icinga (T201174 )
20:04 dzahn@neodymium: conftool action : set/pooled=no; selector: name=cp1080.eqiad.wmnet
19:47 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.19
19:36 marxarelli: Deploying 1.32.0-wmf.19 to all wikis
19:21 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/includes/: I11b390
19:13 krinkle@deploy1001: Synchronized php-1.32.0-wmf.19/resources/src/startup: I13a996e01b48
18:52 moritzm: restarting aphlict on phab1001 to pick up nodejs security update
16:59 godog: xfs_repair on ms-be1041 sdf1 - T199198 (retroactive, started at 15:32
16:35 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet
16:30 gehel: shutting down wdqs1005 for new SSD and reimaging - T202779
16:29 gehel: shutting down wdqs1005 for new SSD and reimaging - T198351
16:15 gehel: restart of logstash to move data directory - T198351
15:17 bstorm_: increased number of nfsd threads on labstore1004 to 300
14:08 moritzm: upgrading mw1262-1265 to wikidiff 1.7.3
13:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119, db1090 (s2 and s7) (duration: 00m 59s)
13:41 moritzm: upgrading mw1261 to wikidiff 1.7.3
13:31 jynus: depooling labsdb1009 due to extra lag
13:10 jynus: sanitizing fixcopyrightwiki on db2094 and children T202820
13:08 jynus: sanitizing fixcopyrightwiki on db1124 and children T202820
13:03 moritzm: upgrading grafana to 4.6.4 (security release)
12:59 elukey: drain + reboot analytics10[28-79]* for kernel updates (will take multiple days)
12:28 godog: roll restart thumbor in eqiad to upgrade to 2.1
12:25 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 36s)
12:23 reedy@deploy1001: Synchronized dblists/: fixcopyrightwiki (duration: 00m 56s)
12:22 moritzm: uploaded grafana 4.6.4 to apt.wikimedia.org
12:13 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fixcopyrightwiki (duration: 00m 56s)
12:11 reedy@deploy1001: Synchronized static/images/project-logos: fixcopyrightwiki (duration: 00m 55s)
12:10 reedy@deploy1001: rebuilt and synchronized wikiversions files: fixcopyright
12:07 reedy@deploy1001: Synchronized dblists/: fixcopyrightwiki (duration: 00m 56s)
11:41 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2070 (duration: 00m 55s)
11:28 zeljkof: EU SWAT finished
11:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Test Wikidata: Use new item ID formatter for all the items (T203147) (duration: 00m 53s)
11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Beta Wikidata: Use new item ID formatter for all the items (T203147) (duration: 00m 57s)
11:09 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add editsitejson to everyone who has editinterface (T190015) (duration: 01m 00s)
10:37 moritzm: upgrading deployment-prep to wikidiff 1.7.3
10:04 godog: roll-restart thumbor to upgrade 2.1
08:52 godog: roll-restart swift-proxy to send requests to thumbor in eqiad and codfw - T201858
07:34 banyek@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2070 (duration: 00m 57s)
06:46 jynus: upgrade and restart db1090
06:43 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090 (s2 and s7) (duration: 00m 56s)
05:52 jynus: upgrade and restart db1119
05:49 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 (duration: 00m 58s)
03:42 mutante: running manual update of en.planet feeds (T203055 )
03:21 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 30 03:21:55 UTC 2018 (duration 10m 14s)
03:11 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 15m 12s)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 20s)
23:56 catrope@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Translate/stringmangler/StringMatcher.php: T202058 (duration: 00m 56s)
23:55 catrope@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/Translate/stringmangler/StringMatcher.php: T202058 (duration: 00m 59s)
19:47 bstorm_: failed tools and cloud vps storage services over to labstore1004
19:24 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 to 1.32.0-wmf.19
19:21 marxarelli: Deploying 1.32.0-wmf.19 to group1
18:55 mutante: puppet deploy of new cluster apache site inclusion for fixcopyright.wm, tested on mwdebug100*,mw1269, apache-fast-test (T202819 ) (gerrit:456194)
18:31 ottomata: debdeploy git-fat update for all nodes - T202100
18:10 mutante: puppet re-enabled on mw* via cumin before last log message (cumin expected to log to SAL automatically?)
18:08 mutante: new apache config in sites-available for new site fixcopyright.wm is being generated by puppet on cluster, but not enabled yet (T202819 )
17:51 imarlier@deploy1001: Finished deploy [performance/coal@7457c86]: (no justification provided) (duration: 00m 06s)
17:51 imarlier@deploy1001: Started deploy [performance/coal@7457c86]: (no justification provided)
17:51 mutante: disabling puppet on mw hosts as a precaution before deploying apache change
16:49 marostegui: Force RAID controller to WB policy T202051
16:21 rxy: tgr@deploy1001 Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata: Use new item ID formatter for Q1-Q10000 (T201834) (duration: 00m 56s) (by logmsgbot)
16:21 rxy: T202323 labstore1005 reboot (by arturo)
16:17 arturo: T202323 failover drbd primary from labstore1005 to labstore1004
15:53 arturo: T202323 labstore1004 now running latest kernel
15:51 marlier: deploy coal
15:51 imarlier@deploy1001: Finished deploy [performance/coal@5d995a3]: (no justification provided) (duration: 00m 06s)
15:51 imarlier@deploy1001: Started deploy [performance/coal@5d995a3]: (no justification provided)
15:48 elukey: roll restart aqs on aqs100[4-9] to pick up the new druid config settings
15:47 arturo: T202323 labstore1004 reboot
15:37 joal@deploy1001: Finished deploy [analytics/aqs/deploy@c33f6e5]: Update wikistats2 endpoints to use user_text and page_title (duration: 11m 38s)
15:26 joal@deploy1001: Started deploy [analytics/aqs/deploy@c33f6e5]: Update wikistats2 endpoints to use user_text and page_title
15:00 arturo: T202323 starting operations
14:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 (duration: 00m 58s)
13:45 gehel: restart relforge for plugin upgrade, data dir migration and kernel upgrade
13:41 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs2001.codfw.wmnet
13:39 gehel: shutting down wdqs2001 for new SSD and reimaging - T202777
13:11 kartik@deploy1001: Finished deploy [cxserver/deploy@c3385eb]: Update cxserver to afe0d1f T202970 ) (duration: 03m 46s)
13:07 kartik@deploy1001: Started deploy [cxserver/deploy@c3385eb]: Update cxserver to afe0d1f T202970 )
11:53 hashar: European SWAT completed (2)
11:51 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: yphc.ir to the wgCopyUploadsDomains whitelist - T201237 (duration: 00m 56s)
11:46 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use HD logos for various Wikibooks - T177506 (duration: 00m 56s)
11:42 hashar@deploy1001: Synchronized static/images/project-logos: Upload HD logos for various wikibooks - T177506 (duration: 00m 55s)
11:37 hashar: European SWAT completed.
11:37 hashar: deploy1001: "mwscript updateCollation.php --wiki=azwiki --previous-collation=uppercase" completed | T201770
11:35 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new logos for advisorywiki - T202844 (duration: 00m 57s)
11:33 hashar@deploy1001: Synchronized static/images/project-logos: Upload new logos for advisorswiki - T202844 (duration: 00m 55s)
11:30 hashar: last sync did not synchronized due to ssh / keyholder isses
11:30 hashar@deploy1001: Synchronized static/images/project-logos: Upload new logos for advisorswiki - T202844 (duration: 00m 37s)
11:30 moritzm: rearmed keyholder on sarin
11:25 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Allow subpages in main namespace in zhwikiversity - T202007 (duration: 00m 56s)
11:20 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Translation of scnwiktionary sitename was removed, add it back - T202926 (duration: 00m 56s)
11:13 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Allow sysops to remove flood flag - T202599 (duration: 00m 56s)
11:06 hashar: mwscript updateCollation.php --wiki=azwiki --previous-collation=uppercase # T201770
11:06 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set $wgCategoryCollation = uca-az on azwiki - T201770 (duration: 00m 56s)
10:41 elukey@deploy1001: Finished deploy [analytics/refinery@1c6423f]: (no justification provided) (duration: 02m 56s)
10:38 elukey@deploy1001: Started deploy [analytics/refinery@1c6423f]: (no justification provided)
10:30 joal@deploy1001: Finished deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs - try 2 (duration: 08m 28s)
10:22 joal@deploy1001: Started deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs - try 2
09:52 marostegui: Deploy schema change on db1091
09:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 58s)
09:48 marostegui: Deploy schema change on db1102:s4
09:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 57s)
09:33 elukey: cr1/2-eqiad: update analytics-in4 filter with the new archiva host, add a new term 'archiva' to analytics-in6 filter - T198623
09:09 volans: upgraded python{,3}-conftool,spicerack on sarin
09:05 joal@deploy1001: Finished deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs (duration: 11m 23s)
09:04 godog: switch statsd and carbon CNAMEs to graphite1004 - T196484
09:03 volans: uploaded spicerack_0.0.2-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T177385
08:56 volans: uploaded python{,3}-conftool_1.0.2-1{,+deb9u1} to apt.wikimedia.org {jessie,stretch}-wikimedia - T177385
08:54 joal@deploy1001: Started deploy [analytics/refinery@1c6423f]: Fix over yesterday weekly deploy of analytics Hadoop jobs
08:49 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=wtp1043.eqiad.wmnet
08:42 volans: uploaded cumin_3.0.2-2+deb9u1 to apt.wikimedia.org stretch-wikimedia - T177385
08:31 godog: repair sdh1 on ms-be2043 - T199198
08:31 godog: repair sdd1 on ms-be2040 - T199198
08:14 marostegui: Force WriteBack policy on db2042 T202051
08:10 addshore@deploy1001: Synchronized wmf-config/InterwikiSortOrders.php: DOCS ONLY: T170745 InterwikiSortOrders.php doc / comment (duration: 00m 57s)
08:07 moritzm: rolling reboot of wtp1* servers for kernel security update
07:47 marostegui: Reboot db2042 - T202051
07:31 moritzm: rebooting cp1008/pinkunicorn
06:03 elukey: migrate archiva.wikimedia.org to archiva1001 (upgrading archiva to its latest upstream version + Debian Stretch + Java 8)
06:02 marostegui: Deploy schema change on db1083 (labswiki) - T89737
05:58 marostegui: Deploy schema change on db1073 (labswiki) - T187089
05:55 marostegui: Deploy schema change on db1073 (labswiki) - T114117 T51191 T67448
05:46 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2088 - T202822 (duration: 00m 55s)
05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 54s)
05:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 01m 07s)
04:28 kart_: Finished old draft purge for ContentTranslation script run on mwmaint1001 (T201895 )
03:13 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 29 03:13:13 UTC 2018 (duration 10m 14s)
03:03 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.19) (duration: 16m 23s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 08m 01s)
00:05 tzatziki: reset email for User:Galahad
22:37 jforrester@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/JsonConfig/includes/JCSingleton.php: Hot-deploy T203006 fix (duration: 00m 56s)
22:36 jforrester@deploy1001: Synchronized php-1.32.0-wmf.19/extensions/JsonConfig/includes/JCSingleton.php: Hot-deploy T203006 fix (duration: 00m 57s)
21:51 chasemp: install nmap on stat1006 && chmod 500 /usr/bin/nmap
20:33 legoktm@deploy1001: Synchronized wmf-config/CommonSettings.php: beta only: Only enable TemplateWizard if TemplateData is also enabled (duration: 00m 56s)
19:47 herron: puppetmaster reboots finished, re-enabling puppet agents
19:26 herron: temporarily disabling puppet agents for rolling kernel updates/reboots on puppetmaster hosts
19:15 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.19
18:37 kaldari@deploy1001: Synchronized wmf-config/CommonSettings.php: for TemplateWizard (duration: 00m 55s)
18:36 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: for TemplateWizard (duration: 00m 55s)
18:35 kaldari@deploy1001: Synchronized wmf-config/InitialiseSettings.php: for TemplateWizard (duration: 00m 55s)
18:33 kaldari@deploy1001: Synchronized wmf-config/extension-list: (no justification provided) (duration: 00m 55s)
18:31 kaldari@deploy1001: Synchronized wmf-config/extension-list: (no justification provided) (duration: 00m 57s)
18:11 arlolra: Updated Parsoid to 61086f6 T199246 , T198221 )
18:11 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.19 and rebuild l10n cache (duration: 65m 35s)
18:00 arlolra@deploy1001: Finished deploy [parsoid/deploy@d29d829]: Updating Parsoid to 61086f6
17:52 moritzm: rebooting multatuli for microcode tests
17:51 arlolra@deploy1001: Started deploy [parsoid/deploy@d29d829]: Updating Parsoid to 61086f6
17:05 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.19 and rebuild l10n cache
16:25 marxarelli: starting branch cut for 1.32.0-wmf.19
15:38 marostegui: Deploy schema change on db1097:3314
15:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 00m 48s)
15:35 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet
15:35 gehel: shutting down wdqs1004 to add new disks - T202779
15:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 48s)
14:49 moritzm: depooling scb1002 for hardware diagnostics
13:15 godog: repair sdd1 on ms-be2041 - T199198
12:34 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/TemplateStyles/: SWAT: Hoist selectors for html and body element (T197617) (duration: 00m 48s)
12:31 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/vendor/: SWAT: 455770|Update wikimedia/css-sanitizer to 2.0.0 (T197617) (duration: 01m 16s)
11:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Permissions changes in ruwikinews (T201265) (duration: 00m 48s)
11:29 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create namespace aliases in zhwikiversity (T202821) (duration: 00m 48s)
11:24 zfilipin@deploy1001: Synchronized dblists/commonsuploads.dblist: SWAT: Remove uzwiki from commonsuploads.dblist (T202847) (duration: 00m 48s)
11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Wikidata: Use new item ID formatter for Q1-Q100 (T201833) (duration: 00m 49s)
11:00 arturo: T202549 delete mysql-server from cloudcontrol100[3,4.wikimedia.org
10:53 joal@deploy1001: Finished deploy [analytics/refinery@8cded22]: Regular weekly deploy of analytics Hadoop jobs (duration: 09m 14s)
10:52 marostegui: Import nova_api_eqiad1 and nova_eqiad1 into m5 master - T202549
10:51 arturo: T202549 downtime cloudcontrol1003.wikimedia.org in icinga for 2h
10:44 joal@deploy1001: Started deploy [analytics/refinery@8cded22]: Regular weekly deploy of analytics Hadoop jobs
10:42 marostegui: Deploy schema change on db1081
10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 48s)
10:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 00m 50s)
09:12 kartik@deploy1001: Finished deploy [cxserver/deploy@e2e5674]: Update cxserver to 98cbefd T186715 ) (duration: 03m 38s)
09:08 kartik@deploy1001: Started deploy [cxserver/deploy@e2e5674]: Update cxserver to 98cbefd T186715 )
07:57 moritzm: rebooting multatuli for some tests
07:47 marostegui: Deploy schema change on s3:advisorswiki - T202904 T199368
07:46 marostegui: Deploy schema change on s3:advisorswiki - T202904 T192926
07:45 marostegui: Deploy schema change on s3:advisorswiki - T202904 T196379
07:41 marostegui: Deploy schema change on s3:advisorswiki - T202904 https://phabricator.wikimedia.org/T195193
07:39 marostegui: Deploy schema change on s3:advisorswiki - T202904 T197891
07:37 moritzm: imported python-requests-mock/1.3.0-3~wmf1+stretch to apt.wikimedia.org/stretch-wikimedia
07:36 moritzm: imported linux-meta/1.19 to apt.wikimedia.org/jessie-wikimedia
06:57 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2033 - T201757 (duration: 00m 49s)
06:47 marostegui: Deploy schema change on s8 primary master (db1071)
05:56 volans@deploy1001: Finished deploy [netbox/deploy@5e70423]: Security upgrade of dependency (duration: 00m 34s)
05:55 volans@deploy1001: Started deploy [netbox/deploy@5e70423]: Security upgrade of dependency
05:28 krinkle@deploy1001: Synchronized php-1.32.0-wmf.18/resources/src/startup/startup.js: one line to rule them all - I4942bf
05:07 marostegui: Deploy schema change db1103:3314
05:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 49s)
04:56 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: (beta) 8d773b6cf
04:55 krinkle@deploy1001: sync-file aborted: (beta) 8d773b6cf
04:21 eileen: process-control config revision is d0de40dec9
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 28 02:45:08 UTC 2018 (duration 10m 12s)
02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 13m 17s)
01:39 eileen: civicrm revision changed from 4e0dcf7cb4 863ed3f1a2 5a629055fa
21:36 gehel: repooling wdqs2003, new SSD, reimaged and data loaded - T195285
21:04 krinkle@deploy1001: Synchronized wmf-config/mc.php: doc - Ic7349f4ce
20:39 Amir1: the ORES deployment is done
20:37 ladsgroup@deploy1001: Finished deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240 ) (duration: 25m 01s)
20:12 ladsgroup@deploy1001: Started deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240 )
20:10 ladsgroup@deploy1001: Finished deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240 ) (duration: 05m 38s)
20:09 Amir1: rolling back
20:04 ladsgroup@deploy1001: Started deploy [ores/deploy@0e8cc73]: Deploy renaming wp10 models to articlequality (T196240 )
19:34 XioNoX: adding graphite1004 and graphite2003 to analytics-in4 - T202846
19:19 XioNoX: adding IX bgp session to AS10089 on cr1-eqsin
18:58 XioNoX: pushing labs-instance-in4 changes to cr1/2-eqiad - T199437
18:56 catrope@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/PageTriage/includes/ArticleMetadata.php: Allow deferred writes on GET for pages with missing metadata (T202815 ) (duration: 00m 47s)
18:50 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add mhs.ox.ac.uk to $wgCopyUploadsDomains (T201604 ) (duration: 00m 48s)
18:47 catrope@deploy1001: Synchronized wmf-config/throttle.php: Throttle exception for 2018-08-29 (T202288 ) (duration: 00m 48s)
18:34 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable PageTriage log channel (T202815 ) (duration: 00m 57s)
18:21 XioNoX: merge Per DC alerting on sudden traffic drop (454613) - T201630
18:12 XioNoX: trunking cloud-instances1-b-codfw to labtestneutron2002:eth1 and labtestneutron2001:eth1 - T202636
17:37 XioNoX: pushing the above analytics-in changes to cr1/2-eqiad - T198623
17:27 volans: uploaded spicerack_0.0.1-1_amd64.deb to apt.wikimedia.org jessie-wikimedia - T199079
17:17 gehel@deploy1001: Finished deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (duration: 10m 24s)
17:07 gehel@deploy1001: Started deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater
17:05 gehel@deploy1001: Finished deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 27s)
17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@22869f0]: new version of wdqs GUI and updater (wdqs1009 only)
16:31 anomie: Running populateContentTables.php on advisorswiki for T183488 and T202904
15:50 marostegui: Upgrade kernel and mariadb on db2088
15:49 anomie: Re-running populateContentTables.php on aawikibooks, gotwikibooks, kswikiquote, lvwikibooks, nostalgiawiki, wawikibooks and wikimania2005wiki for T183488
15:48 gehel: reenabling puppet on install[12]002
15:26 gehel@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=codfw,cluster=wdqs,name=wdqs2003.eqiad.wmnet
15:26 gehel: taking wdqs2003 offline for SSD installation - T202778
15:12 herron: decreasing logstash elasticsearch index replica count to 1 on indices older than 1 day
15:12 godog: set transient low watermark to 80% for elasticsearch logstash cluster to allow shard replica allocation - T201971
14:52 papaul: up Shutting down db2088 for BIOS upgrade
14:43 anomie: Running deduplicateArchiveRevId.php on gotwikibooks, kswikiquote, lvwikibooks, nostalgiawiki, wawikibooks and wikimania2005wiki for T202032
14:42 anomie: Running deduplicateArchiveRevId.php on aawikibooks for T202032
14:04 mobrovac@deploy1001: Finished deploy [citoid/deploy@fe96789]: Resolve DOIs/URLs all the way to end - T197242 (duration: 05m 37s)
13:58 mobrovac@deploy1001: Started deploy [citoid/deploy@fe96789]: Resolve DOIs/URLs all the way to end - T197242
13:57 mobrovac@deploy1001: Finished deploy [proton/deploy@17fc7bb]: Ignore HTTPS errors for the time being (duration: 00m 33s)
13:57 mobrovac@deploy1001: Started deploy [proton/deploy@17fc7bb]: Ignore HTTPS errors for the time being
13:25 anomie@deploy1001: Synchronized php-1.32.0-wmf.18/includes/Storage/RevisionStore.php: Backport for T202032 (duration: 00m 49s)
12:12 akosiaris: rolling restart of parsoid service on wtp hosts for picking up https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/
12:02 addshore: swat done
12:00 marostegui: Create empty databases nova_api_eqiad1 and nova_eqiad1 on m5 master (db1073) - T202549
11:59 addshore@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Wikibase/lib/includes: gerrit:455542 Pass IDBAccessObject flag to RevisionStore in WikiPageEntityRevisionLookup T202706 (duration: 00m 51s)
11:50 godog: repair on ms-be2042 sdd - T199198
11:49 moritzm: rebooting labweb* for kernel security update
11:45 akosiaris: enable puppet on all hosts including puppet class service::configuration and start a rolling puppet run on them
11:40 tgr@deploy1001: Synchronized php-1.32.0-wmf.18/includes/diff/DifferenceEngine.php: SWAT: Fix DifferenceEngine revision loading logic (T201218, T202454) (duration: 00m 49s)
11:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: *.pensoft.net should be in wgCopyUploadsDomains whitelist instead of pensoft.net (T202832) (duration: 00m 48s)
11:28 zfilipin@deploy1001: Synchronized multiversion/MWMultiVersion.php: SWAT: Fix "seperated" typo in MWMultiVersion.php file (T201491) (duration: 00m 48s)
11:26 zfilipin@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Enable AbuseFilter "block" on it.wikibooks (T202808) (duration: 00m 48s)
11:19 marostegui: Deploy schema change on dbstore1002:s4
11:16 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Remove sitewide and user CSS/JS editing from old groups Enforce that interface-admin is the only group that can edit non-own CSS/JS (T190015 ) (duration: 00m 48s)
11:15 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove sitewide and user CSS/JS editing from old groups Enforce that interface-admin is the only group that can edit non-own CSS/JS (T190015 ) (duration: 00m 48s)
10:58 marostegui: Deploy schema change on db2051 (s4 codfw master) with replication, this will generate lag on s4 codfw
10:38 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
10:38 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
10:29 akosiaris: enable and run puppet on scb1001, restart services
10:29 akosiaris: enable and run puppet on restbase2001, restart restbase
10:19 akosiaris: disable puppet on hosts including service::configuration for merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/ . See https://phabricator.wikimedia.org/P7486
10:15 akosiaris: disable puppet on maps*, scb*, restbase* hosts for merge of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/454574/
10:09 moritzm: reimaging wtp2020 to stretch
09:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 48s)
09:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 48s)
09:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 49s)
09:20 addshore: deploy slot done
09:16 moritzm: rebooting bast4002 for kernel security update
09:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: gerrit:455389 Wikidata: Use new item ID formatter for Q1 T201832 (duration: 00m 49s)
08:59 marostegui: Drop sul, phadmin and phuser from dbstore1002
08:56 marostegui: Drop eventlogcleaner user from dbstore1002
08:53 moritzm: rebooting bast5001 for kernel security update
08:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 48s)
08:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 48s)
08:31 moritzm: installing ca-certificates updates for jessie/stretch
08:27 moritzm: reimaging wtp2017-wtp2019 to stretch
08:22 marostegui: deploy schema change on db1087 with replication (lag on labs:s8 will be generated)
08:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 48s)
08:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 (duration: 00m 52s)
08:05 gehel: restarting kartotherian / tilerator on maps* for nodejs upgrade
07:59 moritzm: installing nodejs security updates on maps* hosts
07:44 elukey: force remount of /mnt/hdfs on stat1005 (transport not connected errors)
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 (duration: 00m 50s)
07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 48s)
07:10 moritzm: reimaging wtp2014-wtp2016 to stretch
06:53 marostegui: Deploy schema change on labtestwiki - T89737
06:52 moritzm: installing discover updates from stretch 9.5 point release
06:50 marostegui: Deploy schema change on labtestwiki - T187089
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 00m 48s)
06:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 (duration: 00m 49s)
05:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 (duration: 00m 50s)
05:16 moritzm: reimaging wtp2011-wtp2013 to stretch
05:14 marostegui: Deploy schema change on db1066 (s2 primary master)
02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 27 02:46:56 UTC 2018 (duration 10m 12s)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 13s)
05:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2088 - T202822 (duration: 00m 54s)
22:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/WikimediaEvents/: I17452c
17:44 James_F: scap sync-dir php-1.32.0-wmf.18/extensions/WikimediaEvents/ No-op bump this code to avoid dirty deploy repo
17:44 James_F: scap sync-file php-1.32.0-wmf.18/extensions/Echo/includes/gateway/UserNotificationGateway.php Hot-deploy of Ifdfa93059 T202672 log error
17:44 James_F: scap sync-file php-1.32.0-wmf.18/extensions/WikiEditor/modules/jquery.wikiEditor.dialogs.config.js Hot-deploy of I364ac118255
15:07 marostegui: Deploy schema change on dbstore1002:s8
13:45 marostegui: Deploy schema change on s8 codfw master with replication (this will generate lag on s8 codfw)
12:33 akosiaris: upload apertium-kaz-tat to apt.wikimedia.org/jessie-wikimedia/main T199962
12:33 akosiaris: upload apertium-eo-fr to apt.wikimedia.org/jessie-wikimedia/main T199962
11:58 elukey: upgrade nodejs packages on aqs* for security upgrade (rolling restart of aqs daemon included)
11:19 ema: powercycle cp3030, stuck rebooting T200445
10:47 moritzm: reimage wtp2008-wtp2010 to stretch
09:28 moritzm: reimage wtp2005-wtp2007 to stretch
09:09 marostegui: Restart MySQL on db2035 (codfw s2 master) for mysql upgrade and pick up new mysql options after merging https://gerrit.wikimedia.org/r/455103
09:04 marostegui: Deploy schema change on db1095:3312
07:56 marostegui: Deploy schema change on db1122
07:53 moritzm: reimage wtp2002-wtp2004 to stretch
06:52 marostegui: Deploy schema change on db1074 with replication (this will generate lag on s2 labs)
06:38 moritzm: reimage wtp2001 to stretch
06:09 moritzm: imported php-igbinary, libzip5, php-apcu-bc to thirdparty/php72 (T202704 )
05:26 marostegui: Deploy schema change on db1076
00:21 mutante: ms-be2043 - mount -a ; re-enabled puppet, running puppet
23:47 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202548 (duration: 01m 01s)
23:43 Reedy: created wikilove tables on zh_yuewiki T202548
23:42 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201236 (duration: 00m 55s)
23:37 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202599 (duration: 00m 53s)
23:34 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202599 (duration: 00m 49s)
23:28 Reedy: running `mwscript updateCollation.php --wiki=trwiki --previous-collation=uppercase` T184943
23:28 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T184943 (duration: 00m 48s)
23:23 reedy@deploy1001: Synchronized wmf-config/flaggedrevs.php: T202139 (duration: 00m 48s)
23:19 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202680 (duration: 00m 48s)
23:16 reedy@deploy1001: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
23:16 reedy@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
23:15 reedy@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 00s)
22:53 XioNoX: Add v4 session in eqdfw to AS8075
22:46 XioNoX: Add v6 session in eqsin to 10089
22:39 XioNoX: deactivating down AMSIX peer (no reply to emails)
22:36 XioNoX: clear peer 80.249.208.51 AS 8708 on cr2-esams
22:31 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.18
22:27 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/FlaggedRevs/backend/FlaggedRevs.class.php: Follow-up 413e11e2f: Skip CurrentRevisionCallback if not stabilized T202659 (duration: 00m 56s)
19:34 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: revert all wikis to 1.32.0-wmf.18
19:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 5, onthisday is timing out. Just push this through.. (duration: 03m 27s)
19:31 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 5, onthisday is timing out. Just push this through..
19:26 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.18
18:54 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 4, onthisday is timing out. Just push this through.. (duration: 04m 39s)
18:50 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 4, onthisday is timing out. Just push this through..
18:48 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is timing out. Just push this through.. (duration: 03m 59s)
18:44 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is timing out. Just push this through..
18:41 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is failing (duration: 03m 53s)
18:37 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.18 (duration: 00m 54s)
18:37 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 3, onthisday is failing
18:37 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 2, onthisday is failing (duration: 06m 30s)
18:37 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.18
18:33 XioNoX: Equinix updated their MAC filter, IX sessions up - T196941
18:33 thcipriani: starting train a little early to play catchup
18:32 RoanKattouw: Deployed patch for T199993
18:30 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage, take 2, onthisday is failing
18:30 ppchelko@deploy1001: Finished deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage (duration: 03m 54s)
18:30 XioNoX: remove MAC filter workaround on cr2-eqdfw - T196941
18:28 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/FlaggedRevs/backend/FlaggedRevs.class.php: Fix incorrect return value in CurrentRevisionCallback T202580 (duration: 00m 54s)
18:28 RoanKattouw: Deployed patch for T151910
18:26 ppchelko@deploy1001: Started deploy [restbase/deploy@b2b61e8]: Resurrect MCS on wikivoyage
18:15 otto@deploy1001: Finished deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 with requirements setup (duration: 00m 03s)
18:15 otto@deploy1001: Started deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 with requirements setup
18:09 otto@deploy1001: Finished deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003 (duration: 00m 36s)
18:08 otto@deploy1001: Started deploy [analytics/superset/deploy@de75f23]: 0.26.3 to analytics-tool1003
17:28 XioNoX: enable v4 neighbors on cr2-eqdfw - T196941
17:16 XioNoX: enable v6 neighbors on cr2-eqdfw - T196941
17:01 XioNoX: powering off cr1-eqdfw - T196941
16:49 filippo@neodymium: conftool action : set/pooled=yes; selector: name=restbase2003.codfw.wmnet
16:22 godog: depool and cassandra-drain restbase2003, then reboot
16:20 filippo@neodymium: conftool action : set/pooled=no; selector: name=restbase2003.codfw.wmnet
16:17 XioNoX: deactivating IX/Transit links on cr1-eqdfw - T196941
16:15 godog: upgrade hp raid firmware on restbase2003 - T201804 T141756
15:15 marostegui: Deploy schema change on db1090:3312
15:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 (duration: 00m 54s)
15:14 jynus: upgrading db2089
15:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 (duration: 00m 55s)
15:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113 (duration: 00m 54s)
14:47 jynus: upgrading db1113
14:37 otto@deploy1001: Finished deploy [analytics/turnilo/deploy@50a8845]: Deploying 1.7.2 to analytics-tool1002 with dep fixes (duration: 00m 08s)
14:37 otto@deploy1001: Started deploy [analytics/turnilo/deploy@50a8845]: Deploying 1.7.2 to analytics-tool1002 with dep fixes
14:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113 (duration: 00m 57s)
14:26 otto@deploy1001: Finished deploy [analytics/turnilo/deploy@240b357]: Deploying 1.7.2 to analytics-tool1002 (duration: 05m 07s)
14:21 otto@deploy1001: Started deploy [analytics/turnilo/deploy@240b357]: Deploying 1.7.2 to analytics-tool1002
13:39 jynus: upgrading dbstore1001
13:38 marostegui: Deploy schema change on db1103:3312
13:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 (duration: 00m 54s)
13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 (duration: 00m 54s)
13:29 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T198396 Revert "Limit page creation and edit rate on Wikidata" gerrit:454785 (duration: 00m 56s)
13:06 jynus: upgrading dbstore200[12], will take a while due to ongoing alter tables
12:34 moritzm: install Java updates on Hadoop nodes
12:06 moritzm: created new archiva-deployers LDAP group (T200454 )
12:00 marostegui: Deploy schema change on db1105:3312
12:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 (duration: 00m 55s)
11:56 addshore: SWAT done
11:50 addshore@deploy1001: Synchronized wmf-config: T199800 gerrit:Cleanup wikdiff2 mobile moved paragraph config (duration: 00m 55s)
11:40 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202555 Require autoconfirmed status to edit the 828 namespace at es.wikibooks (duration: 00m 56s)
11:35 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T202065 Switch 'wikidata-staff' add/remove 'interface-admin' for own account (duration: 00m 55s)
11:19 moritzm: installing shared-mime-info updates from stretch 9.5 point release
11:18 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: T199800 Enable moved paragrah detection everywhere (duration: 00m 55s)
11:01 marostegui: Deploy schema change on dbstore1002:s2
11:00 gehel: new SSL certs / tlsproxy deployed on elastic nodes - T198351
10:56 marostegui: Deploy schema change on s2 codfw masters (db2035) with replication - this will generate lag on s2 codfw
10:35 moritzm: installing openldap updates from stretch 9.5 point release
10:27 akosiaris: reboot analytics-tool* hosts for the IP renumbering change T202559
10:27 akosiaris: purge rec dns caches for analytics-tool* hosts T202559
10:15 elukey: restart druid-broker on druid100[4-5] due to unresponsiveness (still unclear why)
10:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 (duration: 00m 54s)
09:59 marostegui: Deploy schema change on s6 primary master (db1061)
09:50 moritzm: installing debdeploy update
09:40 moritzm: restarting etherpad-lite for nodejs security update
09:29 moritzm: installing nodejs security updates on wtp* (Parsoid servers)
09:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 (duration: 00m 55s)
09:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 (duration: 00m 56s)
09:17 gehel: starting to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/444610 as part of T198351 , including regeneration of SSL certs. Disabling puppet on elastic* during the operation
08:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 00m 55s)
08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 (duration: 00m 55s)
08:42 addshore: done with deploy window
08:41 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T201832 Stuff for new link formatter on testwikidata gerrit:454510 (duration: 00m 55s)
08:39 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T201832 Stuff for new link formatter on testwikidata gerrit:454510 gerrit:452365 gerrit:454764 (duration: 00m 56s)
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 00m 55s)
07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 (duration: 00m 56s)
07:30 moritzm: installing openssl updates
07:23 marostegui: Drop blob_orphans and blob_tracking from s3 on codfw (lag will be generated on codfw) T59186
07:12 marostegui: Drop blob_orphans and blob_tracking from s1 T59186
06:47 marostegui: Drop blob_orphans and blob_tracking from s7 T59186
06:31 marostegui: Enable semi-sync on es2 - T202364
06:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 (duration: 00m 54s)
06:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 (duration: 00m 55s)
05:57 marostegui: Drop blob_orphans and blob_tracking from s2 T59186
05:50 marostegui: Drop blob_orphans and blob_tracking from s4 T59186
05:47 marostegui: Drop blob_orphans and blob_tracking from s6 T59186
05:19 marostegui: Drop blob_orphans and blob_tracking from s5 T59186
04:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 (duration: 00m 55s)
04:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 (duration: 00m 57s)
03:14 ejegg: updated payments-wiki from 268539c2bd 763e14b6df
03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 23 03:06:25 UTC 2018 (duration 10m 22s)
02:56 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 13m 27s)
02:34 ejegg: updated CiviCRM from affd1b5e9a 4e0dcf7cb4
02:29 ejegg: updated SmashPig standalone deploy from 3603e191a3 cd46beefb7
02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 09m 10s)
00:36 twentyafterfour: phabricator will be down momentarily for apache restart, downtime scheduled in icinga
00:08 twentyafterfour: preparing phabricator update
23:17 thcipriani@deploy1001: Synchronized wmf-config/WikibaseSearchSettings.php: SWAT: Use proper data types for indexing items T199884 (duration: 00m 55s)
23:15 SMalyshev: repooled wdq3 after recovery
22:45 SMalyshev: temp depooled wdq3 to make it catch up faster
21:57 mutante: releases1001/2001: sudo find /srv/org/wikimedia/releases/wikidiff2 -name wikidiff2\* -exec chown :releasers-wikidiff2 {} \; (T202473 )
20:37 ladsgroup@deploy1001: Finished deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097 (duration: 31m 42s)
20:32 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@f7fa1df]: Update mobileapps to 141ff20 T202105 T202237 ) (duration: 03m 43s)
20:28 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@f7fa1df]: Update mobileapps to 141ff20 T202105 T202237 )
20:26 thcipriani@deploy1001: Synchronized php: Revert "Group1 wikis to 1.32.0-wmf.18" (duration: 00m 54s)
20:24 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Revert "Group1 wikis to 1.32.0-wmf.18"
20:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@8dd6d74]: bump to master, msearch fixes (duration: 03m 41s)
20:07 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@8dd6d74]: bump to master, msearch fixes
20:05 ladsgroup@deploy1001: Started deploy [ores/deploy@8ff4da1]: Update ORES to use git lfs for model T197097
19:55 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.18 (duration: 00m 54s)
19:54 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.18
19:38 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/Flow/includes/TemplateHelper.php: Work around exception in DifferenceEngine::showDiffStyle() T202454 (duration: 00m 57s)
19:01 anomie: re-running populateContentTables.php on cawiki for T183488
18:14 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@00671e6]: Add prometheus-client based metrics export (duration: 04m 22s)
18:10 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@00671e6]: Add prometheus-client based metrics export
18:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@ebb466b]: Add prometheus-client based metrics export (duration: 01m 36s)
18:08 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@ebb466b]: Add prometheus-client based metrics export
17:50 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@6fdae90]: updates to support msearch daemon (duration: 04m 16s)
17:45 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@6fdae90]: updates to support msearch daemon
17:31 jforrester@deploy1001: Synchronized php-1.32.0-wmf.18/extensions/JsonConfig/includes/JCSingleton.php: T202469 fix UBN trainblocker for wmf.18 (duration: 00m 56s)
17:21 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@32a81be]: Revisit jobs concurrencies T202107 (duration: 00m 49s)
17:20 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@32a81be]: Revisit jobs concurrencies T202107
17:19 Jeff_Green: authdns-update to deploy commit If54277153b6
16:58 godog: start backfilling metrics from graphite1001 into graphite1004 - T196484
16:51 godog: repair on ms-be2042 sdk/sdn - T199198
16:51 godog: correction, ms-be2040 - T199198
16:48 godog: repair on ms-be2020 sdh/sdc - T199198
16:21 ema: prometheus-trafficserver-exporter 0.0.2-1 uploaded to stretch-wikimedia T202381
16:06 marostegui: Deploy schema change on db1096:3316
16:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 (duration: 00m 55s)
16:02 XioNoX: add static NAT for 208.80.155.15 - T202520
15:54 XioNoX: Update NAT for 208.80.152.231 - T202536
14:57 elukey: upload archiva 2.2.3-2 to stretch-wikimedia
14:57 XioNoX: load eqiad-1534946573 on pfw3-eqiad - T202537
14:56 elukey: update pcc's facts
14:55 XioNoX: load codfw-1534946573 on pfw3-codfw - T202537
14:24 marostegui: Deploy schema change on dbstore1002
14:11 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1011 (duration: 00m 55s)
14:03 moritzm: installing openssl updates
13:15 addshore@deploy1001: Synchronized wmf-config: BETA ONLY (1 wikidata patch) (duration: 00m 56s)
13:14 addshore@deploy1001: sync-file aborted: BETA ONLY (1 wikidata patche) (duration: 00m 00s)
13:07 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set default for wgMultiContentRevisionSchemaMigrationStage T197816 (duration: 00m 56s)
12:59 marostegui: Deploy schema change on s6 codfw masters (db2039) this will generate lag on s6 codfw
12:46 addshore@deploy1001: Synchronized wmf-config: BETA ONLY (2 wikidata patches) (duration: 00m 55s)
12:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 (duration: 00m 57s)
12:15 marostegui: Deploy schema change on db1070 (s5 primary master) - T67448 T114117 T51191
11:40 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2069 - T201603 (duration: 00m 56s)
11:37 marostegui: Deploy schema change on db1113:3315
11:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 (duration: 00m 58s)
10:43 elukey: upload archiva 2.2.3-1 to stretch-wikimedia/main - T192639
10:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #5 (duration: 06m 15s)
10:30 Amir1: deploying ores gerrit:454283 to beta (T197097 )
10:25 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #5
10:25 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #4 (duration: 08m 31s)
10:17 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #4
10:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #3 (duration: 08m 28s)
10:08 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points, take #3
10:08 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202105 (duration: 13m 33s)
09:59 jynus: restarting backups on codfw due to grant issue
09:54 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202105
09:53 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202425 (duration: 03m 25s)
09:51 moritzm: uploaded clustershell 1.8-1~deb9u1 to apt.wikimedia.org/stretch-wikimedia
09:49 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c]: Expand CSS end points - T202425
09:49 mobrovac@deploy1001: Finished deploy [restbase/deploy@5d03f1c] (dev-cluster): Expand CSS end points (duration: 03m 35s)
09:45 mobrovac@deploy1001: Started deploy [restbase/deploy@5d03f1c] (dev-cluster): Expand CSS end points
09:34 moritzm: installing nodejs security updates on restbase*
09:19 jynus: reimage es1011 to stretch
09:06 addshore@deploy1001: Synchronized php-1.32.0-wmf.18/includes: T202483 The BlobStoreFactory constructor needs an LBFactory (duration: 01m 18s)
08:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1011 (duration: 00m 56s)
08:46 marostegui: Stop replication on x1 codfw master (db2034) for data fixes - T104459 T201603
08:40 moritzm: installing clamav security updates on mendelevium (bundling libmspack security update along)
08:39 ema: repool eqiad for front-edge traffic
07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Change x1 eqiad weights to give more weight to the most powerful server (duration: 00m 55s)
07:06 moritzm: installing apache update on deploy1001/planet1001
06:57 moritzm: imported libargon-2.1 for thirdparty/php7.2 repo (T202478 )
06:30 jynus: finished es2 switchover T202364
06:20 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Switchover es2 master eqiad from es1011 to es1015 (duration: 00m 55s)
06:12 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es2 (duration: 01m 05s)
06:11 jynus: depool es2 to failover master from es1011 to es1015
05:53 marostegui: Start topology changes for es2 failover - T202364
05:53 jynus: restarting heartbeat on es1011
03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 22 03:06:38 UTC 2018 (duration 10m 29s)
02:56 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.18) (duration: 15m 36s)
02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 09m 06s)
01:10 mutante: puppetdb2001 - dumping postgres db to local backup path
23:42 thcipriani@deploy1001: Synchronized wmf-config: SWAT: Switch entity reference type indexing from opt-in to opt-out T199884 (duration: 00m 57s)
22:49 eileen_: civicrm revision changed from 8910c34502 affd1b5e9a 4c055834dd
22:32 mutante: vega - cd /srv/org/wikimedia/TransparencyReport ; git reset --hard origin ; puppet agent -tv (to fix transparency.wikimedia.org content repo and puppet run, just like on bromine)
22:29 mutante: bromine - last log line refers to /srv/org/wikimedia/TransparencyReport (transparency.wikimedia.org)
22:28 mutante: bromine - git reset --hard origin ; git checkout master ; git pull origin ; puppet agent -tv (to fix broken puppet run due to unclean git repo.. unknown why it broke)
21:00 moritzm: rebooting mw2232 for some tests
20:17 herron: re-enabling puppet agents after puppetmaster apache update
20:11 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.18
20:07 herron: temporarily disabling puppet agents during puppetmaster apache updates
19:49 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache (duration: 34m 54s)
19:14 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.18 and rebuild l10n cache
19:11 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.13 (duration: 01m 41s)
19:08 mutante: deploy1001 - chmod -R g+w /srv/mediawiki-staging/php-1.32.0-wmf.13/.git
18:58 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.12 (duration: 03m 15s)
18:52 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 (duration: 07m 37s)
18:17 volans: restarting ircecho on einsteinium - T202314
17:58 volans: restarting ircecho on einsteinium - T202314
17:13 thcipriani: starting branch cut for 1.32.0-wmf.18
16:32 XioNoX: enabling ae1 on asw-a-eqiad then ae1 on cr1-eqiad - T202075
16:27 godog: upgrade hp raid firmware on ms-be2020 - T141756
16:21 XioNoX: disable ae1 on asw-a-eqiad - T202075
16:18 XioNoX: disable ae1 on cr1-eqiad - T202075
16:17 papaul: replacing PEM2 on cr2-codfw
15:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096 (duration: 00m 50s)
15:24 ejegg: restarted CiviCRM deduplication jobs
15:10 moritzm: installing apache updates from stretch 9.5 point release
14:58 moritzm: rolling restart of proton* to pick up nodejs security update
13:44 jynus: stop db1096 (both s5 an s6) for upgrade
13:36 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096 (duration: 00m 49s)
13:27 jynus: stop db2090 for upgrade
12:38 ema: power-cycle ms-be2020: multiple alerts, no ssh access, I/O errors in console
12:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
12:31 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 50s)
12:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 49s)
12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 (duration: 00m 49s)
12:13 moritzm: rolling restart of services in scb/eqiad to pick up the nodejs security update
12:05 jynus: stop es1015 for upgrade
11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 50s)
11:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 (duration: 00m 50s)
11:16 zeljkof: EU SWAT finished
11:16 zfilipin@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Update several Wikidata-related configs (duration: 00m 50s)
11:05 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: Add IP ranges for ptWiki editathon (T202354) (duration: 00m 52s)
09:58 moritzm: installing libxcursor security updates
09:46 moritzm: installing libxml2 security updates on trusty
09:27 moritzm: installing mutt security updates
09:16 moritzm: installing jetty9 security updates
08:37 marostegui: Rename blob_tracking and blob_orphans on db1089 - T59186
07:55 marostegui: Slowly reload haproxies on dbproxy10XX - T201021
06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 00m 50s)
05:12 marostegui: Stop puppet on dbproxy1006 to do some haproxy logging tests - https://phabricator.wikimedia.org/T201021
05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 00m 51s)
04:54 marostegui: Set max_connections back from 800 to 500 on db1073 - T188589
02:47 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: rm unused Id5f5295d
02:43 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: rm unused I741b16452
02:39 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 21 02:39:12 UTC 2018 (duration 10m 18s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 32s)
01:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter operations for all wikis (duration: 00m 51s)
00:07 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters (duration: 00m 46s)
00:07 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@9fbf9fa]: Don't clean up the queue size counters
23:09 twentyafterfour: Finished evening SWAT
23:08 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ Phab Task: T202065 (duration: 00m 51s)
23:06 twentyafterfour: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/454083/ deployed to mwdebug1001
22:53 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges (duration: 00m 45s)
22:52 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c947567]: Use timing for queue sizes as a workaround to grafana not plotting gauges
22:29 eileen: civicrm revision changed from 3f6dd6c9a6 8910c34502 d787269882
22:27 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107 (duration: 00m 46s)
22:26 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@541932a]: Make the queue sized report stats once a second T202107
22:13 mutante: ganeti1001 - creating 3 new VMs for analytics-tools
21:08 eileen: civicrm revision is 3f6dd6c9a6 d787269882
21:07 fdans@deploy1001: Finished deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering (duration: 10m 17s)
20:56 fdans@deploy1001: Started deploy [analytics/refinery@f59ce0c]: deploying changes to virtualpageview spam site filtering
20:43 aaron@deploy1001: Synchronized wmf-config/mc.php: Set "cluster" parameter for mcrouter broadcast routing prefixes (duration: 00m 50s)
20:36 arlolra: Updated Parsoid to 129d71f T130224 , T199926 )
20:35 mbsantos@deploy1001: Finished deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d T202105 ) (duration: 06m 19s)
20:29 mbsantos@deploy1001: Started deploy [mobileapps/deploy@cae24fe]: Update mobileapps to 95e976d T202105 )
20:26 arlolra@deploy1001: Finished deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f
20:14 arlolra@deploy1001: Started deploy [parsoid/deploy@44aa5e8]: Updating Parsoid to 129d71f
19:35 jforrester@deploy1001: Finished scap: i18n sync for WikimediaMessages clean-up following T202095 (duration: 31m 12s)
19:04 jforrester@deploy1001: Started scap: i18n sync for WikimediaMessages clean-up following T202095
19:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/WikimediaMessages/: SWAT T202095 Reinstate "Rename global OTRS-member group to otrs-member" (will need scap sync later) (duration: 00m 49s)
18:56 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/TimedMediaHandler/: SWAT T202208Fix regression: double-reg of file types and incorrect mp4 (duration: 00m 52s)
18:44 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT T201783 T202014 Add Contact link to footers of fr,ruwiki (duration: 00m 50s)
18:36 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (duration: 10m 25s)
18:32 jforrester@deploy1001: Synchronized wmf-config/throttle.php: SWAT T202003 Add throttle exemption for 24 August (duration: 00m 51s)
18:31 volans: restarted ircecho on einsteinium
18:26 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater
18:25 gehel@deploy1001: Finished deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
18:24 gehel@deploy1001: Started deploy [wdqs/wdqs@df2da41]: new version of wdqs GUI and updater (wdqs1009 only)
18:05 volans: restarting icinga, re-occurrence of T196336
17:44 chasemp: install iotop on stat1004
17:08 XenoRyet: updated payments-wiki from 360bea8331 268539c2bd
16:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107 (duration: 00m 55s)
16:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@4989231]: Create metrics to track the actual concurrent job executions T202107
14:14 moritzm: rolling upgrade of scb in codfw to latest nodejs security update
13:46 moritzm: installing jetty9 security updates
13:00 marostegui: Increase max_connections from 500 to 800 on db1073 to triage issues - T188589
12:51 jynus: reload dbproxy1005
12:29 moritzm: upgrading scb2006 to latest nodejs along with rolling restarts of node-based services
12:02 zeljkof: EU SWAT finished
12:01 arturo: T202261 extend icinga downtime 1D for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet neutron not properly syncing with agents
11:56 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Adding Chinese Wikiversitys logos (T202127) (duration: 00m 50s)
11:50 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for lfnwiki (T202228) (duration: 00m 50s)
11:41 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for lfnwiki (T202228) (duration: 00m 50s)
11:36 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Rollbacker User Group at ru.wikiquote (T200201) (duration: 00m 53s)
11:25 arturo: T202261 icinga downtime 1h for cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet previous to patch merge
11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow all bureaucrats to remove interface-admin (duration: 00m 54s)
11:01 arturo: T202261 disabled puppet in cloudcontrol1003.wikimedia.org, cloudcontrol1004.wikimedia.org, clounet1003.eqiad.wmnet, cloudnet1004.eqiad.wmnet
10:31 moritzm: rebooting mw1261 for kernel update/some tests
10:23 moritzm: rebooting mw2234 for some kernel tests
10:07 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2 to apt.wikimedia.org
09:31 moritzm: uploaded nodejs 6.11.0~dfsg-1+wmf2+jessie to apt.wikimedia.org
08:50 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request (duration: 06m 29s)
08:43 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request
08:43 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974 (duration: 18m 17s)
08:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 (duration: 00m 50s)
08:31 moritzm: installing systemd updates from stretch 9.5 point release
08:25 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3]: Remove contentmodel from MW API revision request - T201974
08:24 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request (duration: 06m 02s)
08:18 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request
08:16 mobrovac@deploy1001: Finished deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974 (duration: 04m 16s)
08:15 marostegui: Deploy schema change on db1100 to check for regressions
08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 50s)
08:12 mobrovac@deploy1001: Started deploy [restbase/deploy@a3ae0d3] (dev-cluster): Remove contentmodel from MW API revision request - T201974
08:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db1100 (duration: 01m 02s)
07:29 gehel: resetting management card on elastic1022
06:57 volans: reset-failed debmonitor failed session on ms-be2024 ^^^^
02:47 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 20 02:47:17 UTC 2018 (duration 10m 22s)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 23s)
16:24 gehel: disabling systemd state check for elastic eqiad until T202120 is fixed
14:38 moritzm: rebooting elnath for some kernel tests
13:02 jynus: stopping db2084 for upgrade
12:49 imarlier@deploy1001: Finished deploy [performance/coal@aff3793]: (no justification provided) (duration: 01m 06s)
12:48 imarlier@deploy1001: Started deploy [performance/coal@aff3793]: (no justification provided)
12:46 gehel: rebooting relforge for JVM and kernel upgrade
12:36 moritzm: installing openjdk-8 security updates on elastic* in codfw (eqiad already has it via the recent reimage to stretch)
12:30 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097 (duration: 00m 53s)
12:07 gehel: repooling wdqs2003, catched up on updates
10:58 moritzm: rebooting mw2152-mw2162 for kernel security update (also bundling wikidiff and apache updates)
10:35 moritzm: powercycling mw2286
09:47 moritzm: rebooting mw2243-mw2290 for kernel security update (also bundling wikidiff and apache updates)
09:03 gehel: depool wdqs2003 to catch up on updates
08:29 jynus: stopping db2073 for upgrade
08:13 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097 (duration: 00m 54s)
08:06 gehel: silencing systemd check as mjolnir is flapping
07:59 gehel: restarting mjolnir on all elastic / cirrus nodes
07:42 jynus: stopping db2085 for upgrade
07:41 moritzm: rebooting mw2220-mw2242 for kernel security update (also bundling wikidiff and apache updates)
00:42 krinkle@deploy1001: Synchronized wmf-config/: rm StartProfiler.php / Ia83751 T201782 (duration: 00m 50s)
00:39 krinkle@deploy1001: Synchronized docroot/noc/: Ia83751
23:41 ejegg: re-activated CiviCRM scheduled jobs
23:34 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/skins/MinervaNeue/resources/skins.minerva.content.styles.images/magnifying-glass.svg: Correct MinervaNeue search icon (T199000 ) (duration: 00m 51s)
23:18 ejegg: disabled CiviCRM scheduled jobs for version upgrade
23:15 eileen: civicrm revision changed from b60ceb3d2f 3f6dd6c9a6 74ae910b0e
22:32 XioNoX: re-activating BGP sessions between cr1/2-ulsfo and the office's router2
22:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail (duration: 00m 14s)
22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: retry git fat init fail
22:24 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib (duration: 00m 11s)
22:24 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@e70f9d5]: dont override the spark sharelib
21:56 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1 (duration: 00m 16s)
21:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@651904b]: use spark 2.3.0, oozie still doesnt like 2.3.1
21:56 ebernhardson@deploy1001: deploy aborted: try again after git-fat init fail (duration: 00m 01s)
21:55 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
21:51 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail (duration: 00m 18s)
21:50 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: try again after git-fat init fail
21:46 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 04m 56s)
21:44 mutante: rebooting cobalt (gerrit.wikimedia.org) for kernel upgrade
21:41 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
21:39 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0 (duration: 00m 14s)
21:39 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.0
21:29 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1 (duration: 00m 18s)
21:28 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5731563]: point oozie sharelib at spark2.3.1
21:23 thcipriani: contint1001 - installing jenkins upgrade
21:08 mutante: contint2001 - installing jenkins upgrade
21:05 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 16s)
21:04 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
21:01 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail (duration: 00m 06s)
21:01 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: debug git-fat initialization fail
21:00 mutante: releases1001 - installing package upgrade like on releases2001 before, scheduling downtime, reboot
20:53 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided) (duration: 00m 02s)
20:53 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: (no justification provided)
20:50 mutante: releases2001 - rebooting
20:48 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 00m 46s)
20:48 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
20:47 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x (duration: 01m 33s)
20:46 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@76dddd2]: point transfer_to_es at spark 2.x
20:38 mutante: releases2001: upgrading openjdk, systemd, jenkins
20:33 mutante: releases1001/2001: upgrading apache2 packages
20:26 mutante: gerrit2001 - scheduled downtime, rebooting
19:54 mutante: phab1002 closing idle root screen that was used for rsyncing repos
19:15 thcipriani: clearing gerrit accounts cache
19:11 twentyafterfour: twentyafterfour and thcipriani performing online maintenance on gerrit All-Users repo
19:10 legoktm: T201314 mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=enwiki --logwiki=metawiki 'EricEnfermero' 'Larry Hockett' --ignorestatus
19:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided) (duration: 00m 17s)
19:05 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@9fb53c4]: (no justification provided)
18:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle (duration: 00m 18s)
18:56 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@040690e]: coordinator.properties should reference a coordinator, not bundle
18:52 arlolra: Updated Parsoid to dbbad6a T201115 )
18:47 arlolra@deploy1001: Finished deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a
18:45 gehel: reimage of elasticsearch eqiad completed - T198391 / T193649
18:37 arlolra@deploy1001: Started deploy [parsoid/deploy@59f6585]: Updating Parsoid to dbbad6a
18:25 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang (duration: 00m 17s)
18:25 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@5d87cc0]: now without the shebang
18:22 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES filters in PageTriage on testwiki (duration: 00m 50s)
18:16 catrope@deploy1001: Synchronized wmf-config/throttle.php: Throttle exemptions for cswiki (T202038 ) (duration: 00m 53s)
18:14 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling (duration: 03m 49s)
18:10 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@0a704b6]: push new python dependency handling
18:06 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling (duration: 00m 05s)
18:06 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@d994cb9]: push new python dependency handling
17:59 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling (duration: 00m 20s)
17:59 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@122080c]: push new python dependency handling
17:57 ebernhardson@deploy1001: Finished deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job (duration: 00m 08s)
17:57 ebernhardson@deploy1001: Started deploy [wikimedia/discovery/analytics@fec00bc]: Push updated transfer-to-es oozie job
17:36 gehel: reimaging elastic1029
16:12 gehel: banning, depooling and shutting down elastic1029 for memory replacement - T201991
15:55 jynus: stopping db2034 for upgrade
15:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2019 (duration: 00m 51s)
15:34 jynus: stopping es2019 for upgrade
15:29 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool es2018, depool es2019 (duration: 00m 50s)
15:16 bblack: restarting pybal on lvs1016 - T201694
15:13 cmjohnson1: lvs1016 moving uplink from asw2-a-eqiad to asw-a-eqiad
15:09 bblack: stopping pybal on lvs1016 to fail traffic to lvs1006 for T201694
15:07 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[78]\.eqiad\.wmnet
15:05 cmjohnson1: cp107[78] moving uplink from asw2-a-eqiad to asw-a-eqiad
15:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[78]\.eqiad\.wmnet
15:04 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
15:03 cmjohnson1: cp1076 moving uplink from asw2-a-eqiad to asw-a-eqiad
15:02 cmjohnson1: cp1075 moving uplink from asw2-a-eqiad to asw-a-eqiad
15:01 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
15:01 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp107[56]\.eqiad\.wmnet
15:00 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp107[56]\.eqiad\.wmnet
15:00 jynus: stopping es2018 for upgrade
14:58 cmjohnson1: torrelay1001 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:57 cmjohnson1: ms-be1040 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:56 cmjohnson1: dbproxy1013 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:54 cmjohnson1: labstore1009 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:51 papaul: shutting down mw2184 for maintenance
14:50 cmjohnson1: db1066 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:49 cmjohnson1: db1118 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:46 cmjohnson1: db1116 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:44 cmjohnson1: labstore1008 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:43 bblack@neodymium: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
14:43 cmjohnson1: dbproxy1012 moving uplink from asw2-a-eqiad to asw-a-eqiad
14:41 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d T201979 ) (duration: 06m 03s)
14:40 cmjohnson1: dns1001 moving uplink from asw2-a eqiad to asw-a-eqiad
14:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=dns1001.wikimedia.org
14:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool es2018 (duration: 00m 55s)
14:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@166eafa]: Update mobileapps to a808c9d T201979 )
14:33 cmjohnson1: cloudelastic1001 moving uplink from asw2-a eqiad to asw2-a2
14:27 cmjohnson1: lvs1015 moving cross connect from asw2-a2 to asw2-a5 T201694
14:25 XioNoX: starting moving asw2-a-eqiad servers' uplinks for T201694
14:22 gehel: repooling wdqs[12]003
14:11 moritzm: rebooting labtestweb2001 for kernel security update
13:54 moritzm: rebooting labtestvirt2003 for kernel security update
13:46 moritzm: rebooting labtestservices2002/2003 for kernel security update
13:31 moritzm: rebooting seaborgium for kernel security update
13:15 moritzm: rebooting serpens for kernel security update
13:05 gehel: restarting blazegraph on wdqs[12]003
12:44 moritzm: upgrading wikidiff to 1.7.2 on labweb* (HHVM bytecode cache is pruned during update)
12:08 arturo: T201473 install a new version of `prometheus-pdns-exporter` (0.3) into jessie-wikimedia, due to errors in the postinst script
12:06 gehel: depooling wdqs[12]003 to catchup on updates
12:00 moritzm: installing ruby2.3 security updates
11:37 arturo: T201473 copy `prometheus-pdns-exporter` from trusty-wikimedia to jessie-wikimedia in reprepro
11:34 Amir1: EU mid-day SWAT is done
11:31 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/populateChangeTagDef.php: SWAT: Add option to populateChangeTagDef not to update the count (duration: 00m 53s)
11:12 jynus: stopping db2042 for maintenance T202051
11:09 moritzm: upgrading wikidiff to 1.7.2 on mw1339-mw1348 (HHVM bytecode cache is pruned during update)
11:03 arturo: manually delete glance rsync image cronjob from the glancesync user in labcontrol1001.wikimedia.org (leftover after glance merge in main/eqiad1)
10:43 moritzm: upgrading wikidiff to 1.7.2 on mw1285-mw1290 and mw1312-mw1317 (HHVM bytecode cache is pruned during update)
10:30 _joe_: restarting changeprop on scb1002, not listening on its tcp port
10:27 _joe_: restarting cpjobqueue on scb1002, not listening on its tcp port
10:00 volans: add jiji to the 'ops' LDAP group - T201849
09:46 gehel: all elasticsearch nodes reimaged (except elastic1029, waiting on memory issue) - T198391 / T193649 / T201991
09:16 gehel: reimaging elastic1022
08:40 moritzm: upgrading wikidiff to 1.7.2 on mw1319-mw1333 (HHVM bytecode cache is pruned during update)
08:37 ema: reboot cp2009 for kernel upgrade
08:35 vgutierrez: Reimaging radon as spare system - T202040
08:32 moritzm: uploaded jenkins 2.121.3 to apt.wikimedia.org (for jessie and stretch)
08:16 moritzm: upgrading wikidiff to 1.7.2 on mw1334-mw1338/mw1307/mw1318/ (HHVM bytecode cache is pruned during update)
07:49 gehel: reimaging elastic10(23|24)
07:45 volans@deploy1001: Finished deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8 (duration: 00m 31s)
07:45 volans@deploy1001: Started deploy [debmonitor/deploy@1f01fd1]: Release v0.1.8
07:27 moritzm: rebooting install1002 for kernel security update
07:24 moritzm: rebooting install2002 for kernel security update
05:12 _joe_: moving away corrupted commitlog file on aqs1007 cassandra-a instance, trying to restart it
02:38 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 16 02:38:46 UTC 2018 (duration 10m 17s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 08m 38s)
22:43 krinkle@deploy1001: Finished scap: rm StartProfiler.php - T201782 (duration: 31m 46s)
22:33 XioNoX: update old eqiad recdns IPs in cr1/2-eqiad bgp group Anycast4 with dns1001/2
22:11 krinkle@deploy1001: Started scap: rm StartProfiler.php - T201782
21:57 krinkle@deploy1001: Synchronized php-1.32.0-wmf.10: rm StartProfiler.php - T201782 (duration: 03m 01s)
21:48 krinkle@deploy1001: Synchronized wmf-config/: I173a02910c
21:46 krinkle@deploy1001: Synchronized scap/plugins/: I173a02910c
20:49 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I207421
19:07 gehel: reimaging elastic10(18|19|20)
17:20 XioNoX: Rollback: redirect ns2 to authdns1001 - T201608
17:16 moritzm: reboot eeden for kernel security update
17:07 XioNoX: redirect ns2 to authdns1001 - T201608
17:00 moritzm: reboot radon for kernel security update
16:50 XioNoX: redirect ns0 to authdns1001 - T201608
16:42 XioNoX: rollback: redirect ns1 to radon - T201608
16:35 moritzm: reboot authdns2001 for kernel security update
16:27 XioNoX: redirect ns1 to radon - T201608
14:58 gehel: reimaging elastic1028
14:50 ejegg: disabled CiviCRM dedupe jobs
14:47 ejegg: updated SmashPig free-standing deployment from 2736e41045 3603e191a3
14:13 gehel: reimaging elastic102[567]
13:47 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
13:35 moritzm: installing samba security updates
12:51 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw complete (T200037 )
12:22 moritzm: rebooting mw2190-mw2219 for kernel security update (also bundling wikidiff and apache updates)
12:19 jmm@puppetmaster1001: conftool action : set/pooled=inactive; selector: name=mw2184.codfw.wmnet
12:04 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.16/includes/changetags/ChangeTags.php: Swap SET and WHERE statements in ChangeTags::undefineTag (T201934) (duration: 00m 55s)
10:52 moritzm: rebooting mw2163-mw2189 for kernel security update (also bundling wikidiff and apache updates)
09:39 gehel: reimaging elastic103[012], this will trigger master re-election
09:17 moritzm: rebooting mw2135-mw2147 for kernel security update (also bundling wikidiff and apache updates)
08:59 moritzm: rebooting deployment-mediawiki-09
08:14 gehel: masking cassandra-a instance on aqs1007 since it is flapping - T201986
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 15 02:45:47 UTC 2018 (duration 10m 24s)
02:15 ejegg: updated payments-wiki from 6002d9edc0 46ed86f389
00:12 catrope@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/PageTriage/: SWAT: PageTriage fixes (T199357 , T201812 , T201560 , T201373 , T201253 ) (duration: 00m 51s)
23:55 TimStarling: restarted populateContentTables.php on s2
23:48 tzatziki: deleting three images for legal compliance
23:39 tzatziki: Correction: Change email for User:Textorus
23:38 tzatziki: Change password for User:Textorus
23:20 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable CentralNotice EventLogging at a low sample rate (0.01) (duration: 00m 50s)
23:20 fdans@deploy1001: Finished deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing (duration: 08m 27s)
23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable wgLegacyJavaScriptGlobals on all group0 wikis (T35837 ) (duration: 00m 54s)
23:11 fdans@deploy1001: Started deploy [analytics/refinery@21e07ae]: Deploying revert to prevent partition dropping jobs from failing
22:51 ejegg: updated payments-wiki from 7efaea7953 6002d9edc0
21:19 jgleeson: payments-wiki changed from 6393e83ce0 7efaea7953 a5bce0fc89
19:35 krinkle@deploy1001: Synchronized php-1.32.0-wmf.16/includes/cache/MessageCache.php: I6093113a T201893 (duration: 00m 52s)
19:11 mforns@deploy1001: Finished deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist (duration: 13m 04s)
19:06 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1008,service=gelf
18:58 mforns@deploy1001: Started deploy [analytics/refinery@a4d1d99]: adding hashing to EL whitelist
18:44 XioNoX: delete peer 4589 on cr2-esams (no more direct peering)
18:27 XioNoX: renumber v4 IP of 8560 on cr1-eqord
18:20 mforns@deploy1001: Finished deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70 (duration: 14m 27s)
18:15 XioNoX: bump PCCW max accepted prefixes on cr2-eqiad
18:13 XioNoX: bump PCCW max accepted prefixes on cr2-esams
18:10 XioNoX: re-activate peer 13285 on cr2-ulsfo
18:07 XioNoX: update NTP servers on pfw
18:05 mforns@deploy1001: Started deploy [analytics/refinery@cb57843]: corresponding to refinery-source v0.0.70
18:02 ejegg: re-enabled CiviCRM Omnimail import jobs
18:00 volans@deploy1001: Finished deploy [netbox/deploy@eae2c9d]: Fix broken dependency (duration: 00m 33s)
17:59 Trey314159: reindexing Polish wikis on elastic@eqiad and elastic@codfw (T200037 )
17:59 volans@deploy1001: Started deploy [netbox/deploy@eae2c9d]: Fix broken dependency
17:22 XioNoX: configuring eqiad A switch ports for T201694
17:19 gehel: restarting elasticsearch on elastic1050 (high load)
16:57 gehel: reimage of elastic103[345]
16:18 volans@deploy1001: Finished deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency (duration: 00m 34s)
16:17 volans@deploy1001: Started deploy [netbox/deploy@e2fd41d]: Security upgrade of dependency
16:01 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 23s)
15:59 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
15:39 volans@deploy1001: Finished deploy [netbox/deploy@792d4d5]: Security upgrade of dependency (duration: 01m 03s)
15:38 volans@deploy1001: Started deploy [netbox/deploy@792d4d5]: Security upgrade of dependency
15:01 moritzm: rebooting meitnerium/archiva.wikimedia.org for kernel security update
14:59 ema: cache_text eqiad: restart varnish-be without transient storage caps
14:53 moritzm: rebooting cloudelastic* for kernel update
14:50 gehel: reimage of elastic103[678]
14:41 bstorm@deploy1001: Finished deploy [striker/deploy@13da520]: (no justification provided) (duration: 01m 13s)
14:40 bstorm@deploy1001: Started deploy [striker/deploy@13da520]: (no justification provided)
14:35 Krinkle: Copying xenon/logs/daily/2018-*{all,load,index,api,RunSingleJob}.log from mwlog1001 to webperfX002 hosts
14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with full weight (duration: 00m 52s)
14:21 ema@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet,service=varnish-be
14:05 ema: restart varnish-be on cp108[79], fetch failures after child crashes
13:47 gehel: restarting elasticsearch on elastic1051 (overloaded)
13:21 gehel: restarting elasticsearch on elastic1043 (overloaded)
13:11 moritzm: upgrading wikidiff to 1.7.2 on mw1308-mw1311/mw1293-mw1296 (HHVM bytecode cache is pruned during update)
12:57 gehel: restarting tilerator on maps eqiad
12:54 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw complete (T200204 )
12:49 moritzm: upgrading wikidiff to 1.7.2 on snapshot hosts
11:58 moritzm: upgrading wikidiff to 1.7.2 on mw1276-mw1283 (HHVM bytecode cache is pruned during update)
11:40 TimStarling: restarted populateContentTables.php on s2 (T183488 )
11:20 zeljkof: EU SWAT finished
11:19 moritzm: upgrading wikidiff to 1.7.2 on mw1299-mw1306 (HHVM bytecode cache is pruned during update)
11:19 zfilipin@deploy1001: Synchronized vendor/perftools/xhgui-collector/external/header.php: SWAT: Fix "the the" typo in vendor/perftools/xhgui-collector/external/header.php (T201491) (duration: 00m 49s)
11:17 zfilipin@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Fix the the typo in wmf-config/CirrusSearch-common.php (T201491) (duration: 00m 51s)
11:14 moritzm: repooled mw1233 (debugging completed)
10:45 moritzm: repooled mw1227 (was probably overlooked to repool after previous debugging)
10:41 volans: upgraded cumin on labpuppetmaster* to fix cumin with the new openstack region - T201881
10:16 moritzm: upgrading wikidiff to 1.7.2 on mw1221-mw1235 (HHVM bytecode cache is pruned during update)
10:12 volans: uploaded cumin_3.0.2-2_amd64.deb to apt.wikimedia.org jessie-wikimedia
09:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1102 with low load (duration: 00m 50s)
09:32 jynus: stop and restart db1122 for maintenance
09:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1102 (duration: 00m 52s)
09:23 addshore: for i in {10000..12500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
09:20 moritzm: upgrading wikidiff to 1.7.2 on mw1238-mw1258 (HHVM bytecode cache is pruned during update)
09:17 addshore: for i in {6000..10000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
09:13 addshore: for i in {3000..6000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
09:08 addshore: for i in {1000..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
09:07 addshore: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki --skip-exists-check # T198301
08:51 addshore: addshore@labweb1001:~$ mwscript extensions/OATHAuth/maintenance/disableOATHAuthForUser.php --wiki=labswiki GoranSMilovanovic # T201122
08:19 moritzm: upgrading wikidiff to 1.7.2 on mw1266-mw1275 (HHVM bytecode cache is pruned during update)
07:56 moritzm: installing ghostscript security updates
07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:s7 and db1101:s8 (duration: 00m 52s)
06:45 _joe_: rolling restart of hhvm in eqiad api, high load
06:37 _joe_: depooling mw1233 from live traffic for debugging
05:13 TimStarling: killed populateContentTables.php for s2 at jcrespo's request
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 14 02:45:46 UTC 2018 (duration 10m 25s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 13m 57s)
00:35 aaron@deploy1001: Synchronized wmf-config/mc.php: Enable broadcasted mcrouter cache operations for test wikis and mw.org (duration: 00m 49s)
00:15 mutante: graphite2002 - stopping carbon-local-relay, running puppet to start it again to confirm no issues with gerrit:448779
00:09 aaron@deploy1001: Synchronized wmf-config/mc.php: Only do cache writes to mcrouter for all wikis (duration: 00m 52s)
23:26 urandom: Restart Cassandra in RESTBase dev env -- T201863
23:24 godog: restart carbon-c-relay on graphite1001 to mirror traffic to graphite1004 - T196484
23:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable wp10 and draftquality ORES models on testwiki (T198997 ) (duration: 00m 51s)
22:32 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/resources/src/startup/mediawiki.js: backport I3cd2d7
22:31 Trey314159: reindexing Indonesian wikis on elastic@eqiad and elastic@codfw (T200204 )
22:30 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/OutputPage.php: backport I3cd2d7
22:29 bawolff@deploy1001: Synchronized php-1.32.0-wmf.16/includes/ContentSecurityPolicy.php: backport I3cd2d7
22:29 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw complete (T200204 )
21:30 bawolff@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Deploy adjust CSP settings I74a79dc7defaa
21:29 mutante: elastic1049 - started ferm
19:32 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204 )
18:47 otto@deploy1001: Finished deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908 (duration: 11m 10s)
18:35 otto@deploy1001: Started deploy [analytics/refinery@c7f68b7]: camus - Add --check-java-opts and --check-emails-to option - T198908
17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (duration: 10m 54s)
17:15 thcipriani@deploy1001: Synchronized wmf-config/wikitech.php: wikitech.php: update keystone URLs (duration: 00m 53s)
17:05 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI
17:02 gehel@deploy1001: Finished deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
17:01 gehel@deploy1001: Started deploy [wdqs/wdqs@a62b1ac]: new version of wdqs GUI (wdqs1009 only)
15:41 otto@deploy1001: Finished deploy [analytics/refinery@a051125]: fix for T198908 (duration: 08m 06s)
15:33 papaul: shutting down lvs2009 to disable on board NICs
15:33 otto@deploy1001: Started deploy [analytics/refinery@a051125]: fix for T198908
15:10 otto@deploy1001: Finished deploy [analytics/refinery@9006a4e]: refinery changes for T198908 (duration: 10m 30s)
15:00 otto@deploy1001: Started deploy [analytics/refinery@9006a4e]: refinery changes for T198908
14:51 moritzm: installing ghostscript security updates
14:47 cmjohnson1: disabling puppet and shutting down db1052 for final decommissioning
14:25 moritzm: installing dpkg updates from stretch point release
14:10 arturo: T201504 disable keystone in main and eqiad1 deployments, all has been downtimed in icinga
14:09 andrewbogott: stopping nodepool, downtiming horizon for T201504
14:05 moritzm: upgrading mw1262-1265 to wikidiff 1.7.2 (T199801 )
12:56 gehel: start reimaging of elasticsearch / cirrus / eqiad cluster (RAID0 / Stretch) - T193649 / T198391
12:56 gehel: reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) completed - T193649 / T198391
12:53 moritzm: upgrading mw1261 servers to wikidiff 1.7.2 (T199801 )
12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2 (T199801 )
12:28 moritzm: upgrading mwdebug* servers to wikidiff 1.7.2
12:24 moritzm: uploaded hhvm-wikidiff2 1.7.2 to apt.wikimedia.org (source package name is still php-wikidiff2 for historical reasons) (T199801 )
12:06 zeljkof: EU SWAT finished
12:00 moritzm: rebooting cloudcontrol* for kernel security update
11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set mobile wordmark for pswikivoyage (T200152) (duration: 00m 52s)
11:47 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Translate extension on oldwikisource (T201814) (duration: 00m 50s)
11:42 moritzm: rebooting dbmonitor* hosts for kernel security update
11:29 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/: SWAT: Add missing PNG files in mobile logo folder (T200152) (duration: 00m 49s)
11:05 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
11:04 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
11:01 moritzm: rebooting kraz/irc.wikimedia.org for kernel security update
10:29 jynus: upgrade and restart db2036
08:42 jynus: upgrade and restart db1101
08:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:s7 and db1101:s8 (duration: 00m 51s)
07:16 legoktm@deploy1001: Synchronized php-1.32.0-wmf.16/maintenance/dumpTextPass.php: Add the 'full' option explicitly to dumpTextPass.php (T201803 ) (duration: 00m 58s)
06:53 ema: finish up cache reboots for kernel updates: cp5001.eqsin.wmnet,cp3035.esams.wmnet,cp[4021-4022].ulsfo.wmnet
06:46 oblivian@neodymium: conftool action : set/pooled=inactive; selector: name=restbase2003.codfw.wmnet
06:34 _joe_: powercycling restbase2003, in kernel panic
02:46 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 13 02:46:41 UTC 2018 (duration 10m 28s)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 14m 21s)
01:45 TimStarling: on mwmaint1001 running populateContentTables.php on all wikis using ~tstarling/pct-list T183488
15:43 gehel: full cache invalidation of maps tiles - T201772
06:15 ejegg: stopped CiviCRM omnimail jobs
21:50 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/UploadWizard: T201708 UBN fix (e498c7d
18:45 bblack: restart varnish-be on cp1087
18:44 bblack: restart varnish-be on cp1079
18:44 bblack: restart varnish-be on cp1075
18:41 bblack: restart varnish-be on cp1081
17:49 jgleeson: Released config update containing Google API Key (37a1a28dfb)
17:34 bblack: cp1085 + cp1089: restart varnish with wiped storage, repool
17:04 ema: restart varnish-be on cp1083
16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1089.eqiad.wmnet
16:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1085.eqiad.wmnet
16:36 bblack: depool cp1089, cp1085
16:12 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw abandoned (T200204 )
16:02 Trey314159: reindexing Malay wikis on elastic@eqiad and elastic@codfw (T200204 )
14:48 ema: powercycle cp5005, stuck rebooting
12:27 gehel: resetting management card on elastic2017 - T201671
12:18 gehel: restarting icinga on einsteinium - T196336
11:43 moritzm: installing busybox security updates
11:38 moritzm: installing jansson security updates for trusty
11:17 moritzm: installing ant security updates
11:06 moritzm: installing libxcursor security updates
09:35 mobrovac@deploy1001: Finished deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242 (duration: 03m 04s)
09:32 mobrovac@deploy1001: Started deploy [citoid/deploy@983d80c]: Remove the bibtex spec.yaml x-ample - T197242
09:27 jynus: upgrade and restart db1125
09:19 moritzm: rearmed keyholder on labpuppetmaster*
09:12 moritzm: rebooting labpuppetmaster1002 for kernel security update/microcode update
09:03 jynus: upgrade and restart db1124
08:58 moritzm: rebooting labpuppetmaster1001 for kernel security update/microcode update
08:29 jynus: upgrade and restart db2095
08:28 arturo: rebuild python-pykube for stretch-wikimedia and add it to apt.wikimedia.org T200660
08:03 jynus: upgrade and restart db2094
07:57 ema: powercycle cp3040, kernel crash
07:03 moritzm: installing mutt security updates
23:40 ejegg: updated payments-wiki from d78c98cf0f 6393e83ce0
23:33 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451810/ (duration: 00m 51s)
23:27 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451792/ (duration: 00m 51s)
23:12 maxsem@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-ps.svg: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/451791/ (duration: 00m 52s)
22:31 XioNoX: disable fpc1-fpc3 - T201145
22:09 bblack: re-enabling pybal on lvs1016
21:31 XioNoX: disable fpc3-fpc4 - T201145
21:21 XioNoX: disable fpc5-fpc6 - T201145
21:21 bblack: stop pybal on lvs1016 (should move services to lvs1006)
21:06 XioNoX: disable fpc4-fpc6 - T201145
21:01 XioNoX: enable fpc5-fpc6 - T201145
20:54 XioNoX: enable fpc3-fpc4 - T201145
20:49 XioNoX: enable fpc1-fpc3 - T201145
20:44 XioNoX: disable fpc3-fpc5 and fpc5-fpc7 - T201145
20:31 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library (duration: 07m 25s)
20:23 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@8c6a7f8]: add python snappy library
20:18 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1080.eqiad.wmnet
19:16 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.16 refs T191062
19:09 twentyafterfour: Preparing to deploy 1.32.0-wmf.16 to group2 wikis.
19:02 herron: logstash1008:~# systemctl restart logstash T200960
19:02 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add temporary debug logging to ThumbnailRenderJob T201305 (duration: 00m 57s)
18:58 niharika29@deploy1001: Synchronized php-1.32.0-wmf.16/includes/jobqueue/jobs/ThumbnailRenderJob.php: Add temporary debug logging T201305 (duration: 01m 11s)
18:49 bblack: remaining cache_misc sites will move to cache_text shortly - https://gerrit.wikimedia.org/r/451659
18:29 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config (duration: 03m 33s)
18:26 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@39be980]: Repair daemons failing to load logging config
18:23 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins (duration: 03m 35s)
18:19 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: temp fix while waiting for jenkins
18:17 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add media.farsnews.com to wgCopyUploadDomains T200872 (duration: 01m 00s)
18:15 niharika29@deploy1001: sync-file aborted: (no justification provided) (duration: 00m 01s)
17:59 XioNoX: connecting asw2-a5-eqiad to asw2-a6-eqiad - T201145
17:54 moritzm: installing gnupg security updates on trusty (Debian already fixed)
17:52 awight@deploy1001: Finished deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518 (duration: 31m 58s)
17:35 mforns@deploy1001: Finished deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68 (duration: 08m 22s)
17:26 mforns@deploy1001: Started deploy [analytics/refinery@9f4267c]: deploy refinery together with refinery-source v0.0.68
17:21 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef T191640 ) (duration: 06m 35s)
17:20 awight@deploy1001: Started deploy [ores/deploy@1712122]: fawiki wp10; ORES update: T201518
17:14 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@6581b28]: Update mobileapps to 616ffef T191640 )
15:14 mobrovac@deploy1001: Finished deploy [citoid/deploy@92e5071]: Record format stats (duration: 17m 07s)
14:57 mobrovac@deploy1001: Started deploy [citoid/deploy@92e5071]: Record format stats
14:52 moritzm: rebooting dns2002 for kernel security update/enabling microcode updates
14:50 bblack: all cpNNNN: max tcp per-flow rate reducing 1Gbps -> 256Mbps - https://gerrit.wikimedia.org/r/c/operations/puppet/+/451535
14:44 moritzm: rebooting dns2001 for kernel security update/enabling microcode updates
14:34 moritzm: rebooting maerlant for kernel security update/enabling microcode updates
14:28 herron: rolling reboot of mx servers for kernel updates
14:26 moritzm: rebooting nescio for kernel security update/enabling microcode updates
14:18 herron: rebooting fermium (lists) to switch out of -rt kernel variant
14:12 moritzm: rebooting dns4002 for kernel security update/enabling microcode updates
13:55 moritzm: rebooting dns4001 for kernel security update/enabling microcode updates
13:48 moritzm: rebooting dns5002 for kernel security update/enabling microcode updates
13:28 moritzm: rebooting dns5001 for kernel security update/enabling microcode updates
12:22 moritzm: rebooting ununpentium for kernel security update
12:14 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Do not leak local $wgWBShared… variables to the global scope (duration: 00m 56s)
12:13 moritzm: rebooting netmon1003 (servermon.wikimedia.org) for kernel security update
12:10 moritzm: rearmed keyholder on netmon1002
12:09 hoo@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RDF export for lexicographical data (T201153 ) (duration: 00m 56s)
12:06 moritzm: rebooting netmon1002 for kernel security update
12:05 hoo@deploy1001: Synchronized wmf-config/Wikibase.php: Enable RDF export for lexicographical data (T201153 ) (duration: 00m 58s)
12:05 moritzm: rearmed keyholder on netmon2001
11:57 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikibooks (T201562) (duration: 00m 55s)
11:50 moritzm: rebooting netmon2001 for kernel security update
11:42 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Update logo for hewikibooks (T201562) (duration: 01m 00s)
11:30 ema: resume rolling reboots of caches for numa_networking T193865
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Transwiki import in zhwikiversity (T201328) (duration: 00m 56s)
11:19 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove noratelimit from epcoordinator group on cswiki (T201010) (duration: 00m 58s)
11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles everywhere (T199909) (duration: 00m 58s)
10:57 mobrovac: truncating commons and others mobile tables - T201103
10:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 fully (duration: 00m 59s)
10:38 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2069 due to crash (duration: 01m 00s)
10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103 (duration: 06m 27s)
10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #5 - T201103
10:31 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103 (duration: 12m 11s)
10:31 moritzm: rebooting archiva1001 for kernel security update
10:25 jynus: disabling alerts for db2069
10:21 jynus: handling db2069 crash, no user impact
10:19 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #4 - T201103
10:19 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103 (duration: 04m 20s)
10:14 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #3 - T201103
10:14 mobrovac@deploy1001: Finished deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103 (duration: 08m 38s)
10:05 mobrovac@deploy1001: Started deploy [restbase/deploy@ece750a]: Drop mobile-sections, feed and media end points from non-WPs, take #2 - T201103
10:03 moritzm: rebooting url downloaders for kernel security update (alsafi, actinium, aluminium, alcyone)
10:02 mobrovac@deploy1001: Finished deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 04m 26s)
09:57 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
09:57 moritzm: rebooting dubnium for kernel security update
09:56 mobrovac@deploy1001: deploy aborted: Drop mobile-sections, feed and media end points from non-WPs - T201103 (duration: 06m 14s)
09:50 mobrovac@deploy1001: Started deploy [restbase/deploy@cb6b4b4]: Drop mobile-sections, feed and media end points from non-WPs - T201103
09:42 moritzm: rebooting pollux for kernel security update
09:24 moritzm: rebooting iron for kernel security update
09:23 moritzm: reset racadm on iron, com2 was unresponsive
09:08 moritzm: rebooting bast5001 for kernel security update
09:02 moritzm: rebooting bast4002 for kernel security update
08:57 moritzm: rebooting bast2001 for kernel security update
08:37 moritzm: rebooting bast2002 for kernel security update
07:26 ema: cp3032 mbox lagged: reboot w/ numa_networking
05:52 kartik@deploy1001: Finished deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 T201085 ) (duration: 03m 59s)
05:48 kartik@deploy1001: Started deploy [cxserver/deploy@957ff6a]: Update cxserver to 27813b6 T201085 )
05:03 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2004 and pc2005 after BIOS upgrade - T201387 (duration: 01m 00s)
03:23 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Aug 9 03:23:32 UTC 2018 (duration 10m 34s)
03:12 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 15m 23s)
02:38 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 16m 05s)
02:06 twentyafterfour: restarted apache on phab1001 to hotfix antivandalism bug
01:39 mutante: baham - installing BIOS upgrade (2.4.2 for Dell R320) - server is on role(spare) and the last that did not get the upgrade on T162850
00:14 reedy@deploy1001: Synchronized php-1.32.0-wmf.16/extensions/VisualEditor/: Bug fix (duration: 00m 58s)
00:05 twentyafterfour: finished phabricator upgrade
00:02 twentyafterfour: deploying phabricator release/2018-08-08/1
23:55 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: satwiki sitename (duration: 01m 05s)
23:48 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sq* collation (duration: 00m 56s)
23:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: extension registration config for GWToolset (duration: 00m 56s)
23:39 reedy@deploy1001: Synchronized wmf-config/extension-list: GWToolset to extension.json (duration: 00m 57s)
23:27 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove old or same image config (duration: 00m 56s)
23:22 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgVariantArticlePath for zhwikiversity (duration: 01m 05s)
21:04 twentyafterfour: 1.32.0-wmf.16 appears to be stable, no noticeable increase in errors logged.
20:59 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.16 refs T191062 (duration: 00m 57s)
20:58 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.16 refs T191062
20:49 James_F: Updated mobileapps to 162ebcf
20:48 jforrester@deploy1001: Finished deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf
20:40 jforrester@deploy1001: Started deploy [mobileapps/deploy@4ef02e1]: Update mobileapps to 162ebcf
19:57 _joe_: powercycling cp1077, network card problems
19:54 bblack: reboot cp1088 for network card
19:53 bblack: reboot cp1090 for network card
19:53 _joe_: powercycling cp1079, network card problems
19:46 bblack: rebooting cp1084 to debug eth hardware
19:35 XioNoX: fix interface description on asw-a-eqiad uplinks
19:19 volans: completed forced puppet run where it was failed
19:18 bblack: depooling front-edge traffic from eqiad in DNS - https://gerrit.wikimedia.org/r/c/operations/dns/+/451401
19:11 volans: force puppet run where it's still failed
19:01 twentyafterfour: Waiting to deploy the train until after I've established that network issues are resolved.
18:23 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103 (duration: 01m 29s)
18:22 ppchelko@deploy1001: Started deploy [changeprop/deploy@f0246f7]: Only rerender mobile-sections for wikipedia T201103
17:55 herron: performing rolling restart of logstash instances T200960
17:01 jforrester@deploy1001: Synchronized php-1.32.0-wmf.16/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.16 (duration: 00m 58s)
17:00 jforrester@deploy1001: Synchronized php-1.32.0-wmf.15/includes/logging/LogFormatter.php: SWAT T185049 Unbreak views of invalid title log entries, wmf.15 (duration: 01m 01s)
16:53 bblack: reimagine cp1071-4, cp1099
16:20 bblack: reimaging cp1060, cp1062-8
15:58 bblack: reimaging cp1050.eqiad.wmnet cp1052.eqiad.wmnet cp1053.eqiad.wmnet cp1054.eqiad.wmnet cp1055.eqiad.wmnet cp1059.eqiad.wmnet
15:54 bblack: downtimed all ipsec checks :P
15:45 bblack: reimaging cp1046-9
14:55 jynus: switching over, stopping, upgrading and restarting labsdb1011
13:34 papaul: BIOS update in progress on pc2005
13:10 papaul: BIOS update in progress on pc2004
12:53 jynus: switching over, stopping, upgrading and restarting labsdb1010
12:00 jynus: testing new read only check on labsdb wikireplicas
11:21 jynus: switching over, stopping, upgrading and restarting labsdb1009
11:14 marostegui: Stop MySQL on pc2005 for BIOS upgrade - T201387
11:14 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 - T201387 (duration: 00m 59s)
11:02 ariel@deploy1001: Finished deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation (duration: 00m 04s)
11:02 ariel@deploy1001: Started deploy [dumps/dumps@d6bd774]: fix up getConfiguration invocation
10:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 and db1123 with low load after maint (duration: 00m 57s)
10:11 ema: power-cycle cp2022, stuck rebooting
09:08 ema: begin rolling reboots of caches for numa_networking T193865
08:58 marostegui: Drop unused grants from 208.80.153.48
08:44 marostegui: Drop unused grants from 208.80.154.12
08:41 marostegui: Drop unused grants from 208.80.154.136
08:03 jynus: stop db1123 for provisioning and upgrade
07:55 jynus: stop db1100 for provisioning and upgrade
07:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 and db1123 (duration: 00m 58s)
07:27 marostegui: Stop MySQL on pc2004 - T201387
06:17 kartik@deploy1001: Finished deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba T199308 , T199512 , T199320 , T200665 , T200453 , T106437 ) (duration: 03m 32s)
06:13 kartik@deploy1001: Started deploy [cxserver/deploy@6a0cab1]: Update cxserver to 951fdba T199308 , T199512 , T199320 , T200665 , T200453 , T106437 )
05:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2004 - T201387 (duration: 00m 56s)
05:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2075 (duration: 01m 06s)
03:12 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 8 03:12:43 UTC 2018 (duration 10m 26s)
03:02 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.16) (duration: 16m 06s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 40s)
00:11 ejegg: updated payments-wiki from 8cb7c86b12 d78c98cf0f
23:46 ejegg: updated payments-wiki from 0c9c24d30c 8cb7c86b12
23:28 TimStarling: restarted populateContentTables.php for commonswiki, it died at rev_id 60164000
22:14 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1048.eqiad.wmnet
21:58 ejegg: updated CiviCRM from d626907f2c b60ceb3d2f
21:17 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.16 refs T191062
21:07 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1049.eqiad.wmnet
20:45 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.16 refs T191062 (duration: 68m 20s)
20:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1050.eqiad.wmnet
20:34 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1062.eqiad.wmnet
19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/jessie-wikimedia/main
19:45 herron: upload prometheus-logstash-exporter_0.1.2-1 to apt.wikimedia.org/stretch-wikimedia/main
19:36 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.16 refs T191062
19:30 andrewbogott: shutting down labnodepool1002 in advance of a rename. T201439
19:09 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes) (duration: 01m 51s)
19:07 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (all nodes)
19:07 otto@deploy1001: Finished deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only) (duration: 00m 20s)
19:06 otto@deploy1001: Started deploy [eventstreams/deploy@07033d4]: Deploying eventstreams with timestamp in Last-Event-ID (scb2001 only)
19:03 twentyafterfour: Branching 1.32.0-wmf.16
18:24 mutante: netbox - restored database from dump file - backed up and back-up (T190184 )
18:13 mutante: netbox - temp dropped databae to test restoring from dump
18:02 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1063.eqiad.wmnet
18:02 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1076.eqiad.wmnet
16:51 hoo: Deployed patch for T201418
16:50 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two (duration: 01m 51s)
16:48 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@22a50af]: bump to master, prep for deploy to cirrus servers, take two
16:47 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1052.eqiad.wmnet
16:47 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1075.eqiad.wmnet
16:44 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers (duration: 02m 54s)
16:41 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@65e6bb9]: bump to master, prep for deploy to cirrus servers
15:55 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019, db1122 and db1081 with full weight (duration: 00m 51s)
15:26 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1053.eqiad.wmnet
15:26 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1077.eqiad.wmnet
14:58 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1054.eqiad.wmnet
14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1079.eqiad.wmnet
14:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1064.eqiad.wmnet
14:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1078.eqiad.wmnet
14:26 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 47s)
14:18 herron: double again logstash jvm heap size to 1g and rolling restart logstash instances
14:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 and db1102 with low load after maintenance (duration: 00m 52s)
14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-be
14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-be
14:11 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-be
14:11 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-be
13:42 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=varnish-fe
13:42 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=varnish-fe
13:41 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=varnish-fe
13:41 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet
13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet
13:39 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet
13:39 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet
13:37 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1055.eqiad.wmnet,service=nginx
13:37 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1081.eqiad.wmnet,service=nginx
13:36 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1071.eqiad.wmnet,service=nginx
13:36 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1082.eqiad.wmnet,service=nginx
13:10 otto@deploy1001: Started restart [analytics/aqs/deploy@6fafc63]: Bouncing AQS for mediawiki_history index update
12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=hydrogen.wikimedia.org
12:58 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
12:52 bblack: rebooting authdns1001
12:44 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1002.wikimedia.org
12:41 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns1001.wikimedia.org
11:55 marostegui: Drop unused grants repl@208.80.155.117
11:19 zeljkof: EU SWAT finished
11:18 mobrovac@deploy1001: Synchronized rpc/RunSingleJob.php: RunSingleJob: remove unneeded base64 decoding (duration: 00m 49s)
11:17 marostegui: Remove unused repl grants for 10.64.0%
11:07 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove hewiki interface-editor group (T200698) (duration: 00m 49s)
10:51 jynus: shutdown and upgrade of db1081 (last message was wrong)
10:51 jynus: shutdown and upgrade of db1082
10:45 jynus: shutdown and upgrade of db1122
10:44 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 and db1081 (duration: 00m 49s)
10:33 godog: bounce logstash on logstash1007 for tests
10:19 jynus: shutting down es1019 for hw maintenance T201132
10:18 ema: reboot lvs-eqiad primaries for kernel upgrade
10:04 godog: reboot einsteinium for kernel upgrade
09:57 marostegui: Deploy schema change on db2075 - T67448 T114117 T5119
09:55 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2075 (duration: 00m 48s)
09:43 ema: reboot lvs-codfw primaries for kernel upgrade
09:15 _joe_: rolling restart of HHVM in the api-eqiad cluster
09:13 _joe_: restarting hhvm on mw1231, high load
09:10 _joe_: restarting hhvm on mw1226, then repooling it
09:07 ema: reboot lvs3001 for kernel upgrade
09:02 _joe_: depool mw1226 for investigation
08:50 ema: reboot lvs-eqsin primaries for kernel upgrade
08:29 gehel: start reimaging of elasticsearch / cirrus / codfw cluster (RAID0 / Stretch) - T193649 / T198391
08:28 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 49s)
08:28 ema: reboot lvs-ulsfo primaries for kernel upgrade
08:13 godog: reboot tegmen with a new kernel
07:37 jynus: restarted es1014 prometheus exporter (last message was wrong)
07:37 jynus: restarted es1019 prometheus exporter
07:29 volans: migrated puppetboard and debmonitor to cache_text - T164609
07:05 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2006 T200641 (duration: 00m 49s)
06:58 marostegui: Deploy schema change on db1075 (s3 master) T144010 T51190 T199368
06:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 (duration: 00m 50s)
06:53 ema: reboot lvs secondaries for kernel upgrade
05:21 marostegui: Deploy schema change on db1123 T144010 T51190 T199368
05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 (duration: 00m 50s)
03:46 TimStarling: on mwmaint1001 running populateContentTables.php concurrently on wikidatawiki and commonswiki (T183488 )
02:34 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Aug 7 02:34:24 UTC 2018 (duration 10m 31s)
02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 08m 46s)
01:06 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1065.eqiad.wmnet,service=nginx
01:06 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1083.eqiad.wmnet,service=nginx
01:05 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1072.eqiad.wmnet,service=nginx
01:05 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1084.eqiad.wmnet,service=nginx
00:29 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads on all wikis (duration: 00m 49s)
23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=nginx
23:42 James_F: SWAT complete.
23:42 bblack@neodymium: conftool action : set/weight=1; selector: dc=esams,cluster=cache_text,service=varnish-fe
23:41 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Revert PageImages: Make it possible to add extra namespaces, part II, fatalmonitor unhappy (duration: 00m 49s)
23:38 jforrester@deploy1001: sync-file aborted: SWAT PageImages: Add NS_CATEGORY for Commons T198716 (duration: 00m 04s)
23:35 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part II - I3f225d051
23:35 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=nginx
23:33 bblack@neodymium: conftool action : set/weight=1; selector: dc=eqiad,cluster=cache_upload,service=varnish-fe
23:32 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT PageImages: Make it possible to add extra namespaces, part I - I6c51c1fe1
23:30 jforrester@deploy1001: Synchronized wmf-config/extension-list: SWAT Load TimedMediaHandler i18n via static extension registration (duration: 00m 48s)
23:27 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Load TimedMediaHandler via static extension registration T140852 (duration: 00m 48s)
23:21 jforrester@deploy1001: Synchronized multiversion/: SWAT Delete multiversion/submodules.json (duration: 00m 48s)
23:15 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Add new wikimania wiki to CORS origins I493cadb871
23:11 ejegg: updated CiviCRM from 03012506b9 d626907f2c
23:07 James_F: jforrester@mwmaint1001 SWAT Updating sewiki collation via updateCollation.php T182431
23:06 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Update sewiki collation config T182431 (duration: 00m 51s)
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=varnish-fe
22:17 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=varnish-fe
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1084.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1082.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1078.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1076.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1086.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1088.eqiad.wmnet,service=nginx
22:16 bblack@neodymium: conftool action : set/weight=4; selector: name=cp1090.eqiad.wmnet,service=nginx
22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1066.eqiad.wmnet
22:09 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1085.eqiad.wmnet
22:09 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1073.eqiad.wmnet
22:08 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1086.eqiad.wmnet
20:39 arlolra: Updated Parsoid to e02124b T199849 , T198400 , T199577 , T199509 , T200403 , T201054 )
20:31 arlolra@deploy1001: Finished deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b
20:19 arlolra@deploy1001: Started deploy [parsoid/deploy@6b16b57]: Updating Parsoid to e02124b
20:00 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (duration: 10m 55s)
19:50 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater
19:46 gehel@deploy1001: Finished deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 22s)
19:46 gehel@deploy1001: Started deploy [wdqs/wdqs@d552447]: new version of wdqs GUI and updater (wdqs1009 only)
19:14 mutante: bast1002 - rebooting
19:04 mutante: bast1002 - the last log line is about bast1002, not bast1001
19:03 mutante: bast1001 - installing package updates, incl systemd,kernel
18:56 herron: rebooting fermium (lists) for security updates
18:54 herron: rolling reboot of mx servers for security updates
18:16 moritzm: rebooting radium for kernel security update
18:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable Special:Block Feedback Request (duration: 00m 52s)
18:09 mutante: rebooting bast3002 for kernel upgrade
18:08 mutante: bast2002 - installing package upgrades (future bastion, to be setup)
18:04 herron: double logstash jvm heap size from 256m to 512m and rolling restart of logstash instances T200960
17:57 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1067.eqiad.wmnet
17:57 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1087.eqiad.wmnet
17:57 mutante: rebooting bast4001
17:56 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1099.eqiad.wmnet
17:56 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1088.eqiad.wmnet
17:37 mobrovac@deploy1001: Finished deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357 (duration: 03m 29s)
17:36 moritzm: uploaded linux 4.9.110+deb9u1~wmf1 for jessie-wikimedia to apt.wikimedia.org
17:34 mobrovac@deploy1001: Started deploy [citoid/deploy@4f5ba15]: Use WorldCat for open search as well - T162357
17:04 gehel@deploy1001: Finished deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 33s)
17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@0d3c6a6]: new version of wdqs GUI and updater (wdqs1009 only)
16:53 andrewbogott: power down labservices1001 for thermal paste fix, T196252
16:53 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1068.eqiad.wmnet
16:52 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1089.eqiad.wmnet
15:54 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp1074.eqiad.wmnet
15:50 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp1090.eqiad.wmnet
15:50 bblack: pooling cp1090 (upload@eqiad, first of new hardware)
15:05 papaul: shutting down pc2006 for maintenance
15:02 bblack: rebooting cp1075-90
14:56 marostegui: Stop MySQL for onsite maintenance - T200641
14:30 godog: roll-restart thumbor after adding new private containers - T201187
14:07 marostegui: Reload haproxy to apply new configuration
11:32 zeljkof: EU SWAT finished
11:31 zfilipin@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: deployment-prep: Change urldownloader host (T201160) (duration: 00m 50s)
11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add suppressredirect permission to rollbacker and patroller at zhwiki (T201160) (duration: 00m 49s)
11:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 48s)
11:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
11:08 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Give hewiki interface-admins the rights interface-editors have (T200698 ) (duration: 00m 50s)
10:54 godog: repair sde on ms-be2040
10:38 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 38s)
10:33 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
10:33 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 04m 44s)
10:29 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
10:28 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 10m 51s)
10:18 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
10:17 mobrovac@deploy1001: Finished deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants (duration: 11m 12s)
10:06 mobrovac@deploy1001: Started deploy [restbase/deploy@478652a]: Add new wikis and fix lang fallback for lang variants
09:33 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462 (duration: 00m 44s)
09:32 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: CirrusSearch jobs: Increase checker concurrency to 10 - T198462
09:31 mobrovac@deploy1001: Finished deploy [cpjobqueue/deploy@62716a5]: (no justification provided) (duration: 00m 27s)
09:31 mobrovac@deploy1001: Started deploy [cpjobqueue/deploy@62716a5]: (no justification provided)
08:24 marostegui: Remove unused mysql users allowed to connect from 208.80.152.226
08:23 _joe_: restarting logstash on logstash1007, losing packets as well
08:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 (duration: 00m 51s)
08:03 _joe_: restarting logstash on logstash1009, losing packets again
07:51 _joe_: restarting logstash on logstash1008, losing packets again
06:49 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labsdb:s3 T144010 T51190 T199368
06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 (duration: 00m 53s)
04:29 TimStarling: on mwmaint1001 running populateContentTables.php on metawiki T183488
04:09 TimStarling: on mwmaint1001 running populateContentTables.php on mediawikiwiki T183488
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Aug 6 02:45:42 UTC 2018 (duration 10m 28s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 13m 40s)
02:41 andrewbogott: rebooting labservices1001 from mgmt
16:40 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/resourceloader/ResourceLoaderWikiModule.php: I5cb93c173
16:37 krinkle@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: I5cb93c173
14:50 godog: roll-restart logstash because of increasing packet loss
23:07 XioNoX: - restart asw2-b5-eqiad into loader - T201145
18:07 marxarelli: restarting jenkins on contint1001
18:05 thcipriani: restarting releases-jenkins
15:45 herron@puppetmaster1001: conftool action : set/pooled=yes; selector: name=mw1239.eqiad.wmnet
14:13 jynus: stopping haproxy on dbproxy1006
14:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.15/includes/Storage/DerivedPageDataUpdater.php: Fix article counting logic in DerivedPageDataUpdater (duration: 00m 50s)
13:06 ema: reboot cp3030 to SSBD-enabled microcode/kernel
12:51 ema: trafficserver 7.1.3+ds-4wm2 uploaded to stretch-wikimedia T200178
12:05 gehel: reimage relforge to stretch - T193649
11:27 jynus: setting dbstore1001:s1 mariadb as read_only=1
10:54 jynus: setting dbstore2002 instances as read only
10:39 hoo: Running populateSitesTable.php on all Wikidata clients for T201003
10:31 jynus: setting read only=1 on all instances on dbstore2001
09:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 (duration: 00m 49s)
09:50 jynus: testing new monitoring on dbstore2001
09:36 paravoid: asw2-b-eqiad: set disable for xe-2/0/5 (cloudvirt1023), xe-7/0/22 (cloudvirt1024), ge-8/0/23 (rdb1009)
09:36 jynus: disabling puppet on mariadb multiinstance hosts
08:24 filippo@neodymium: conftool action : set/pooled=no; selector: name=logstash1007.eqiad.wmnet
08:23 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
02:35 ejegg: updated CiviCRM from a0a7ea846a 03012506b9
00:57 mutante: restarted postgresql on netmon2001 as part of debugging issue with failed icinga replication check
23:26 maxsem@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaEvents/+/450080/ (duration: 00m 49s)
23:03 XioNoX: adding package anycast-healthchecker to stretch-wikimedia
22:44 XioNoX: replacing package python3-jsonlogger with python3-json-logger in stretch-wikimedia
22:21 mutante: webperf2002 - systemctl start proc-sys-fs-binfmt_misc.automount to fix "Check systemd degraded" Icinga check
22:11 XioNoX: importing python3-jsonlogger_0.1.9 into stretch-wikimedia
20:30 bd808@deploy1001: Finished deploy [striker/deploy@2329901]: Update Striker to 2329901 T177407 , T198076 , T190543 ) (duration: 01m 26s)
20:28 bd808@deploy1001: Started deploy [striker/deploy@2329901]: Update Striker to 2329901 T177407 , T198076 , T190543 )
20:23 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/VisualEditor/includes/ApiVisualEditorEdit.php: sync https://gerrit.wikimedia.org/r/450090/ to unbreak prod refs T201083 T191061 (duration: 00m 49s)
19:10 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.15 refs T191061
19:07 twentyafterfour: T191061 is free of blockers, proceeding with the train deployment: group2 wikis to 1.32.0-wmf.15
18:21 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=wikidatawiki and --old (T200680 )
18:20 Amir1: Morning SWAT is done
18:20 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ORES/maintenance/PurgeScoreCache.php: SWAT: Join decomposition on maintenance/PurgeScoreCache.php (T200680) (duration: 00m 57s)
18:15 gehel: un-banning and repooling elastic1030 - T201039
18:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix a typo in ORES models config (duration: 00m 57s)
16:00 milimetric@deploy1001: Finished deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script (duration: 07m 52s)
15:53 milimetric@deploy1001: Started deploy [analytics/refinery@5e5b5a9]: Quick fix for sqoop script
14:27 dcausse: restarting elastic1030 to trigger a master election
13:45 ema: bounce traffic back from lvs3004 to lvs3002
13:38 ema: reboot lvs3002 to SSBD-enabled microcode/kernel
13:30 reedy@deploy1001: Synchronized dblists/: Update size dblists (duration: 00m 55s)
13:18 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 fully (duration: 00m 56s)
13:12 ema: bounce traffic from lvs3002 to lvs3004 (SSBD-enabled)
13:01 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: fix wikimaniawiki wgserver and wgcanonicalserver (duration: 00m 56s)
12:58 marostegui: Sanitize zhwikiversity satwiki T199599 T198401
12:57 ema: reboot lvs3004 to SSBD-enabled microcode/kernel
12:51 dcausse: banning elastic1030 (master struggling)
12:48 marostegui: Sanitize wikimaniawiki - T201001
12:45 dcausse: depooling elastic1030 (master struggling)
12:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: update foundation url (duration: 00m 56s)
12:34 reedy@deploy1001: Synchronized wmf-config/missing.php: Update foundation urls (duration: 00m 55s)
12:21 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 25s)
12:19 Reedy: Wikis created T196748 T198401 T199599
12:18 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: fix id_internal (duration: 00m 55s)
12:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis!
12:11 reedy@deploy1001: Synchronized wmf-config/: New wikis (duration: 00m 55s)
12:10 reedy@deploy1001: Synchronized dblists/: new wikis (duration: 00m 55s)
12:08 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: add id_internal (duration: 00m 55s)
12:07 reedy@deploy1001: Synchronized langlist: Add sat (duration: 00m 55s)
12:06 reedy@deploy1001: Synchronized static/images/project-logos/: New logos! (duration: 00m 57s)
11:22 godog: temporarily remove multiline filter from logstash100[789] and bump pipeline workers to 4 - T200960
11:21 moritzm: installing jansson security updates on trusty
11:17 godog: temporarily remove multiline filter from logstash to allow using multiple workers - T200960
11:16 moritzm: installing tomcat8 security updates
11:12 zeljkof: EU SWAT finished
11:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable moved paragraph detection for inline diffs on beta cluster (T200975) (duration: 00m 58s)
11:03 godog: comment "pipeline.workers: 1" from logstash1007 - T200960
10:47 godog: test bumping heap to 512mb on logstash1007 - T200960
10:35 volans@deploy1001: Finished deploy [netbox/deploy@ac54feb]: Security upgrade of dependency (duration: 00m 05s)
10:35 volans@deploy1001: Started deploy [netbox/deploy@ac54feb]: Security upgrade of dependency
10:30 volans@deploy1001: Finished deploy [debmonitor/deploy@73b640d]: Release v0.1.7 (duration: 01m 01s)
10:29 volans@deploy1001: Started deploy [debmonitor/deploy@73b640d]: Release v0.1.7
10:25 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.7 (duration: 00m 51s)
10:24 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.7
10:19 godog: test bumping rmem_default to 4MB on logstash1007 - T200960
10:11 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: Consistency (duration: 00m 55s)
10:10 reedy@deploy1001: Synchronized php-1.32.0-wmf.14/languages/Language.php: Revert out deprecation warning of hook (duration: 00m 57s)
09:18 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert again (duration: 00m 55s)
09:14 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org ) (duration: 22m 08s)
09:14 legoktm@deploy1001: Scap failed!: 10/11 canaries failed their endpoint checks(http://en.wikipedia.org )
08:52 legoktm@deploy1001: Started scap: try once more
08:40 legoktm@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/LabeledSectionTransclusion/: revert (duration: 00m 56s)
08:36 legoktm@deploy1001: scap failed: RuntimeError Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org ) (duration: 24m 33s)
08:36 legoktm@deploy1001: Scap failed!: 11/11 canaries failed their endpoint checks(http://en.wikipedia.org )
08:14 marostegui: Deploy schema change on db2043 (s3 codfw master) this will generate lag on codfw:s3 T144010 T51190 T199368
08:11 legoktm@deploy1001: Started scap: LST: Use modern i18n mechanisms for localization (T198173 , T200960 )
07:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 with low load (duration: 00m 56s)
07:20 TimStarling: restarting logstash on logstash1008 and logstash1009
07:06 TimStarling: on logstash1007 increased net.core.rmem_default from 212992 to 851968 in soft state and restarted logstash
06:59 TimStarling: on logstash1007 restarting logstash
06:58 jynus: stop db1092 for reimage
06:53 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
06:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 00m 55s)
06:18 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labsdb:s1 T144010 T51190 T199368
06:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 00m 54s)
06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 (duration: 00m 55s)
06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 (duration: 00m 57s)
05:30 TimStarling: on mwmaint1001 running populateContentTables.php as described in T183488
05:14 TimStarling: restarted stashbot
05:07 TimStarling: ran patch-slot-origin.sql on labswiki and labtestwiki to fix editing on labswiki (some SAL entries missing)
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 42s)
23:39 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 56s)
23:37 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscodeJob.php: SWAT: Work around transcode failures in newer ffmpeg (duration: 00m 57s)
23:17 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Re-enable VP8 video transcodes to fix playback regression (duration: 00m 56s)
22:58 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7
22:57 krinkle@deploy1001: Synchronized wmf-config: I4db5d03f8af7
20:36 twentyafterfour: T191061 Finished deploying 1.32.0-wmf.15 to group1 wikis. Fatalmonitor is quiet and everything appears to be stable. See you tomorrow for group2.
20:24 twentyafterfour@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.15 refs T191061 (duration: 00m 55s)
20:23 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.15 refs T191061
20:17 ejegg: updated payments-wiki from 626ff1cb72 0c9c24d30c
20:10 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 T200464 T200459 ) (duration: 05m 42s)
20:04 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@c2448e0]: Update mobileapps to 282f368 T200464 T200459 )
19:53 twentyafterfour@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/Collection/: Sync Change: https://gerrit.wikimedia.org/r/449796 Bug: T189636 unblocks: T191061 (duration: 00m 57s)
19:07 twentyafterfour: preparing to deploy 1.32.0-wmf.15 to group1 wikis. Gonna merge a couple of patches first to fix some logspam.
food: updated DjangoBannerStats from 73d74f1b5e 02be6cbb74
18:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4 (duration: 03m 21s)
18:10 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #4
18:09 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3 (duration: 06m 27s)
18:03 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #3
18:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2 (duration: 05m 08s)
17:58 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens, take #2
17:57 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens (duration: 12m 23s)
17:47 ejegg: updated DjangoBannerStats from 43e47f98a2 73d74f1b5e
17:45 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c]: Multi-content bucket: delete old revisions when a new render happens
17:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens (duration: 02m 52s)
17:42 mobrovac@deploy1001: Started deploy [restbase/deploy@4c9966c] (dev-cluster): Multi-content bucket: delete old revisions when a new render happens
17:22 aaron@deploy1001: Synchronized wmf-config/mc.php: Use mcrouter for cache reads for test wikis (duration: 00m 55s)
17:19 XioNoX: enable puppet on all cp2* servers after merging https://gerrit.wikimedia.org/r/c/operations/puppet/+/449760 - T195365
16:51 mforns: Finished deploying refinery using scap, then refinery-deploy-to-hdfs
16:41 vgutierrez: Turn off AES128-SHA support - T192555 && T147202
16:38 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 55s)
16:36 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/PageTriage/modules/ext.pageTriage.views.list/ext.pageTriage.listControlNav.js: SWAT: Fix namespaces shown in NPP filters T200826 T200821 (duration: 00m 58s)
16:34 mforns@deploy1001: Finished deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist (duration: 07m 54s)
16:26 mforns@deploy1001: Started deploy [analytics/refinery@7fba0fc]: fixing EL sanitization whitelist
16:24 mforns: deploying analytics-refinery using scap
16:08 moritzm: upgrading proton* to chromium 68.0.3440.75-1~deb9u1 (new release tested successfully in deployment-prep)
15:50 moritzm: installing libarchive-zip-perl security updates
15:43 akosiaris: upload mapnik_3.0.20+ds-1 to apt.wikimedia.org/stretch-wikimedia/main T200284
15:16 XioNoX: disable puppet on all codfw cp* servers - T195365
14:52 elukey: created user 'research' on db1108 (eventlogging slave - only select grant for the log database)
13:49 moritzm: upgrade application servers in beta to wikidiff 1.7,2 (T199801 )
13:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 (duration: 00m 55s)
13:29 marostegui: Deploy schema change on db1083 T144010 T51190 T199368
13:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 00m 55s)
13:02 volans: restarting ircecho on einsteinium to unblock icinga IRC bot
12:59 moritzm: installing tomcat7 security updates on jessie
12:35 marostegui: Deploy schema change on db1105:3311 T144010 T51190 T199368
12:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 (duration: 00m 55s)
12:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 (duration: 00m 55s)
12:16 moritzm: installing fuse security updates
12:16 marostegui: Deploy schema change on db1099:3311 T144010 T51190 T199368
12:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 (duration: 00m 56s)
11:56 zeljkof: EU SWAT finished
11:54 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: CleanupParent for draftquality model when PageTriage is used (T199357) (duration: 00m 56s)
11:45 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.15/skins/MinervaNeue/: SWAT: Restore page issues (T200867) (duration: 00m 57s)
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on the isiZulu Wikipedia (T200522) (duration: 00m 56s)
11:20 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Fix config for $wgLocalisationUpdateRepositories (T148965) (duration: 00m 57s)
11:17 moritzm: installing openjpeg2 security updates on jessie
10:44 ema: reboot cp1045 for kernel update
10:40 marostegui: Reload haproxy on dbproxy1008 and dbproxy1003
09:50 moritzm: installing wireshark security updates
09:50 marostegui: Deploy schema change on db2048 (s1 codfw master) with replication, this will generate lag on codfw s1 T144010 T51190 T199368
09:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 00m 55s)
09:41 ema: cp3044 (upload) repooled after upgrade to stretch T200445
09:39 marostegui: Run community_metrics.sh on phab1001
09:36 moritzm: installing clamav security updates
09:19 marostegui: Deploy schema change on db1092 T144010 T51190 T199368
09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 (duration: 00m 55s)
09:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 (duration: 00m 56s)
08:38 elukey: restart hadoop-yarn-nodemanager on analytics10[31-77] to apply the new memory settings
08:37 marostegui: Deploy schema change on db1109:3318 T144010 T51190 T199368
08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 (duration: 00m 55s)
08:22 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php --sleep 2 (T193873 )
08:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php --sleep 3 (T193873 )
08:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 56s)
08:16 marostegui: Stop MySQL on db2078
07:59 elukey: restart hadoop-yarn-nodemanager on analytics10[28-30] to test new memory settings
07:12 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 55s)
07:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1052 from config as it will be decommissioned - T199861 (duration: 00m 56s)
07:09 marostegui: Remove db1052 from tendril - T199861
06:59 elukey: restart eventlogging on eventlog1002 to pick up new logging settings
06:58 elukey@deploy1001: Finished deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/ (duration: 00m 07s)
06:58 elukey@deploy1001: Started deploy [eventlogging/analytics@762ca2b]: Deploy https://gerrit.wikimedia.org/r/#/c/eventlogging/+/449422/
06:47 marostegui: Deploy schema change on db1087 with replication, this will generate lag on labs:s8 T144010 T51190 T199368
06:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 55s)
06:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic db1089 (duration: 00m 55s)
06:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 (duration: 00m 55s)
06:22 brion: restarting transcode batch job (errors believed caused by hhvm config restarts)
06:11 brion: stopping requeueTranscodes.php for now
06:11 brion: seeing spike in errors on video scalers
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1089 (duration: 00m 57s)
05:02 marostegui: Stop MySQL on db1089 for a binlog format change (and also upgrade kernel)
05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 00m 57s)
04:47 marostegui: Stop MySQL on db1052 to copy its content to dbstore1001 - https://phabricator.wikimedia.org/T199861
04:46 marostegui: Deploy schema change on db1101:3318 T144010 T51190 T199368
04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 (duration: 01m 06s)
03:17 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Aug 1 03:17:12 UTC 2018 (duration 10m 24s)
03:06 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.15) (duration: 14m 45s)
02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 14m 25s)
00:15 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813 ) (duration: 00m 56s)
00:09 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.15/extensions/ParserFunctions/includes/ExtParserFunctions.php: SWAT: Remove & from $mwDefault variable assignment T200772 (duration: 00m 57s)
23:55 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/TimedMediaHandler/WebVideoTranscode/WebVideoTranscode.php: SWAT: Workaround for job queue reporting 0 length (T200813 ) (duration: 00m 57s)
23:41 ejegg: updated payments-wiki from 6afaca3de3 626ff1cb72
23:32 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgCiteResponsiveReferences for Meta-Wiki T200707 (duration: 00m 56s)
23:21 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles on Meta-Wiki T200613 (duration: 00m 57s)
22:56 eileen: update CiviCRM civicrm revision changed from 24b497072d a0a7ea846a 35a258d015
21:37 twentyafterfour@deploy1001: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.15
21:05 twentyafterfour@deploy1001: Finished scap: testwikis wikis to 1.32.0-wmf.15 (duration: 44m 49s)
20:20 twentyafterfour@deploy1001: Started scap: testwikis wikis to 1.32.0-wmf.15
19:00 urandom: enabling `unchecked_tombstone_compaction` on enwiki_T_mobile__ng_remaining -- T192689
18:20 XioNoX: stopping puppet across the fleet for puppetmaster1001 uplink move
18:04 XioNoX: stopping pybal on lvs1002
18:01 XioNoX: repool lvs1001
17:53 XioNoX: stopping pybal on lvs1001
17:34 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
17:19 twentyafterfour: branching 1.32.0-wmf.15 refs T191061
17:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all hosts in row B - T183585 (duration: 00m 51s)
16:40 elukey: repool druid1005 after network maintenance
16:26 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,service=pdns_recursor,name=chromium.wikimedia.org
16:25 godog: repool thumbor100[12] - T183585
16:25 elukey: pool eventbus on kafka1002 after network maintenance
16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
16:24 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
16:16 elukey: precautionary restart of eventbus on kafka1002 after network downtime (DNS name res errors, Kafka broker conn issues, etc..)
16:07 marostegui: Reload dbproxy1003 and dbproxy1008
16:06 ema: reboot cp3044 for kernel update
16:01 ayounsi@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
15:51 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=druid-public,service=druid-public-broker,name=druid1005.eqiad.wmnet
15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=cassandra,name=aqs1004.eqiad.wmnet
15:50 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=aqs,service=aqs,name=aqs1004.eqiad.wmnet
15:46 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=eventbus,service=eventbus,name=kafka1002.eqiad.wmnet
15:45 ayounsi@puppetmaster1001: conftool action : set/pooled=no; selector: dc=eqiad,cluster=prometheus,service=prometheus,name=prometheus1004.eqiad.wmnet
15:36 ema: power-cycle cp3031, stuck rebooting
15:29 _joe_: restarting apache on phab1001
15:20 godog: depool thumbor100[12] ahead of switch move - T183585
15:02 XioNoX: starting the eqiad row B servers move - T183585
13:51 moritzm: installing ffmpeg 3.2.12 security updates on video scalers
13:34 akosiaris: repool mathoid eqiad cluster in discovery
13:18 akosiaris: upgrade kubernetes1004 to 1.9.9
13:14 akosiaris: upgrade kubernetes1002 to 1.9.9
13:14 akosiaris: upgrade kubernetes1003 to 1.9.9
13:03 akosiaris: upgrade kubernetes1001 to 1.9.9
12:58 akosiaris: upgrade chlorine.eqiad.wmnet to kubernetes 1.9.9
12:54 akosiaris: upgrade argon.eqiad.wmnet to kubernetes 1.9.9
12:45 akosiaris: depool eqiad mathoid in discovery
12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1005.eqiad.wmnet
12:29 gehel@puppetmaster1001: conftool action : set/weight=15; selector: dc=eqiad,cluster=wdqs,name=wdqs1004.eqiad.wmnet
12:29 gehel: rebalance LVS weights to send less traffic to wdqs1003 - T200563
12:13 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove logging now in mw core (duration: 00m 48s)
12:11 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: wmg - for Sentry (duration: 00m 48s)
12:09 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove wmg - for Sentry (duration: 00m 48s)
12:06 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Load Sentry with wfLoadExtension (duration: 00m 48s)
12:05 reedy@deploy1001: Synchronized wmf-config/liquidthreads.php: Remove lqt cruft, load with wfLoadExtension (duration: 00m 48s)
11:44 gehel: restarting blazegraph on wdqs1003 - JVM unresponsive
11:38 zeljkof: EU SWAT finished
11:37 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Introduce autopatrolled on bnwikisource (T199475) (duration: 00m 48s)
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos for hewiktionary (T200470) (duration: 00m 48s)
11:24 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Upload HD logos for hewiktionary (T200470) (duration: 00m 48s)
11:16 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Milestone logo for atjwiki (T200713) (duration: 00m 49s)
11:10 zfilipin@deploy1001: Synchronized static/images/project-logos/hewikiquote.png: SWAT: Update hewikiquote logo (T200296) (duration: 00m 50s)
10:55 _joe_: test log
10:40 akosiaris@deploy1001: scap-helm mathoid finished
10:40 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
10:39 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
10:39 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
09:46 ema: reboot cp1045 (jessie) for kernel updates
09:29 ema: upload varnishkafka 1.0.13-1 to stretch-wikimedia T200445 T186250
08:44 jynus: stop and reimage db1104 to stretch
08:17 akosiaris: repool mathoid codfw in discovery, the kubernetes upgrade is done
08:08 akosiaris: upgrade kubernetes2004 to 1.9.9
08:05 akosiaris: upgrade kubernetes2003 to 1.9.9
08:00 akosiaris: upgrade kubernetes2002 to 1.9.9
07:55 ema: reboot cp2018 for kernel updates
07:54 moritzm: reboot cp1008 for kernel tests
07:53 akosiaris: upgrade kubernetes2001 to 1.9.9
07:50 ema: reboot cp2025 for kernel updates
07:47 akosiaris: upgrade acrux.codfw.wmnet to kubernetes 1.9.9
07:44 jynus: stopping db1118 mariadb and starting mysql there
07:36 moritzm: reboot multatuli for kernel tests
07:35 akosiaris: upgrade acrab.codfw.wmnet to kubernetes 1.9.9
07:32 moritzm: run decomission_appserver on terbium
07:27 akosiaris: backup kubetcd2001 etcd data before kubernetes upgrade
07:24 akosiaris: depool codfw mathoid from discovery for kubernetes upgrade
07:10 moritzm: installing mutt security updates
07:05 marostegui: Remove unused and old grants from codfw hosts - T146149 #4456082
06:49 jynus: dropping unnecesary tendril web users from tendril db backends
06:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool row B es hosts, reduce db1089 load (duration: 00m 49s)
06:11 moritzm: installing ant security updates
05:01 marostegui: Deploy schema change on db1099:3318 T144010 T51190 T199368
04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all hosts in row B - T183585 (duration: 00m 50s)
04:47 marostegui: Stop change_tag_def populate maintenance script for wikidata and enwiki - T193873
02:37 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 31 02:37:40 UTC 2018 (duration 10m 20s)
02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 08m 12s)
01:20 ejegg: updated payments-wiki from 344233cff8 6afaca3de3
00:44 thcipriani@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Switch in WebM VP9/Opus video transcodes to replace WebM VP8/Vorbis T63805 (duration: 00m 48s)
00:39 thcipriani@deploy1001: Finished scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes (duration: 33m 44s)
00:07 ejegg: updated payments-wiki from 8c4d6aaa13 344233cff8
00:05 thcipriani@deploy1001: Started scap: SWAT: Convert to extension.json T87981 Use maps for wgEnabledTranscodeSet T118080 Adjust VP9 encoding for speed, quality More conservative max bitrate for VP9 video transcodes
23:35 XioNoX: re-enabling puppet on cp40* - T195365
23:27 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/RevisionSlider/modules: SWAT: RevisionSlider: Fix missing pin icon T200263 (duration: 00m 49s)
23:15 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (2) T199919 (duration: 00m 49s)
22:35 XioNoX: applying static route + fixed MTU to cp4025 - T195365
22:29 XioNoX: - puppet disabled on cp40* hosts - T195365
22:10 thcipriani: restarting jenkins after plugin updates
21:55 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt) (duration: 00m 10s)
21:55 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (cobalt)
21:51 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only) (duration: 00m 09s)
21:51 thcipriani@deploy1001: Started deploy [gerrit/gerrit@10e3207]: Updating go-import plugin (gerrit2001 only)
21:01 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration (duration: 04m 12s)
20:56 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@debb5b0]: Update spark.yaml with latest training configuration
20:41 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point (duration: 02m 30s)
20:39 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@69ac0d5]: fix wrong yaml import in cli entry point
20:14 ejegg: updated CiviCRM from d76e3997e6 24b497072d
20:10 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon (duration: 01m 35s)
20:09 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@61c5f47]: ship missing bc alias for deployed daemon
19:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3 (duration: 06m 25s)
19:47 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 3
19:46 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2 (duration: 05m 06s)
19:40 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 attempt 2
19:40 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465 (duration: 14m 31s)
19:26 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b]: Language variants for summaries T198465
19:24 ppchelko@deploy1001: Finished deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465 (duration: 03m 05s)
19:22 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided) (duration: 01m 28s)
19:21 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@b1e18ca]: (no justification provided)
19:20 ppchelko@deploy1001: Started deploy [restbase/deploy@9f9685b] (dev-cluster): Language variants for summaries T198465
19:19 ebernhardson@deploy1001: Finished deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided) (duration: 01m 32s)
19:17 ebernhardson@deploy1001: Started deploy [search/mjolnir/deploy@7aa39b7]: (no justification provided)
18:57 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/WikimediaMessages/WikimediaMessages.hooks.php: SWAT: Escape Special:Block Feedback Request Message T194301 (duration: 00m 49s)
18:25 thcipriani@deploy1001: Synchronized wmf-config/mc.php: SWAT: Make all wikis write to both nutcracker and mcrouter (3) T198239 (duration: 00m 48s)
18:10 krinkle@deploy1001: Finished deploy [performance/navtiming@f79b313]: (no justification provided) (duration: 00m 05s)
18:10 krinkle@deploy1001: Started deploy [performance/navtiming@f79b313]: (no justification provided)
17:12 XioNoX: repool ulsfo
17:11 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (duration: 09m 17s)
17:02 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI
17:01 gehel@deploy1001: Finished deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
17:00 gehel@deploy1001: Started deploy [wdqs/wdqs@137780f]: new version of wdqs GUI (wdqs1009 only)
15:57 ema: pool cp2012, upgraded to stretch T200445
15:50 ema: reboot cp2012 for kernel update
15:27 jynus: restart and upgrade mariadb at db1118
15:14 bblack: reboot cp1075
15:08 godog: upgrade hp raid firmware on ms-be1028 - T141756
15:05 Amir1: running ladsgroup@mwmaint1001:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php on all wikis
13:58 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
13:39 ema: reboot cp2006 for kernel update
12:59 zeljkof: EU SWAT finished
12:58 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/AbuseFilter/: SWAT: Fix jQuery selector when editing filters (T200604) (duration: 00m 55s)
12:55 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/FileImporter/: SWAT: Fix flipped array indexes in template removal code (T200406) (duration: 00m 57s)
12:45 godog: fix group write on deploy1001 /srv/mediawiki-staging/php-1.32.0-wmf.14/.git/objects
12:40 godog: reboot ms-be2040 with page_poisoning=1 - T199198
12:33 tgr@deploy1001: Finished scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default (duration: 60m 10s)
11:39 ema: pool cp2006, upgraded to stretch T200445
11:39 moritzm: upgrading intel-microcode to 20180703
11:33 tgr@deploy1001: Started scap: T190015 Create separate user group for editing sitewide CSS/JavaScript that does not include administrators by default
11:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from change_tag_def everywhere (T199334) (duration: 00m 55s)
11:10 volans: upgrading cumin to 3.0.2 on the remaining cumin masters (neodymium, labpuppetmaster*)
11:01 volans: upgrading cumin to 3.0.2 on sarin
10:50 volans: uploaded cumin_3.0.2-1_amd64.deb to apt.wikimedia.org jessie-wikimedia
10:25 godog: run xfs_repair on sdc1 on ms-be2040 - T199198
10:04 akosiaris@deploy1001: scap-helm help finished
10:04 akosiaris@deploy1001: scap-helm help cluster codfw completed
10:04 akosiaris@deploy1001: scap-helm help cluster eqiad completed
10:04 akosiaris@deploy1001: scap-helm help [namespace: help, clusters: eqiad,codfw]
10:03 akosiaris: upgrade kubernetes staging cluster to 1.9.9
10:01 akosiaris: reboot kubestage1001 for kubernetes upgrade
09:59 akosiaris: reboot kubestage1002 for kubernetes upgrade
09:31 moritzm: uploaded intel-microcode 20180703
09:30 jynus: migrate dbtree to dbmonitor1001
09:25 akosiaris@deploy1001: scap-helm mathoid finished
09:14 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
09:13 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
09:09 akosiaris@deploy1001: scap-helm mathoid upgrade production stable/mathoid [namespace: mathoid, clusters: eqiad,codfw]
09:08 akosiaris: upgrade mathoid helm chart from 0.0.5 to 0.0.9
09:07 akosiaris@deploy1001: scap-helm mathoid finished
09:07 akosiaris@deploy1001: scap-helm mathoid cluster codfw completed
09:07 akosiaris@deploy1001: scap-helm mathoid cluster eqiad completed
09:07 akosiaris@deploy1001: scap-helm mathoid upgrade -h [namespace: mathoid, clusters: eqiad,codfw]
08:49 marostegui: Deploy schema change on dbstore1002:s8 T144010 T51190 T199368
08:07 elukey@deploy1001: Finished deploy [eventlogging/analytics@54d43e4]: Band aid for T200630 (duration: 00m 05s)
08:07 elukey@deploy1001: Started deploy [eventlogging/analytics@54d43e4]: Band aid for T200630
07:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Give some traffic to db1120 to on x1 (duration: 00m 53s)
07:43 moritzm: installing opencv security updates
07:39 moritzm: installing mercurial security updates
07:32 marostegui: Deploy schema change on db2045 (s8 codfw master) this will generate lag on s8 codfw T144010 T51190 T199368
07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Add db1120 to x1 T196376 (duration: 00m 54s)
07:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Add db1120 to x1 T196376 (duration: 00m 59s)
05:28 marostegui: Deploy schema change on db1068 (commonswiki master) - T51190
05:00 marostegui: Deploy schema change on db1062 (s7 primary master) T144010 T51190 T199368
03:13 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 30 03:13:42 UTC 2018 (duration 10m 29s)
03:03 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 33s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 11m 41s)
09:56 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2006 T200641 (duration: 00m 57s)
02:35 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Neuter EducationProgram T188410 (duration: 00m 58s)
17:49 Krinkle: Navigation Timing alerts indicate we're losing a significant about of traffic
17:35 elukey: restart eventlogging on eventlog1002 after tons of kafka disconnects (still not clear what happened)
08:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 fully (duration: 00m 57s)
22:05 ejegg: updated payments-wiki from f0138ab066 8c4d6aaa13
21:39 ejegg: updated payments-wiki from 5e01f50983 f0138ab066
20:40 SMalyshev: re-pooled wdqs1003
19:04 mutante: terbium is being removed from ferm rules on elasticsearch/relforge, logstash/collector, mariadb/labtestwikitech and mw-maintenance itself (T192092 )
18:57 SMalyshev: temporarily depooled wdq1003 to let it catch up with lag
18:15 mutante: syncing home dirs from mwmaint1001 to mwmaint2001 (once, manually, not currently set to auto-sync, but to keep another copy of former terbium homes in case we failover) (T192092 )
18:12 mutante: terbium: deleting rsyncd config fragment, stopping rsyncd, not used anymore, decom prep
17:06 gehel: repooling wdqs1003
16:43 gehel: depooling wdqs1003 to let it catch up on update lag
15:24 joal@deploy1001: Finished deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric (duration: 04m 12s)
15:20 joal@deploy1001: Started deploy [analytics/aqs/deploy@6fafc63]: Update AQS new-pages metric
15:06 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 with low load (duration: 00m 55s)
14:36 mutante: terbium - removing accountcheck cron job (duplicate mail from the same thing now on mwmaint1001)
14:23 jynus: stop and reimage db1094
14:18 elukey: execute echo 'https://wikimania.wikimedia.org' | mwscript purgeList.php on mwmain1001
13:27 ema: libvmod-re2 1.3.1-2 uploaded to stretch-wikimedia T200445
13:15 ema: libvmod-netmapper 1.7-2 uploaded to stretch-wikimedia T200445
13:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for reimage (duration: 00m 54s)
11:33 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813 (duration: 02m 02s)
11:31 mobrovac@deploy1001: Started deploy [eventstreams/deploy@941e3cf]: Make the main processing loop async - T199813
09:37 ema: vhtcpd 0.1.1-2 uploaded to stretch-wikimedia T200445
09:00 godog: adjust aggregation to 'sum' for MediaWiki.edit sum metrics - T199968
08:36 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist s3 populateChangeTagDef.php --sleep 2 (T193873 )
05:59 akosiaris: reboot webperf1002, webperf2002 for new disk to appear T199853
05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 (duration: 00m 55s)
04:59 marostegui: Deploy schema change on db1094 T144010 T51190 T199368
04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 (duration: 00m 57s)
01:19 aaron@deploy1001: Synchronized php-1.32.0-wmf.14/includes/libs/objectcache/BagOStuff.php: f208a431f912
23:04 XioNoX: depool ulsfo, outages on both physical links to the site
21:57 ejegg: updated payments-wiki from 6390e50abd 5e01f50983
21:33 tgr@deploy1001: Synchronized php-1.32.0-wmf.14/includes/Title.php: SWAT T200456 : Handle $title === null in Title::newFromText (duration: 00m 57s)
21:22 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
21:21 tgr@deploy1001: Synchronized wmf-config/LabsServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 54s)
21:19 tgr@deploy1001: Synchronized wmf-config/ProductionServices.php: SWAT: T200346 Update custom domain for thumbnail rendering (duration: 00m 55s)
21:12 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 54s)
21:11 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: T190015 gerrit 421122 + 421123 prepare for interface-admin group (duration: 00m 55s)
20:55 tgr@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Do not set deprecated value for $wgExternalDiffEngine (duration: 00m 54s)
20:39 addshore@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Wikibase/repo/includes/Store/Sql/SqlChangeDispatchCoordinator.php: Use getClientLockName value for releaseClientLock when dispatching T200420 (duration: 00m 57s)
20:12 ebernhardson: set merge.policy.reclaim_deletes_weight back to 2.0 for wikidatawiki_content
20:11 thcipriani: gerrit upgrade to 2.15.3 complete
20:06 thcipriani: restart gerrit for notedb config update
20:03 thcipriani: running puppet on cobalt for gerrit notedb changes
19:59 mobrovac@deploy1001: Finished deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461 (duration: 00m 33s)
19:58 mobrovac@deploy1001: Started deploy [proton/deploy@883cacd]: Use a more secure MW API template - T198461
19:40 thcipriani: restarting gerrit
19:39 thcipriani: running puppet on cobalt for gerrit.config update
19:38 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813 (duration: 02m 13s)
19:36 mobrovac@deploy1001: Started deploy [eventstreams/deploy@b1f577d]: Provide better stream and error handling and stop using compression - T199813
19:31 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3 (duration: 00m 08s)
19:31 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: upgrade to gerrit 2.15.3
19:22 thcipriani@deploy1001: Finished deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica (duration: 00m 10s)
19:22 thcipriani@deploy1001: Started deploy [gerrit/gerrit@b69b468]: deploy gerrit 2.15.3 to cold-replica
18:19 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "all wikis to 1.32.0-wmf.14"
18:17 ebernhardson: set merge.policy.reclaim_deletes_weight=12.0 for wikidatawiki_content in eqiad to attempt to reduce the deleted documents count
18:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.14
18:08 arlolra: Updated Parsoid to cdf8ace T194806 , T110004 , T156099 , T188478 , T194083 )
17:59 arlolra@deploy1001: Finished deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace
17:58 XioNoX: moving row C vrrp master back to cr2 - T187962
17:54 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491 (duration: 06m 16s)
17:47 arlolra@deploy1001: Started deploy [parsoid/deploy@1dce68c]: Updating Parsoid to cdf8ace
17:47 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test, take #2 - T199491
17:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491 (duration: 19m 36s)
17:45 mutante: created new archiva user tyler for Tyler Cipriani and gave him role of global repo manager (for gerrit uploads instead of using archiva-deploy user which can cause password sync issues, so personalized users )
17:27 mobrovac@deploy1001: Started deploy [restbase/deploy@5ce3a68]: Fix the failing mobile-html test - T199491
17:01 XioNoX: moving row C vrrp master to cr1 - T187962
16:08 gehel: rolling restart of elasticsearch / cirrus / eqiad completed - T156137
15:47 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 01m 04s)
15:46 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
15:46 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 14m 56s)
15:31 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
15:30 mobrovac@deploy1001: Finished deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491 (duration: 37m 37s)
15:27 addshore@deploy1001: Synchronized wmf-config: Remove unused wikibaseDispatchRedisLockManager logging T200420 (duration: 00m 56s)
15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add LockManager logging T200420 (duration: 00m 55s)
14:52 mobrovac@deploy1001: Started deploy [restbase/deploy@5c9bf79]: Expose the mobile-html end point - T199491
14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add wikibaseDispatchRedisLockManager to wmgMonologChannels T200420 (duration: 00m 54s)
14:35 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 56s)
14:34 addshore@deploy1001: sync-file aborted: Add a logger to wikibaseDispatchRedisLockManager T200420 (duration: 00m 02s)
14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf1002 T199853
14:24 akosiaris: sudo gnt-instance modify --disk add:size=150G webperf2002 T199853
13:44 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use new wikibase dispatch lock manager on wikidatawiki T200420 T178652 (duration: 00m 55s)
12:44 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: T200420 - Wikidata, dispatch, select 20 instead of 15 wikis (duration: 00m 55s)
12:38 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .13 T200420
11:57 zeljkof: EU SWAT finished
11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use HD logos in hewikiquote (T200296) (duration: 00m 52s)
11:45 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Upload HD logos for hewikiquote (T200296) (duration: 00m 54s)
11:37 zfilipin@deploy1001: Synchronized tests/InitialiseSettingsTest.php: SWAT: Test spaces in ExtraNamespaces (T199162) (duration: 00m 55s)
11:30 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create TemplateEditor group on enwikivoyage (T198056) (duration: 00m 55s)
11:20 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix unexpected space in pswikis ExtraNamespaces definition (T199480) (duration: 00m 55s)
11:14 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Dont assign sysop-level privileges to bureaucrats explicitly (T197095) (duration: 00m 56s)
11:02 Amir1: ladsgroup@mwmaint1001:~$ foreachwikiindblist wikitech populateChangeTagDef.php (T193873 )
11:01 Amir1: ladsgroup@mwmaint1001:~$ mwscript sql.php --wiki=labtestwiki /srv/mediawiki-staging/php-1.32.0-wmf.14/maintenance/archives/patch-change_tag_def.sql
10:46 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 54s)
10:45 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
10:33 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES on test2wiki (T200412) (duration: 00m 55s)
10:26 Amir1: ladsgroup@mwmaint1001:~$ mwscript extensions/WikimediaMaintenance/createExtensionTables.php --wiki=test2wiki ores (T200412 )
10:24 ema: cache-text: upgrade to varnish 5.1.3-1wm9 and apply alternate domains patch T164609
10:08 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: Revert "group1 wikis to 1.32.0-wmf.14"
10:07 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s7 populateChangeTagDef.php --sleep 2 (T193873 )
09:50 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 53s)
09:49 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
09:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 (duration: 00m 55s)
09:40 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 fully (duration: 00m 55s)
09:27 gehel: rolling restart of elasticsearch / cirrus / eqiad to disable G1 - T156137
09:22 marostegui: Deploy schema change on db1079 with replication, this will generate lag on labsdb:s7 T144010 T51190 T199368
09:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 (duration: 00m 55s)
09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 (duration: 00m 55s)
09:00 jynus: restarting hhvm at mw1226, mw1276, mw1281
08:54 jynus: restarting hhvm at mw1230
08:26 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2 (duration: 29m 02s)
08:20 marostegui: Deploy schema change on db1090:3317 T144010 T51190 T199368
08:18 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Set readOnlyReason to false everywhere for JobQueueEventBus - T199594 (duration: 00m 55s)
08:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 with low load after reimage (duration: 00m 55s)
07:57 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts - try 2
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 (duration: 00m 54s)
07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 (duration: 00m 54s)
07:24 jynus: stop and reimage es1015
07:19 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 (duration: 00m 55s)
06:47 marostegui: Deploy schema change on db1086 T144010 T51190 T199368
06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 55s)
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 (duration: 00m 55s)
05:02 marostegui: Deploy schema change on db1101:3317 T144010 T51190 T199368
05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 (duration: 01m 05s)
04:13 kartik@deploy1001: Finished deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283 ) (duration: 05m 12s)
04:08 kartik@deploy1001: Started deploy [cxserver/deploy@3c99775]: Update cxserver to e98c81vb (T200283 )
03:16 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 26 03:16:04 UTC 2018 (duration 10m 24s)
03:06 twentyafterfour: stopped phabricator rebuild-identities migration as it's suspected of causing database connection exhaustion and replag
03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.14) (duration: 15m 13s)
02:50 Krinkle: phabricator.wikimedia.org is spewing HTTP 500 ("Can Not Connect to MySQL")
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 40s)
01:42 mutante: phab1002 - starting rsync of repo data from phab1001 in a screen session after gerrit:447949 T190568
01:32 mutante: phab1001 - rm /usr/local/sbin/sync-srv-repos that has a reference to non-existing server iridium.eqiad.wmnet (formerly phab) (T190568 )
00:16 twentyafterfour: phabricator update complete with only momentary downtime
00:04 twentyafterfour: deploying phabricator release/2018-07-25/1, scheduled downtime for phab1001 in icinga
00:01 mutante: temp. disable puppet on netmon1002,netmon2001 before applying puppet ferm change out of abundance of caution. applied gerrit:439554 without issues on netmon1003
23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/dm/nodes/ve.dm.MWInlineImageNode.js: SWAT: ve.dm.MWInlineImageNode: Fix undefined data-mw in toDomElements output T198941 T200214 (duration: 00m 57s)
23:05 mutante: added dchen to LDAP group "wmf" (was already in admins as shell user, so didn't have to be added in puppet repo) (T200366 )
20:23 bblack: pooling cp5006 - T187157
20:05 krinkle@deploy1001: Synchronized wmf-config/: Remove wgUploadThumbnailRenderHttpCustomDomain override - T200346 (duration: 00m 57s)
18:59 bstorm_: Added new disk shelf to labstore1007
17:53 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
17:45 krinkle@deploy1001: Synchronized php-1.32.0-wmf.14/extensions/Cite/includes/Cite.php: (no justification provided) (duration: 00m 55s)
17:07 ejegg: Updated SmashPig settings: extended Ingenico Connect API timeout to 12 seconds
16:49 Amir1: morning SWAT is done
16:49 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable reading from the new backend for Special:Tags in several large wikis (T199334) (duration: 00m 56s)
16:40 akosiaris: remove ORES abuser blocking T200338 , let's reevaluate
16:02 joal@deploy1001: Finished deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts (duration: 17m 51s)
15:49 gehel: elasticsearch cluster restart on codfw completed - T156137
15:44 joal@deploy1001: Started deploy [analytics/refinery@9390b63]: Regular weekly deploy of Analytics-Hadoop scripts
15:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 (duration: 00m 55s)
15:31 anomie: running populateContentTables.php on testwiki for T183488
15:26 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 55s)
15:21 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Set wgMultiContentRevisionSchemaMigrationStage to write-both read-old on testwiki (T197817 ) (duration: 00m 55s)
14:45 marostegui: Deploy schema change on db1098:3317 T144010 T51190 T199368
14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 (duration: 00m 54s)
14:39 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: (no justification provided)
14:29 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 fully (duration: 00m 55s)
14:00 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.14 (duration: 00m 55s)
13:59 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.14
13:54 jynus: running compare.py on all es3 databases
13:48 ema: text-eqiad: test alternate domains patch with varnish 5.1.3-1wm9 T164609
13:44 Amir1: restarting ores celery workers on codfw
13:34 ema: repool cp1067 w/ alternate domains patch and varnish 5.1.3-1wm9 T164609
13:27 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.14
13:26 ema: depool cp1067 and test alternate domains patch with varnish 5.1.3-1wm9 T164609
13:00 gehel: resetting postgres data on maps1002 after failing replication - T200228
12:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s5 populateChangeTagDef.php --sleep 2 (T193873 )
12:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache (duration: 59m 27s)
11:20 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
11:16 dcausse: EU SWAT done
11:14 dcausse@deploy1001: Synchronized ./wmf-config/CirrusSearch-common.php: [cirrus] allow term_freq and remove deprecated settings (duration: 00m 48s)
11:01 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.10 [keeping static files] (duration: 05m 13s)
10:21 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1014 with low load after maintenance (duration: 00m 47s)
10:07 marostegui: Deploy schema change on db2040 (s7 codfw master) with replication, this will generate lag on s7 codfw T144010 T51190 T199368
10:05 ema: upgrade varnish to 5.1.3-1wm9 on text-eqiad T164609
09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 47s)
09:53 godog: bounce grafana on krypton
09:31 ema: upload varnish 5.1.3-1wm9 to apt.w.o (fixing POST requests w/ separate VCL) T164609
09:20 gehel: restarting elasticsearch cluster restart on codfw - T156137
08:41 jynus: stop es1014 for reimage
08:14 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb hosts for s4 T144010 T51190 T199368
08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 (duration: 00m 47s)
08:10 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s4 populateChangeTagDef.php --sleep 2 (T193873 )
08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 db1091 (duration: 00m 47s)
08:02 gehel: pausing elasticsearch cluster restart on codfw - T156137
07:58 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1014 (duration: 00m 48s)
07:35 gehel: rolling restart of elasticsearch / cirrus / codfw to disable G1 - T156137
07:12 marostegui: Deploy schema change on db1091 T144010 T51190 T199368
06:53 gehel: resetting postgres data on maps1004 after failing replication - T200228
06:38 marostegui: Stop replication in sync on db1091 and db1097:3314
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 db1091 (duration: 00m 48s)
06:33 jynus: finished es1014 -> es1017 switch T197073
06:27 jynus: enabling semi-sync master on es1017, disabling it as client
06:21 jynus: deploy es3-master dns change
06:02 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Switchover es3 master eqiad from es1014 to es1017 (duration: 00m 24s)
06:01 jynus: switchover es3 eqiad master from es1014 to es1017
04:35 tstarling@deploy1001: Synchronized php-1.32.0-wmf.13/includes/api/ApiMain.php: record all API requests in statsd (duration: 00m 49s)
02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 25 02:43:05 UTC 2018 (duration 10m 19s)
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 28s)
21:59 XioNoX: re-pooling eqsin
21:15 XioNoX: re1 is master routing engine on cr1-eqsin, triggering a re switch
21:10 XioNoX: starting to see recoveries from cr1-eqsin upgrade
21:06 XioNoX: Install done, cr1-eqsin re-rebooting
21:00 XioNoX: restarting cr1-eqsin for software upgrade
20:32 XioNoX: depooling eqsin for cr1-eqsin software upgrade
19:09 gehel: resetting postgres data on maps1003 after failing replication - T200228
18:34 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813 (duration: 02m 18s)
18:32 mobrovac@deploy1001: Started deploy [eventstreams/deploy@690fdad]: Wait for the client to consume the meesage being sent before consuming the next one - T199813
17:40 ema: re-enable puppet on all cache nodes with alternate domains disabled T164609
17:33 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.13
17:18 thcipriani: train window running long, services deploy delayed
17:18 ema: restart varnish-fe on cp1068 to clear "child restarted" alert T164609
17:17 elukey: restart eventstreams on scb2* nodes (hopefully last time before deploying the fix) to avoid mem leaks issues during the EU night
17:06 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.14/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 01s)
16:58 jynus: finishing test on es3 hosts T199224
16:42 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/includes/page/PageArchive.php: PageArchive: Pass correct overrides to newRevisionFromArchiveRow() (T200072) (duration: 01m 03s)
16:07 jynus: test switchover from es2018 to es2017
16:02 dcausse: T156137 : unbanning elastic1031
15:59 dcausse: T156137 : restarting elasticsearch on elastic1031 to disable G1GC
15:55 jynus: test switchover from es2017 to es2018
15:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 (duration: 01m 02s)
15:38 jynus: stopping puppet on es2017, es2018; changing mysql configuration for production testing
15:29 gehel: restart postgres on maps1001 - T200228
14:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 01m 02s)
14:47 dcausse: T156137 : banning elastic1031 due to high load (same "getEntryAfterMiss" symptoms)
14:09 marostegui: Deploy schema change on db1103:3314 T144010 T51190 T199368
13:45 ema: apply alternate domains patch to text-eqiad T164609
13:43 marostegui: Deploy schema change on db1084 T144010 T51190 T199368
13:42 marostegui: Stop replication in sync db1084 and db1103:3314
13:40 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
13:38 marostegui: Stop replication in sync db1081 and db1103:3314
13:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 01m 59s)
13:07 ema: repool cp1067 with alternate domains support T164609
12:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 (duration: 00m 55s)
12:12 gehel: vacuum full of postgres on maps1001 to try to reclaim space - T200228
12:07 ema: depool cp1067 to test alternate domains patch T164609
11:58 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4179557944" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 02m 50s)
11:55 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
11:45 zfilipin@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2212739269" --threads=30 --lang en --quiet' returned non-zero exit status 1 (duration: 03m 42s)
11:42 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.14 and rebuild l10n cache
11:35 ema: disable puppet on cp-text hosts to merge alternate domains patch T164609
11:21 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s1 populateChangeTagDef.php (T193873 )
11:19 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s2 populateChangeTagDef.php --sleep 2 (T193873 )
11:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 (duration: 00m 55s)
11:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 (duration: 01m 06s)
10:44 marostegui: Stop replication in sync on db1084 and db1097:3314
10:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 55s)
09:17 marostegui: Deploy schema change on db1097:3314 T144010 T51190 T199368
09:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 (duration: 00m 54s)
08:38 ema: restart varnish-fe on cache_text instances with cold, labeled VCL T200207
08:21 elukey: rolling restart of kafka jumbo/main-(eqiad|codfw) clusters to pick up the new max open files limit (infinity -> 128k)
06:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 (duration: 00m 55s)
06:32 marostegui: Deploy schema change on dbstore1002:s4 T144010 T51190 T199368
05:33 kartik@deploy1001: Finished deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 T198941 ) (duration: 04m 01s)
05:29 kartik@deploy1001: Started deploy [cxserver/deploy@d378d27]: Update cxserver to d3c9d15 T198941 )
05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 (duration: 00m 54s)
05:05 marostegui: Deploy schema change on db1081 T144010 T51190 T199368
04:54 marostegui: Stop replication in sync on db1081 and db1121
04:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084, db1121 (duration: 00m 56s)
04:44 marostegui: Deploy schema change on db1066 (s2 primary master) T144010 T51190 T199368
03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 24 03:03:08 UTC 2018 (duration 10m 22s)
02:52 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 19s)
02:23 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 44s)
22:53 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: A ggodnight restart - T199813
22:27 thcipriani@deploy1001: Finished scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941 (duration: 33m 21s)
21:53 thcipriani@deploy1001: Started scap: Revert "Accept BCP 47 codes as aliases for nonstandard variants" T199941 Revert "Ensure LanguageCode::bcp47() returns a valid BCP 47 language code" T199941
lunch: T156137 : restarting elastic1035 to disable G1GC
lunch: T156137 : restarting elastic1027 to disable G1GC
20:27 arlolra: Updated Parsoid to a1e851c T199808 , T194083 )
20:22 arlolra@deploy1001: Finished deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c
20:13 arlolra@deploy1001: Started deploy [parsoid/deploy@80384a5]: Updating Parsoid to a1e851c
20:09 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5
20:03 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@565b41a]: Update mobileapps to 254cef5
19:49 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (duration: 11m 35s)
19:38 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI
19:37 gehel@deploy1001: Finished deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 22s)
19:36 gehel@deploy1001: Started deploy [wdqs/wdqs@690bf52]: new version of wdqs GUI (wdqs1009 only)
17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 25s)
17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
17:53 ejegg: updated payments-wiki from d8d8293b87 6390e50abd
17:47 gehel@deploy1001: Finished deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 03s)
17:47 gehel@deploy1001: Started deploy [wdqs/wdqs@67fadb7]: new version of wdqs GUI (wdqs1009 only)
17:29 XenoRyet: updated payments-wiki from 63372154d0 d8d8293b87
17:16 gehel: canceling deployment of new WDQS GUI to all servers, looks like there is a JS issue. Will reschedule once the issue is fixed.
17:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@d7e8292]: new version of wdqs GUI (wdqs1009 only) (duration: 00m 30s)
15:25 gehel: decrease gc_grace_seconds to 4 days on cassandra / maps
15:02 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50) (duration: 01m 38s)
15:00 mobrovac@deploy1001: Started deploy [changeprop/deploy@5cfdf26]: Increase concurrency back to normal levels (from 20 to 50)
14:57 mobrovac@deploy1001: Started restart [eventstreams/deploy@5ed03e0]: Fresh start of the service after alleviating 404s
14:53 elukey: delete empty/not-used/wrongly-created topics in Kafka main-eqiad - T199510
14:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081, db1097:3314 (duration: 00m 53s)
14:17 marostegui: Stop replication in sync on db1081 and db2051
14:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813 (duration: 02m 29s)
13:57 mobrovac@deploy1001: Started deploy [eventstreams/deploy@5ed03e0]: Update node-rdkafka to v2.3.4 - T199813
13:53 marostegui: Stop replication in sync on db1081 and db1097:3317
13:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081, db1097:3314 (duration: 00m 54s)
13:44 ema: !log trafficserver 7.1.3+ds-4wm1 uploaded to stretch-wikimedia T200178
13:36 marostegui: Deploy schema change on s4 codfw master (db2051), this will generate lag on s4 codfw T144010 T51190 T199368
13:23 marostegui: Deploy schema change on labswiki and labstestwiki T144010 T51190 T199368
13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074, db1105:3312 (duration: 00m 55s)
11:34 zeljkof: EU SWAT finished
11:32 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Exempt Template and Module namespace on zhwiktionary (T187783) (duration: 00m 55s)
11:14 zfilipin@deploy1001: Synchronized static/images/project-logos/: SWAT: Change zhwikiquote logo (T199863) (duration: 00m 55s)
10:10 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 54s)
10:09 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 55s)
09:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074, db1105:3312 (duration: 00m 53s)
09:42 marostegui: Stop replication in sync on db1105:3312 db1074
09:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122, db1103:3312 (duration: 00m 54s)
09:00 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s8 populateChangeTagDef.php (T193873 )
08:47 marostegui: Stop replication in sync on db1103:3312 db2035
08:42 godog: enable deprecation page for status.w.o - T199816
08:23 marostegui: Stop replication in sync on db1122 and db1103:3312
08:21 marostegui: Apply schema change T192926 to labswiki and labstestwiki T200140
08:15 marostegui: Apply schema change T160415 to labswiki and labstestwiki T200140
08:11 marostegui: Apply schema change T191316 to labswiki and labstestwiki T200140
08:07 marostegui: Apply schema change T191519 to labswiki and labstestwiki T200140
08:05 marostegui: Apply schema change T190148 to labswiki and labstestwiki T200140
08:04 marostegui: Apply schema change T197891 to labswiki and labstestwiki T200140
07:54 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist s6 populateChangeTagDef.php (T193873 )
07:54 marostegui: Stop replication in sync on db1122 and db2035
07:33 marostegui: Deploy schema change on db1122 T144010 T51190 T199368
07:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122, db1103:3312 (duration: 00m 55s)
06:37 jynus: stop db1102 to clone to db1118
04:42 marostegui: Deploy schema change on db1061 (s6 primary master) T144010 T51190 T199368
03:08 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 23 03:08:23 UTC 2018 (duration 10m 37s)
02:57 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 13m 51s)
02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 09m 42s)
16:15 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.159
12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.76.150
12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 197.101.68.116
12:08 addshore: addshore@deploy1001:/home/legoktm$ mwscript /home/legoktm/resetAuthenticationThrottle.php --wiki=metawiki --signup --ip 196.208.95.30
12:03 addshore@deploy1001: Synchronized wmf-config/throttle.php: T198288 More Wikimania throttle rules (duration: 01m 18s)
22:29 Reedy: Added ct_tag_id to labswiki and labtestwiki T200139
17:35 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
21:50 bawolff: deployed patch T200104
17:16 XioNoX: enable bgp on eqdfw-knams link (GTT)
17:08 XioNoX: enable ospf on eqdfw-knams link (GTT)
15:33 elukey: rolling restart of eventstreams on scb2* nodes to reduce the memory pressure before the weekend (still waiting for a permanent fix)
15:19 elukey: powercycle ms-be1016 - RAID errors, no ssh available, I/O errors in com2 console
14:15 reedy@deploy1001: rebuilt and synchronized wikiversions files: wikidatawiki back to .12 T199983
14:11 Reedy: T199983 wikidata wiki abck to .12 on mwdebug1001
13:33 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085, db1113:3316 (duration: 00m 55s)
13:13 marostegui: Stop replication in sync db1085 and db1113:3316
13:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316, depool db1113:3316 (duration: 00m 54s)
13:06 legoktm@deploy1001: Synchronized wmf-config/throttle.php: Update Wikimania IP address - T198288 (duration: 00m 54s)
12:53 marostegui: Stop replication in sync db1085 and db1096:3318
12:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316, depool db1098:3316 (duration: 00m 54s)
12:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 and db1096:3316 (duration: 00m 56s)
12:32 marostegui: Stop replication in sync db1085 and db1096:3316
12:31 marostegui@deploy1001: sync-file aborted: Depool db1085 and db1096:33156 (duration: 00m 01s)
12:00 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813 (duration: 02m 11s)
11:58 mobrovac@deploy1001: Started deploy [eventstreams/deploy@01fac88]: Lower librdkafka settings related to fetching messages - T199813
10:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312, db1105:3312 (duration: 00m 53s)
10:30 marostegui: Stop replication in sync on db1090:3312 and db1105:3312 - T200061
10:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312, depool db1105:3312 (duration: 00m 54s)
10:17 marostegui: Stop replication in sync on db1090:3312 and db2035 - T200061
10:14 marostegui: Stop replication in sync on db1090:3312 and db1103:3312 - T200061
10:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 and db1103:3312 (duration: 00m 55s)
09:46 marostegui: Stop replication on db2063 to fix db2095 - T200061
08:50 marostegui: Stop replication on s7 codfw master to check arwiki.change_tag - T200061
08:47 Amir1: stopping populateChangeTagDef.php (T200061 )
08:41 foks: reset email for User:Nomden
08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 55s)
07:46 marostegui: Stop replicaiton on s8 codfw master - T200061
07:43 jynus: stopping db1095 mariadb and cloning it to db1102
07:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099, es1019 fully (duration: 00m 54s)
07:11 marostegui: Stop replication in sync on db1074 and db1125:3312
06:13 marostegui: Deploy schema change on db1074 with replication, this will generate lag on labsdb:s2 T144010 T51190 T199368
06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 53s)
06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 54s)
05:49 marostegui: Deploy schema change on db1076 T144010 T51190 T199368
05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
05:08 marostegui: Start to remove some unused files on db1067 - T200039
01:38 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/CentralAuth/includes/CentralAuthUser.php: T170971 (duration: 00m 55s)
00:31 krinkle@deploy1001: Synchronized php-1.32.0-wmf.13/includes/filerepo/: I58706b5610 I40f6ad2a3d T200026 (duration: 00m 56s)
00:30 ejegg: updated CiviCRM from 22a8c47668 d76e3997e6
00:12 bblack: disabled puppet temporary on cp* and kafka-jumbo* for ipsec unconfiguration
23:43 ejegg: updated CiviCRM from 08680be68e 22a8c47668
23:31 ejegg: updated CiviCRM from 2bb0d0f9d0 08680be68e
23:11 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable Special:Block Feedback Request" (duration: 00m 55s)
22:51 XioNoX: disable ospf on eqdfw-knams link (GTT)
22:18 XioNoX: enable ospf on eqdfw-knams link (GTT)
21:52 hashar: CI Jenkins has been upgraded successfully!
21:21 hashar: restarting CI on contint1001 and actually upgrading it
21:19 hashar: starting CI jenkins on contint1001
21:16 hashar: stopping the CI jenkins on contint1001
20:20 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099 with low load (duration: 00m 54s)
19:25 ariel@deploy1001: Finished deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps (duration: 00m 03s)
19:25 ariel@deploy1001: Started deploy [dumps/dumps@f0c4a70]: handle empty tables properly; script to show runtimes of dump steps
19:21 ejegg: updated CiviCRM from 6f311f48d9 2bb0d0f9d0
19:18 XioNoX: setting mtu 9192 to all codfw interface ranges
19:04 elukey: roll restart eventstreams on scb2* hosts to prevent OOM issues over the EU night - T199813
19:00 Amir1: Morning SWAT is done
18:57 ladsgroup@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Add abusefilter-modify-restricted right to sysops on itwiktionary (T199783) (duration: 00m 53s)
18:50 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
18:27 ladsgroup@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/WikibaseQualityConstraints: SWAT: Report SPARQL errors as TODO, not VIOLATION (T199788) (duration: 00m 56s)
18:18 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist all populateChangeTagDef.php (T193873 ) It will take a couple of days
18:14 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set to write the new change tag backend everywhere, enable reading in frwiki (T194165) (duration: 00m 55s)
16:28 jynus: stop, clone away and upgrade db1099 two mysql instances
16:16 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1099, including from s8 (duration: 00m 55s)
16:00 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
15:53 hashar: CI: npmjs is slow. Bumping Quibble jobs timeout from 30 to 45 minutes | https://gerrit.wikimedia.org/r/#/c/446863 | T198348
15:50 jynus: droping undocumented grants on labsdb1005 using replication T199186
15:21 volans: reimaging mw2226 to test py3 reimage with new cumin, conftool and apache test from neodymium - T187773
15:12 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2225.codfw.wmnet
15:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 54s)
15:00 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099 (duration: 00m 54s)
14:59 moritzm: rebooting multatuli for kernel test
14:30 marostegui: Deploy schema change on db1105:3312 T144010 T51190 T199368
14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 54s)
14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 54s)
14:17 elukey: roll restart kafka on kafka-jumbo* and kafka main-codfw (kafka2*) to pick up new Xmx/Xms settings (1g -> 2g)
14:15 volans: upgrading cumin/debdeploy/reimage and other tools on neodymium - T187773
13:57 elukey: restart kafka on kafka-jumbo1001 to raise Xmx/Xms jvm settings (1g -> 2g)
13:43 marostegui: Deploy schema change on db1103:3312 T144010 T51190 T199368
13:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 54s)
13:38 volans: reimaging mw2225 to test py3 reimage with new cumin, conftool and apache test - T187773
13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1103:3314 (duration: 00m 53s)
13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
13:22 moritzm: installing python-pysaml2 security updates
13:05 moritzm: installing ffmpeg security updates
12:48 moritzm: installing xerces-c security updates
12:38 moritzm: uploaded debdeploy 0.0.99.5 to apt.wikimedia.org
11:49 zeljkof: EU SWAT finished
11:46 zfilipin@deploy1001: Synchronized dblists/categories-rdf.dblist: SWAT: Remove labs wikis from the categories-rdf list, dont need them (T198356) (duration: 00m 55s)
11:44 moritzm: installing reportbug update from stretch point release to test py3-based debdeploy
11:40 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Special:Block Feedback Request (T199919) (duration: 00m 55s)
11:33 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Structured Discussions beta feature on orwiki (T199971 ) (duration: 00m 55s)
11:22 volans: testing reimage of californium (spare host) with the upgraded cumin
11:20 chasemp: maintain-views --all-databases --replace-all --table ores_classification --debug (labsdb)
11:19 chasemp: maintain-views --all-databases --replace-all --table ores_model --debug (labsdb)
11:18 tgr@deploy1001: Synchronized wmf-config/InitialiseSettings.php: enable TemplateStyles on enwiki, frwiki, zhwiki T197603 T191452 T189022 (duration: 00m 55s)
11:00 volans: upgrading cumin on sarin - T187773
10:52 legoktm: made myself a crat on test2wiki
10:06 moritzm: installing file/libmagic security updates
09:55 moritzm: installing check-postgres update from stretch 9.5 point release
09:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.13/includes/libs/objectcache/MultiWriteBagOStuff.php: 2cee0ab
09:35 volans: cumin upgrade on labpuppetmaster* hosts completed - T187773
09:33 moritzm: installing openssl security updates
09:33 volans: removed leftover cumin installation from labtestpuppetmaster2001 - T187773
09:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
09:27 godog: run xfs_repair on filesystems reporting negative space available on ms-be1040 - T199198
09:24 moritzm: installing tomcat7 security updates
09:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1019 with low load (duration: 00m 54s)
09:21 moritzm: installing ant security updates
09:14 moritzm: uploaded jenkins 2.121.2 to apt.wikimedia.org (T199448 )
08:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
08:39 volans: start upgrade of cumin to 3.0.1-1 on labpuppetmaster* - T187773
08:32 marostegui: Deploy schema change on dbstore1002:s2 T144010 T51190 T199368
08:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
08:08 elukey: restart eventstreams on scb2* hosts to pick up new Kafka settings (pointing it to main-codfw) - T199813
08:04 kartik@deploy1001: Finished deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e T199380 , T199885 ) (duration: 03m 58s)
08:00 kartik@deploy1001: Started deploy [cxserver/deploy@fe0d521]: Update cxserver to 92e3d2e T199380 , T199885 )
07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3314 after MySQL upgrade (duration: 00m 54s)
07:55 marostegui: Deploy schema change on s2 codfw masters (db2035) this will generate lag on s2 codfw T144010 T51190 T199368
07:50 jynus: stop es1019 and reimage it
07:48 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1019 (duration: 00m 54s)
07:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
07:26 moritzm: removing cdrom and sr_mod (related module) kernel modules across the fleet (following the blacklisting of the kernel module via puppet)
06:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 54s)
05:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1103:3312 after MySQL upgrade (duration: 00m 55s)
05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Slowly repool db1103:3312 after MySQL upgrade (duration: 00m 54s)
05:14 marostegui: Stop MySQL on db1103 to upgrade MySQL
05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for MySQL upgrade (duration: 00m 56s)
04:12 kart_: Finished ContentTranslation draft purge script on mwmaint1001 (T199658 )
04:05 kart_: Starting ContentTranslation draft purge script on mwmaint1001, dry run first and then --really
03:57 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I978f3eda8
03:20 ejegg: updated CiviCRM from b21d783e7f 6f311f48d9
03:09 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 19 03:09:19 UTC 2018 (duration 10m 35s)
02:58 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 15m 05s)
02:24 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 10m 41s)
23:35 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/includes/logging/LogEventsList.php: SWAT: LogEventsList: Use GET in HTMLForm T199856 (duration: 00m 54s)
23:33 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWMediaDialog.js: SWAT: ve.ui.MWMediaDialog: Fix confusion between #getSetupProcess and #getReadyProcess T185944 T199841 (duration: 00m 56s)
22:16 ejegg: updated SmashPig payments listener from 2b4186715e 2736e41045
19:24 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Unset wgVipsThumbnailHost T199937 (duration: 00m 55s)
16:28 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable new backend for Special:Tags on fawikisource (T199334 ) (duration: 00m 54s)
16:14 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on testwiki (T199913 ) (duration: 00m 55s)
15:25 volans: installing conftool 1.0.1 on jessie and stretch
14:50 mobrovac@deploy1001: Finished deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813 (duration: 02m 28s)
14:48 mobrovac@deploy1001: Started deploy [eventstreams/deploy@09f0efe]: Set the maximum rdkafka receive buffer size to 64MB - T199813
14:47 marostegui: Partition commonswiki.logging table on db1103:3314 - T199790
14:46 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
14:43 volans: uploaded conftool (with python-conftool and python3-conftool) version 1.0.1 for jessie and stretch to apt.w.o
14:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table - T199790 (duration: 00m 52s)
14:37 moritzm: rebooting poolcounter2001 for some tests
14:22 moritzm: rebooting vega for some tests
13:59 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 54s)
13:53 moritzm: installing ruby2.1 security updates for jessie
13:17 marostegui: Deploy schema change on db1052 - T187089
13:13 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.13 (duration: 00m 53s)
13:13 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.13
12:58 marostegui: Stop MySQL on db1052 to upgrade socket and MySQL
12:57 elukey: restart eventstreams on scb200[5,6] as precautionary after mem consumption too high
12:57 elukey: restart eventstreams on scb2003 as precautionary after mem consumption too high
12:56 elukey: restart eventstreams on scb2001 as precautionary after mem consumption too high (still investigating a fix)
12:54 dcausse: T156137 : unbanning elastic1030
12:52 dcausse: T156137 : restarting elastic1030 to disable G1GC
12:51 moritzm: installing PHP5 security updates (remaining hosts which only use -cli)
12:48 moritzm: installing PHP security updates on tungsten
12:36 moritzm: installing PHP security updates on icinga.wikimedia.org
12:24 elukey: manually added queued.max.messages.kbytes: 65535 to eventstreams on scb2002 as test for T199813
12:16 moritzm: installing PHP security updates on yubiauth servers
12:02 moritzm: installing PHP security update on bohrium
11:35 moritzm: installing imagemagick security updates
11:34 zeljkof: EU SWAT finished
11:32 zfilipin@deploy1001: Synchronized wmf-config/abusefilter.php: SWAT: Enable AbuseFilter block option on itwiktionary (T199783) (duration: 00m 54s)
11:25 moritzm: installing policykit security updates on trusty
11:20 Amir1: making ores tables on euwiki
11:15 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Publisher namespace in Bengali Wikisource (T199028) (duration: 00m 54s)
11:05 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ores extension wp10 storage in Basque Wikipedia (T198358) (duration: 00m 54s)
10:30 moritzm: ran puppet clean/deactivate for lawrencium, long-standing source of errors in package deployments (T191360 )
10:24 moritzm: installing cups/libcups security updates for jessie
10:10 dcausse: T156137 : banning elastic1030 from the cluster (high load)
10:02 dcausse: clearing internal cache on elastic1030
09:58 hashar: contint1001 dropping unused docker containers
09:36 moritzm: draining restbase1007 for eventual reboot for tests with SSBD microcode
08:51 godog: run xfs_repair on filesystems reporting negative space available on ms-be1043 - T199198
08:50 moritzm: reboot ms-be1040 for tests with SSBD microcode
08:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 53s)
08:38 twentyafterfour: restart apache on phab1001
08:37 chasemp: deploy antivandalism stuff to phab https://gerrit.wikimedia.org/r/c/operations/puppet/+/445329 (needs herald to fully enable)
08:31 elukey: drain + reboot analytics1030 for kernel updates
08:29 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb hosts for s6 T144010 T51190 T199368
08:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 01m 01s)
08:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 53s)
08:06 marostegui: Deploy schema change on db1088 T144010 T51190 T199368
08:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 53s)
07:41 jforrester@deploy1001: Synchronized wmf-config/missing.php: Fix typo in missing.php Ia3aae912d9m (duration: 00m 54s)
07:37 moritzm: upgrading tor on radium to 0.3.3.9
07:25 marostegui: Deploy schema change on db1052 - https://phabricator.wikimedia.org/T191316 https://phabricator.wikimedia.org/T192926 https://phabricator.wikimedia.org/T89737 https://phabricator.wikimedia.org/T195193
07:22 moritzm: updated tor on apt.wikimedia.org to 0.3.3.9
06:45 jynus: enable semi-sync on db1099:3311 and db1105:3311
06:08 marostegui: s1 failover finished T197069
06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: read only OFF after failover T197069 (duration: 00m 53s)
06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set db1067 as master T197069 (duration: 00m 53s)
06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s1 on ready only for maintenance T197069 (duration: 01m 08s)
06:00 marostegui: Starting s1 failover from db1052 to db1067 - T197069
05:11 Krinkle: mwscript deleteEqualMessages.php --wiki dewiki (1 page) - T45917
05:11 Krinkle: mwscript deleteEqualMessages.php --wiki ruwiki (4 pages) - T45917
05:11 Krinkle: mwscript deleteEqualMessages.php --wiki frwiki (5 pages) - T45917
05:09 Krinkle: mwscript deleteEqualMessages.php --wiki metawiki (deleted 1 page) - T45917
05:09 Krinkle: mwscript deleteEqualMessages.php --wiki test2wiki (deleted 2 pages) - T45917
04:56 marostegui: Starting s1 failover pre steps - https://phabricator.wikimedia.org/T197069
03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 18 03:05:46 UTC 2018 (duration 10m 27s)
02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.13) (duration: 09m 43s)
02:28 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 08m 35s)
00:00 reedy@deploy1001: Finished scap: log event fixes (duration: 33m 18s)
23:26 reedy@deploy1001: Started scap: log event fixes
22:43 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: (no justification provided) (duration: 00m 53s)
22:24 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove override for foundationwiki in wgMobileUrlTemplate (duration: 00m 53s)
22:21 reedy@deploy1001: Synchronized tests/urls.txt: testding! (duration: 00m 53s)
20:57 reedy@deploy1001: Synchronized wmf-config/: make foundation.wikimedia.org writeable T188776 (duration: 00m 53s)
20:50 reedy@deploy1001: Finished scap: updating l10n cache for foundationwiki url changes (duration: 58m 32s)
20:17 ebernhardson: un-ban elastic1030 from eqiad search cluster
19:52 reedy@deploy1001: Started scap: updating l10n cache for foundationwiki url changes
19:35 reedy@deploy1001: Synchronized portals: T199808 (duration: 00m 54s)
19:34 reedy@deploy1001: Synchronized portals/wikipedia.org/assets: T199808 (duration: 00m 53s)
18:46 bblack: re-enabling puppet on mw* for more foundationwiki work - T188776
18:39 ebernhardson: ban elastic1030 from eqiad search cluster for latency issues. Likely related to T156137
18:38 reedy@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 53s)
18:36 bblack: disabling puppet on mw* for more foundationwiki work - T188776
18:33 reedy@deploy1001: Synchronized wmf-config/: foundationwiki (duration: 00m 54s)
18:33 elukey: rolling restart eventstreams on scb2* nodes to avoid OOMs during the EU night
18:32 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: foundationwiki (duration: 00m 53s)
18:31 reedy@deploy1001: Synchronized tests/: foundationwiki (duration: 00m 53s)
18:29 reedy@deploy1001: Synchronized docroot/: update foundationwiki urls (duration: 00m 54s)
18:28 reedy@deploy1001: Synchronized errorpages/: update foundationwiki urls (duration: 00m 54s)
18:17 reedy@deploy1001: Synchronized php-1.32.0-wmf.13/extensions/ConfirmEdit: Make a function public again (duration: 00m 56s)
17:55 bblack: re-enabling puppet on mw* for foundationwiki apache config change T188776
17:00 bblack: disabling puppet on most of mw* for foundationwiki apache config deploy - T188776
16:52 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Make foundationwiki readonly (duration: 00m 54s)
16:49 moritzm: installing faad2 security updates
15:51 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/tests/phpunit/includes/libs/objectcache/BagOStuffTest.php: (no justification provided) (duration: 00m 54s)
15:49 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache/BagOStuff.php: 2d919ad
15:42 XioNoX: switching default analytics-in6 term to reject+log - T198623
15:40 moritzm: rebooting ms-fe1006 with SSBD-enabled microcode
15:18 volans@sarin: conftool action : set/pooled=yes; selector: name=mw2224.codfw.wmnet
14:58 volans: repooled mw2224
14:50 moritzm: rebooting ms-fe1005 with SSBD-enabled microcode
14:47 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.13
14:26 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 06m 14s)
14:19 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
14:18 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.999 (duration: 03m 01s)
14:12 moritzm: rebooting wtp1025 with SSBD-enabled microcode
14:07 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.8 (duration: 03m 11s)
14:03 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.7 (duration: 03m 39s)
13:54 moritzm: rebooting mw1261 with SSBD-enabled microcode
13:35 zfilipin@deploy1001: Pruned MediaWiki: 1.32.0-wmf.6 (duration: 06m 18s)
13:27 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache (duration: 62m 17s)
13:25 moritzm: pruned obsolete hosts from debmonitor
13:06 marostegui: Drop unused grants on db1073, db1117:3323, db1117:3325
12:25 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.13 and rebuild l10n cache
12:03 zeljkof: EU SWAT finished
11:58 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Thesaurus NS at thwikt (T198585) Fix problem with namespaces at patch 446005 (T198585) (duration: 00m 51s)
11:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContentTranslation: SWAT: Fix colors on CX1 (T199503) (duration: 00m 55s)
11:22 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ULS webfonts by default at Burmese Wikipedia (mywiki) (T196219) (duration: 01m 50s)
11:15 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4 (duration: 02m 05s)
11:12 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #4
11:12 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3 (duration: 07m 29s)
11:04 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #3
11:03 mobrovac@deploy1001: Finished deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390 (duration: 19m 43s)
10:43 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20, take #2 - T199458 T195390
10:31 moritzm: restarting archiva to pick up OpenJDK security update
10:31 mobrovac@deploy1001: Started deploy [restbase/deploy@622941d]: Expose the data/mobile/javascript end point, deduplicate most-read results and increase page/related response size to 20 - T199458 T195390
10:14 marostegui: Deploy schema change on db1093 T144010 T51190 T199368
10:08 volans: reimage mw2224 to test the new stretch netboot and wmf-auto-reimage changes
10:07 moritzm: restarting zookeeper in codfw to pick up Java security updates
09:41 marostegui: Deploy schema change on db1113:3316 T144010 T51190 T199368
09:38 marostegui: Drop unused grants from db2037, db2042, db2078:3323, db2078:3325
09:11 marostegui: Deploy schema change on db1098:3316 T144010 T51190 T199368
08:51 moritzm: installing libipc-run-perl update from stretch 9.5 point release
08:31 marostegui: Deploy schema change on db1096:3316 T144010 T51190 T199368
07:50 volans@deploy1001: Finished deploy [debmonitor/deploy@691d2f8]: Release v0.1.6 (duration: 00m 41s)
07:50 volans@deploy1001: Started deploy [debmonitor/deploy@691d2f8]: Release v0.1.6
07:29 godog: un xfs_repair on filesystems reporting negative space available on ms-be1042 - T199198
07:24 moritzm: installing xapian-core security updates
07:17 marostegui: Drop unused grants on es eqiad hosts
07:03 moritzm: updated stretch netinst image for 9.5
06:42 moritzm: installing patch security updates
06:39 moritzm: installing subversion security updates
06:32 marostegui: Drop unused grants on es codfw hosts
06:30 moritzm: installing postgresql-9.6 updates from stretch point release
05:56 marostegui: Deploy schema change on db2039 (s6 codfw master) with replication, this will generate lag on codfw T144010 T51190 T199368
05:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 49s)
05:34 marostegui: Deploy schema change on db1070 (s5 primary master) T144010 T51190 T199368
05:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 49s)
05:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 49s)
05:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 49s)
05:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
05:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 49s)
04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 53s)
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 17 02:45:01 UTC 2018 (duration 10m 21s)
02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 51s)
20:57 ejegg: re-enabled payments listener failmail stream
20:11 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 T174809 ) (duration: 09m 01s)
20:02 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@fcae441]: Update mobileapps to bed7b29 T174809 )
18:35 mepps: updated payments-wiki config to 4e0c6c3cfd
18:10 niharika29@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to all wikis T181193 (duration: 00m 51s)
14:54 moritzm: installing openssh updates from stretch point release
14:52 marostegui: Change expire_log_days on db1067 - https://phabricator.wikimedia.org/T197069
14:49 marostegui: Drop unused ceilometer database from db1073
14:47 marostegui: Remove unused grants from db1073
14:39 marostegui: Remove unused grants from labsdb1004 and labsdb1005
14:37 marostegui: Remove unused grants from es2019
14:27 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2076 (duration: 00m 49s)
14:24 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Sync labs config file, no prod impact (duration: 00m 49s)
14:22 anomie@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Explicitly set wgMultiContentRevisionSchemaMigrationStage to current default (T174044 ) (duration: 00m 50s)
13:59 marostegui: Stop replication on db2076 (db2095's master)
13:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2076 for maintenance (duration: 00m 50s)
12:31 moritzm: rebooting mw2136 for microcode tests
12:14 moritzm: rebooting multatuli for microcode tests
11:56 zeljkof: EU SWAT finished
11:55 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create Reconstruction NS at frwikt (T199631) (duration: 00m 49s)
11:38 vgutierrez: power cycle cp3033 - T199677
11:36 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.12/includes/WebResponse.php: SWAT: WebReponse: Use values altered in WebResponseSetCookie hook (T198525) (duration: 00m 54s)
11:20 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 50s)
11:18 zfilipin@deploy1001: Synchronized robots.txt: SWAT: Remove frwiki outdated entries in robots.txt (T199496) (duration: 00m 49s)
11:14 mobrovac@deploy1001: Finished deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386 (duration: 01m 33s)
11:12 mobrovac@deploy1001: Started deploy [changeprop/deploy@ab8f7e9]: Bug fix: Remove anchors in blacklisting URIs and decode event URIs - T198386
11:06 Amir1: start of ladsgroup@mwmaint1001:~$ mwscript populateChangeTagDef.php --wiki=frwiki
11:05 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp5001.eqsin.wmnet
11:04 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Write to the new change tag backend in frwiki (T194165) (duration: 00m 50s)
11:02 vgutierrez: Power cycling cp5001 to attempt recovering it
10:13 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:12 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
08:29 godog: put back ms-be1036 to full weight - T196873
08:27 marostegui: Drop unused grants on db1108 - this might get dbproxy1009 to complain
08:19 godog: run xfs_repair on filesystems reporting negative space available on ms-be1041 - T199198
08:05 marostegui: Drop unused grants on eqiad hosts
07:33 marostegui: Drop unused grants on codfw hosts
07:19 marostegui: Deploy schema change on dbstore1002:s5 T144010 T51190 T199368
07:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
07:11 marostegui: Deploy schema change on db1110 T144010 T51190 T199368
07:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
05:24 marostegui: Deploy schema change on db2059 T144010 T51190 T199368
05:10 marostegui: Deploy schema change on db2075 T144010 T51190 T199368
02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 16 02:43:43 UTC 2018 (duration 10m 19s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 13m 36s)
03:41 krinkle@deploy1001: Synchronized wmf-config/: I2bff4e
03:39 krinkle@deploy1001: Synchronized wmf-config/ProductionServices.php: Ib079ec90ae515
18:48 Trey314159: reindexing Mirandese wikis on elastic@eqiad (T197890 )
18:42 Trey314159: reindexing Mirandese wikis on elastic@codfw (T197890 )
18:39 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 fully (duration: 00m 50s)
18:33 jynus: droping undocumented grants on m5 T199518
14:34 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 with low load (duration: 00m 50s)
14:03 jynus: stop and reimage db1079
13:54 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 fully, depool db1079 (duration: 00m 50s)
11:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 with low load (duration: 00m 50s)
10:12 moritzm: installing cups/libcups security updates on stretch/trusty
09:45 moritzm: installing libpng security updates on trusty (Debian already updated)
09:36 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/objectcache: 5efa9f6
09:35 aaron@deploy1001: sync-dir aborted: (no justification provided) (duration: 00m 01s)
09:33 moritzm: installing ruby-sprockets security updates on jessie
09:27 jynus: stop and reimage db1078
09:05 jynus: apply updated grants to m5 hosts
08:47 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 (duration: 00m 50s)
07:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 (duration: 00m 51s)
06:50 elukey: powercycle ms-be1041 after diagnostic tests
06:42 elukey: unblocked stuck dpkg processes on an107[2,5] that broke puppet
05:59 jynus: stop and reimage db1087
05:51 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2068 after maintenance (duration: 00m 49s)
05:21 marostegui: Stop MySQL on db2068 for upgrade and check unused grants
05:20 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2068 for grants testing (duration: 00m 50s)
05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
05:06 marostegui: Rename wbc_entity_usage.eu_touched column on db1110 - T144010
05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 52s)
23:32 jforrester@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/VisualEditor: SWAT VisualEditor: Add missing i18n keys to manifest T198064 (duration: 00m 52s)
23:22 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Timeless config simplification, part II Ifc02818b6e
23:20 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Timeless config simplification, part I Ifc02818b6e
21:12 XioNoX: advertising 185.15.56.0/24 to the DFZ - T193496
18:40 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Revert - Wikidata dispatch, disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430925/ (duration: 00m 49s)
18:32 niharika29@deploy1001: Synchronized wmf-config/: Wikidata dispatch, Use a LockManager with short TTL for testwikidata T178652 (duration: 00m 51s)
18:18 niharika29@deploy1001: Synchronized wmf-config/Wikibase.php: Disable dispatching for testwikidatawiki https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/430924/ (duration: 00m 50s)
16:56 chasemp: labcontrol1003:~# neutron net-update 7425e328-560c-4f00-8e99-706f3fb90bb4
16:45 moritzm: restarting smokeping on netmon1002 to pick up new config after wasat rename
15:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 (duration: 00m 50s)
15:49 akosiaris: rolling restart apertium-apy on scb nodes
15:49 akosiaris: downgrade apertium-fra-cat to apertium-fra-cat_1.2.0~r78602-1+wmf2
15:46 XioNoX: progressively pushing new (tighter) mgmt firewall policies
15:07 chasemp: cloudvirt1021:~# /sbin/reboot
14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/Wikibase/repo: T199379 Fix missing label for unit items when displayed on entity page gerrit:445413 (duration: 01m 02s)
14:07 marostegui: Drop unused grants from db2068
13:15 akosiaris: upload apertium-fra-cat_1.3.0~r84327-1+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
13:14 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.12
13:14 moritzm: installing openssh updates from stretch 9.4 point release
13:05 godog: run xfs_repair /dev/sdd1 on ms-be1043 - T199198
12:57 marostegui: Drop unused grants from db1073
11:32 dereckson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix bewikibooks logo (T189218 ) and bnwikisource namespace (T199161 ) (duration: 00m 57s)
11:14 dereckson@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/WikimediaEvents/extension.json: Add event logging for WMDE fundraising banners (duration: 00m 58s)
11:08 godog: swift eqiad-prod: more weight to ms-be1036 after hw repair
09:46 godog: shut down ms-be1041 for hardware diagnostics - T199198
09:17 moritzm: ran puppet node clean/deactivate on db2064, hardware is broken for good and caused ongoing connection failures in cumin/debdeploy (T195228 )
08:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully repool db1083 (duration: 00m 56s)
08:44 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@ba672a3]: It logs disconnected consumers after many rebalances during the outage
08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 55s)
08:30 oblivian@puppetmaster1001: conftool action : set/ttl=300; selector: dnsdisc=eventbus,name=.*
08:28 _joe_: forcing puppet run on scb* to pick up the eventstreams change
08:25 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=eventbus,name=.*
08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
08:20 oblivian@puppetmaster1001: conftool action : set/ttl=10; selector: dnsdisc=eventbus,name=.*
08:11 _joe_: reeabling and running puppet on scb1*
08:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1083 (duration: 00m 56s)
07:37 volans: reimaging spare system californium for testing reimage script changes
07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low weight (duration: 00m 57s)
07:18 marostegui: Stop MySQL on db1083 to pick up new binlog format and MySQL upgrade
07:17 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for maintenance (duration: 00m 58s)
07:01 marostegui: Drop unused grants from dbstore2002:3320 dbstore1001:3311 db2037 db2034
06:38 marostegui: Drop unused grants from db1061 db1096:3316 db1113:3316
06:26 elukey: restart rsyslog on wezen - T199406
05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 57s)
05:01 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1075 (s3 primary master) - T187521
05:00 marostegui: Deploy schema change on db1075 (s3 primary master) T146591 T197891 T196379
04:48 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1123 - T187521
04:48 marostegui: Deploy schema change on db1123 T146591 T197891 T196379
04:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 58s)
03:18 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 12 03:18:58 UTC 2018 (duration 10m 25s)
03:08 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 14m 31s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 44s)
00:15 ejegg: updated CiviCRM from d24c2bc08b b21d783e7f
23:32 thcipriani@deploy1001: Synchronized wmf-config/interwiki.php: Updating interwiki cache (duration: 02m 39s)
23:26 thcipriani@deploy1001: Synchronized wmf-config/throttle.php: SWAT: throttle: lift limits for Kaqchikel edit-a-thon; clear expired rules T199040 (duration: 00m 56s)
23:17 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable anonymous cookie blocking T192017 (duration: 00m 58s)
22:25 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d T195325 T189830 T177619 T196523 ) (duration: 06m 44s)
22:23 Reedy: created wikilove_log on ckbwiki T199336
22:19 ejegg: activated SmashPig recurring donation job
22:18 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d T195325 T189830 T177619 T196523 )
21:53 elukey: restart rsyslog on lithium - in:imtcp stuck in EAGAIN (Resource temporarily unavailable) due to a old socket to tegmen.wikimedia.org
21:47 twentyafterfour: disabled phabricator throttling
21:47 elukey: re-enable kafka mirror maker on kafka100[1-3]
21:41 ppchelko@deploy1001: Finished deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency (duration: 01m 17s)
21:40 ppchelko@deploy1001: Started deploy [changeprop/deploy@45c3807]: Temporary decrease concurrency
21:16 elukey: starting kafka on kafka100[1-3] after zk cleanup
21:14 _joe_: repooling mw1280
20:57 krinkle@deploy1001: Synchronized wmf-config/mc.php: Ifa659de6453 T198239 (duration: 00m 58s)
20:49 _joe_: depooling mw1280 for debugging
20:48 _joe_: repooled mw1223 after investigation
20:39 _joe_: depooling mw1223 for debugging
20:20 bearND: rolled back mobileapps deploy
20:20 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d T195325 T189830 T177619 T196523 ) (duration: 03m 30s)
20:16 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@03fa731]: Update mobileapps to b5e152d T195325 T189830 T177619 T196523 )
20:03 volans: rolling restarting mediawiki API in eqiad with the highest load
19:45 gehel: note: updater was not and will not be using kafka
19:45 gehel: restarting wdqs-updater on wdqs1010 (still not using Kafka)
19:31 elukey: cleaned up *change-prop.retry.change-prop.retry* in /srv/kafka/data on kafka100[1-3]
19:22 akosiaris: stop changeprop on all scb hosts
19:16 _joe_: restarting a few hhvm appservers with high load
18:44 akosiaris: ok, change merged, running puppet on scb hosts
18:34 elukey: restarted topic nuke script for kafka main
18:14 elukey: start kafka on kafka1002
18:14 elukey: stop mirror makers on kafka100[1-3]
18:10 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 fully (duration: 00m 57s)
17:36 _joe_: restarted cpjobqueue in codfw
17:31 elukey: restart kafka on kafka1001 (oom registered)
17:12 elukey: restart kafka on kafka1003 with 2G heap settings
17:10 elukey: restart kafka on kafka1002 with 2G heap settings
17:06 elukey: restarted kafka on kafka1001 with Xmx 2G and Xms 2F
17:05 _joe_: restarting cpjobqueue on scb2001
17:00 oblivian@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=eventbus,name=eqiad
16:50 elukey: stop topics cleaner script
16:40 _joe_: masking and stopping cpjobqueue, changeprop everywhere
16:36 elukey: start topic clean procedure on kafka1001 (tmux root session)
16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 with low load (duration: 00m 58s)
16:19 elukey: restart kafka on kafka1003
16:19 Pchelolo: restart cpjobqueue
16:02 ejegg: disabled failmail logstream for SmashPig
15:40 _joe_: restarting eventbus on kafka-main in eqiad
15:26 chasemp: install dtach on labnet1003
15:18 _joe_: restarting kafka on kafka1002
15:16 gehel: killing wdqs-updater on wdqs10(09|10) to diminish load on kafka
15:11 elukey: restart again kafka on kafka100[1,2] - failed for OOM
15:08 Pchelolo: stop cpjobqueue in eqiad
15:03 elukey: restart kafka on kafka1003
14:58 papaul: shutting down furud to disconnect disk array shelves
14:57 elukey: rolling restart of eventbus on kafka100[1-3]
14:53 elukey: restart kafka on kafka1002
14:52 elukey: restart kafka on kafka1001
14:18 ppchelko@deploy1001: Finished deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386 (duration: 01m 22s)
14:16 ppchelko@deploy1001: Started deploy [changeprop/deploy@2e56855]: Move static blacklisting to change-prop T198386
14:15 volans: reset iLO on hpiLO on conf1005 to see if it fixes the issues with the mgmt interface
13:41 thcipriani@deploy1001: Synchronized README: noop: test scap 3.8.4-1 (duration: 00m 56s)
13:25 zfilipin@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.12 (duration: 00m 57s)
13:24 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.12
13:16 akosiaris: restart apertium-apy for apertium-fra-cat upgrade on scb boxes to 1.3.0~r84327-1+wmf1, T189076
13:14 elukey: roll restart of aqs on aqs* to pick up the new Druid config
13:13 akosiaris: upgrade apertium-fra-cat on scb boxes to 1.3.0~r84327-1+wmf1, T189076
12:59 marostegui: Remove unused grants from db1098:3316
12:43 reedy@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/ContactPage: New config! (duration: 00m 57s)
12:41 reedy@deploy1001: Synchronized wmf-config/MetaContactPages.php: ContactPage new config (duration: 00m 56s)
12:40 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: ContactPage new config (duration: 00m 57s)
12:24 raynor: EU SWAT finished
12:23 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued MFForceSecureLogin (duration: 00m 56s)
12:19 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unused MinervaDownloadIcon (duration: 00m 57s)
12:11 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup: Remove unsued VectorExperimentalPrintStyles (duration: 00m 56s)
12:05 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove ambox images on mobile wikis (T191303) (duration: 00m 57s)
11:53 pmiazga@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page previews for all new editors (T197719) (duration: 00m 56s)
11:38 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/EventBus/: SWAT: Properly handle null content format (duration: 00m 59s)
11:23 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable FileExporter for sourceswiki (T198594) (duration: 00m 56s)
11:12 Amir1: start of ladsgroup@mwmaint1001:~$ foreachwikiindblist wikisource populateChangeTagDef.php
11:10 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikisource (T194165) (duration: 00m 58s)
11:09 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009 (duration: 02m 56s)
11:08 arturo: re-enable puppet in install1002
11:06 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere.Take 2, feeds timed out. T169009
11:06 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009 (duration: 08m 07s)
11:05 jynus: stop db1086 for reimage
11:03 akosiaris: restart HHVM on mw1282
10:58 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 2. Everywhere. T169009
10:45 arturo: disabled puppet in install1002 for a debian installer live hack (T199202 )
10:35 moritzm: installing tiff security updates on jessie hosts
10:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Fully depool db1086 (duration: 00m 56s)
10:11 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/includes/libs/rdbms/ChronologyProtector.php: ee89660
10:09 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/rdbms/ChronologyProtector.php: 552dbdb
10:08 vgutierrez: reimage baham as spare system - T199247
09:55 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009 (duration: 11m 14s)
09:44 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. Take 2, check timed out. T169009
09:41 ppchelko@deploy1001: deploy aborted: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009 (duration: 03m 13s)
09:38 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3]: Upgrade cassandra driver to 3.5.0. Part 1. Only codfw. T169009
09:34 ppchelko@deploy1001: Finished deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009 (duration: 04m 20s)
09:33 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 (duration: 00m 57s)
09:30 ppchelko@deploy1001: Started deploy [restbase/deploy@353eca3] (dev-cluster): Upgrade cassandra driver to 3.5.0 T169009
09:07 moritzm: installing remaining php7 security updates for stretch
08:37 godog: reboot ms-be1040 to run hardware diagnostics - T199198
08:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 56s)
08:14 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db1077 with replication, this will generate lag on s3 labs hosts - T187521
08:13 marostegui: Deploy schema change on db1077 with replication, this will generate lag on s3 labs hosts T146591 T197891 T196379
08:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 56s)
08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 57s)
07:57 elukey: roll restart of aqs on aqs* to rollback the druid config
07:51 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on dbstore1002:s3, db1078 - T187521
07:50 marostegui: Deploy schema change on db1078 T146591 T197891 T196379
07:44 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 56s)
07:15 marostegui: Deploy schema change on dbstore1002:s3 T146591 T197891 T196379
06:59 marostegui: Optimize wbc_entity_usage on bewiki cewiki dawiki hywiki ttwiki on db2043 (s3 codfw master), this will generate lag on s3 codfw - T187521
06:40 marostegui: Deploy schema change on s3 codfw master (db2043) with replication, this will generate lag on s3 codfw T146591 T197891 T196379
06:30 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1062 (s7 primary master) - T187521
06:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 55s)
06:26 marostegui: Deploy schema change on db1062 (s7 primary master) T146591 T197891 T196379
06:12 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1086 - T187521
06:12 marostegui: Deploy schema change on db1086 T146591 T197891 T196379
06:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 57s)
06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 57s)
05:55 tstarling@deploy1001: Synchronized wmf-config/CommonSettings.php: fix logspam, ParserMigration breakage (duration: 00m 57s)
05:49 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1079 with replication, this will generate lag on s7 labs hosts - T187521
05:31 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 labs hosts T146591 T197891 T196379
05:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 56s)
05:23 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1090 - T187521
05:20 marostegui: Deploy schema change on db1090 T146591 T197891 T196379
05:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 56s)
05:06 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on db1094 - T187521
05:05 marostegui: Deploy schema change on db1094 T146591 T197891 T196379
05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 01m 07s)
03:15 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 11 03:15:37 UTC 2018 (duration 10m 23s)
03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.12) (duration: 15m 35s)
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 12m 37s)
01:27 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I423f8fac62
01:01 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I0c621368d6
00:44 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I3d02da810
00:40 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia017dea257d
00:35 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1aad80c3bcb
23:50 ejegg: disabled ProcessSmashPigRecurring in Civi scheduled jobs, to be replaced with process-control job
22:27 ejegg: updated payments-wiki from 383f667171 63372154d0
21:40 chasemp: reboot labnet100[34]
21:04 XioNoX: re-configure GTT circuit in eqiad/knams
20:50 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/tests/phpunit/includes/libs/objectcache/MultiWriteBagOStuffTest.php: 4fba9f6a032
20:48 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/MultiWriteBagOStuff.php: 4fba9f6a032
18:33 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter again (duration: 00m 57s)
16:51 zfilipin@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.12
16:03 vgutierrez: replacing baham with authdns2001 - T196664
15:27 ema: merge alternate_domains vcl patch T164609
15:19 zfilipin@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache (duration: 61m 27s)
14:31 marostegui: Set disk #0 offline for replacement - T199056
14:17 zfilipin@deploy1001: Started scap: testwiki to php-1.32.0-wmf.12 and rebuild l10n cache
14:15 moritzm: installing tiff security updates on jessie
13:53 moritzm: installing ruby-sprockets security updates
13:48 moritzm: installing subversion updates from jessie 9.11 point release
13:40 moritzm: rolling restart of thumbor to pick up new exiv2/openssl
13:29 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.5 (duration: 03m 13s)
13:26 moritzm: installing cups security updates on jessie
13:25 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 (duration: 06m 18s)
13:14 thcipriani@deploy1001: Synchronized scap/plugins/clean.py: no op sync for consistancy Scap clean: remove remote cache directory (duration: 00m 51s)
12:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/includes/libs/objectcache/ReplicatedBagOStuff.php: 4ad6b70
12:24 aaron@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/SpamBlacklist/includes/SpamBlacklist.php: 08a2153f7aa
12:20 aaron@deploy1001: Synchronized php-1.32.0-wmf.12/extensions/SpamBlacklist/includes/SpamBlacklist.php: 583dc7a92f9b
12:06 aaron@deploy1001: Synchronized wmf-config/mc.php: Revert "Make all non-test wikis write to both nutcracker and mcrouter" (duration: 00m 56s)
11:56 ppchelko@deploy1001: Finished deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001 (duration: 02m 52s)
11:53 ppchelko@deploy1001: Started deploy [restbase/deploy@2674971]: Roll out cassandra-driver@3.5.0 to restbase2001
11:43 arturo: updated compiler facts `PUPPET_MASTERS=puppetmaster1001.eqiad.wmnet PUPPET_COMPILER=compiler02.puppet3-diffs.eqiad.wmflabs modules/puppet_compiler/files/compiler-update-facts`
11:32 aaron@deploy1001: Synchronized wmf-config/mc.php: Make all non-test wikis write to both nutcracker and mcrouter (duration: 00m 51s)
11:25 dcausse@deploy1001: Synchronized ./wmf-config/InitialiseSettings.php: [cirrus] cleanup unused config vars 2/2 (duration: 01m 40s)
11:24 moritzm: installing PHP 7 security updates on stretch
11:09 dcausse@deploy1001: Synchronized ./wmf-config/: [cirrus] cleanup unused config vars 1/2 (duration: 00m 53s)
10:27 ppchelko@deploy1001: Finished deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out (duration: 06m 39s)
10:20 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header, take 2, checer timed out
10:20 ppchelko@deploy1001: deploy aborted: Fix up the invalid Vary header (duration: 12m 50s)
10:07 ppchelko@deploy1001: Started deploy [restbase/deploy@d724ad1]: Fix up the invalid Vary header
10:07 moritzm: restarting thumbor on thumbor1001 to pick up exiv2 security updates
10:03 elukey: restart analytics100[1,2]'s hadoop resource managers, some I/O socket errors after the ip6 interface change
09:34 elukey: forced umount of /mnt/hdfs on stat1004, several processes hang for it (causing load) and transport not connected
09:25 godog: swift eqiad-prod add ms-be1036 back gradually - T196873
09:05 moritzm: installing libsoup security updates on jessie/stretch
08:38 moritzm: installing libjpeg-turbo security updates on trusty
08:37 godog: drain and restart cassandra-a on restbase2001 to test a restart
08:34 elukey: rolling restart of AQS to apply the new config
08:09 godog: disable puppet on hosts running cassandra before merging 444247 and 443114
08:08 moritzm: installing openslp security updates on trusty
07:58 moritzm: installing ntp security updates on trusty (may trigger some Icinga warnings about clocks, these recover after a while)
07:33 twentyafterfour: deploying fix for phabricator userpage having hard-coded my username
06:53 twentyafterfour: deployed rPHEX03173dd0097451f60faa3a1705abee58a9fe4c5f
06:32 marostegui: Optimize wbc_entity_usage on arwiki cawiki huwiki rowiki ukwiki on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw - T187521
06:23 marostegui: Deploy schema change on codfw s7 master (db2040) with replication, this will generate lag on s7 codfw T146591 T197891 T196379
06:17 marostegui: Deploy schema change on s8 primary master (db1071) T146591 T197891 T196379
06:06 marostegui: Deploy schema change on dbstore1002:s8 T146591 T197891 T196379
06:02 marostegui: Deploy schema change on codfw s8 master (db2045) with replication, this will generate lag on s8 codfw T146591 T197891 T196379
05:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 50s)
05:40 marostegui: Deploy schema change on s4 primary master (db1068) T146591 T197891 T196379
05:36 marostegui: Deploy schema change on db1091 T146591 T197891 T196379
05:36 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 50s)
05:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 49s)
05:27 marostegui: Deploy schema change on db1081 T146591 T197891 T196379
05:27 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 50s)
05:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
05:21 marostegui: Deploy schema change on db1121 with replication, this will generate lag on s4 labs hosts T146591 T197891 T196379
05:17 marostegui: Optimize frwiki.wbc_entity_usage on s6 eqiad hosts T187521
05:16 marostegui: Deploy schema change on db1103:3314 T146591 T197891 T196379
05:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 50s)
05:04 marostegui: Deploy schema change on db1097:3314 T146591 T197891 T196379
05:03 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 51s)
04:59 marostegui: Optimize frwiki.wbc_entity_usage on s6 codfw, this will generate lag on s6 codfw - T187521
04:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
04:48 marostegui: Deploy schema change on db1084 T146591 T197891 T196379
04:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 52s)
04:36 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1066 (s2 primary master) - T187521
04:28 marostegui: Deploy schema change on s1 primary master (db1052) T146591 T197891 T196379
02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 10 02:44:29 UTC 2018 (duration 10m 18s)
00:21 twentyafterfour: deploying https://phabricator.wikimedia.org/rPHABc6f75c918afa1cc59472c5fe226539e093f6c3ef
23:58 ejegg: updated payments-wiki from ed40f33c44 383f667171
23:58 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop trying to set wgLicenseURL T154069 (duration: 00m 50s)
23:55 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: No need for officewiki-specific upload for MP3s any more I42ffbcd76f6
23:53 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgTmhEnableMp3Uploads Idbf1201813
23:52 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgTmhEnableMp3Uploads I6479dbacd4
23:48 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT Cleanup: Stop setting wmgVisualEditorNonAccountEnableProportion I3b8cccc00
23:47 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Stop setting wgVisualEditorNonAccountEnableProportion If2ee24ae
23:43 jforrester@deploy1001: Synchronized wmf-config/CommonSettings-labs.php: SWAT Cleanup: Remove Beta Cluster use of wikieditor-preview preference Ib16e6e19b1
23:42 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT Cleanup: Remove wgWikiEditorFeatures I7580e94ecf7
23:36 jforrester@deploy1001: Synchronized wmf-config/extension-list: Stop loading the MwEmbedSupport extension, part IV (duration: 00m 50s)
23:34 jforrester@deploy1001: Synchronized multiversion/submodules.json: SWAT MwEmbedSupport extension, part III (duration: 00m 50s)
23:33 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT MwEmbedSupport extension, part II (duration: 00m 51s)
23:26 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT 444650 (duration: 00m 49s)
23:19 jforrester@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ORES: ORES fix Ia1b09d7711
23:18 jforrester@deploy1001: Synchronized wmf-config/CommonSettings.php: Stop loading the MwEmbedSupport extension, part I (duration: 00m 50s)
23:16 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT for bd808 (duration: 00m 51s)
22:54 bstorm_: added disk from new shelf to /srv/dumps on labstore1006
21:48 bawolff: createAndPromote.php --wiki=officewiki BWolff_\(WMF\) --sysop --force
21:45 ejegg: updated payments-wiki from f302c5ffa3 ed40f33c44
19:27 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: GlobalPrefs everywhere https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/444652/ (duration: 00m 51s)
19:01 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES damaging filter on srwiki (T197012 ) (duration: 00m 50s)
18:59 ejegg: updated payments-wiki from 198c31a657 f302c5ffa3
18:52 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable ORES edit quality filters on bswiki (T197010 ) (duration: 00m 50s)
18:48 XioNoX: apply analytics-in6 firewall filter to cr1/2-eqiad with a default permit+log
18:41 ejegg: updated payments-wiki from 43989ebc96 198c31a657
18:37 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Avoid losing cached ParserOutput in doEditContent (T198483 ) (duration: 00m 51s)
18:26 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Resyncing to shut up warnings (duration: 00m 48s)
18:23 catrope@deploy1001: Synchronized wmf-config/: Rollout Watchlist Structured Filters to most wikis (T181193 ) (duration: 00m 53s)
18:14 XioNoX: updating mr1-eqsin security policies - T199074
18:13 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on huwiktionary (T198725 ) (duration: 00m 52s)
17:32 XioNoX: refactoring analytics firewall policies
17:24 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (duration: 09m 18s)
17:14 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater
17:12 gehel@deploy1001: Finished deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 32s)
17:12 gehel@deploy1001: Started deploy [wdqs/wdqs@744613b]: new version of wdqs GUI and updater (wdqs1009 only)
16:09 marostegui: Set disk 32:0 offline on db1069 for a replacement - T199056
15:32 elukey: enabled snappy compression for varnishkafka eventlogging
15:09 akosiaris: repool eqiad mathoid discovery RR
14:52 akosiaris: upgrade eqiad kubernetes cluster to 1.8.14
14:25 akosiaris: depool eqiad mathoid discovery RR for kubernetes upgrade
14:16 moritzm: rmmoding floppy kernel module from hosts which had it loaded prior to blacklisting
13:57 akosiaris: repool codfw mathoid discovery
13:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 50s)
13:32 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on s4 codfw T146591 T197891 T196379
13:29 akosiaris: upgrade kubernetes200{1,2,3,4} to kubernetes 1.8.14
13:29 akosiaris: upgrade acrab to kubernetes 1.8.14
13:26 marostegui: Deploy schema change on db1080 T146591 T197891 T196379
13:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 00m 50s)
13:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 00m 50s)
13:11 marostegui: Deploy schema change on db1114 T146591 T197891 T196379
13:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 50s)
13:10 akosiaris: upgrade acrux to kubernetes 1.8.14
13:00 akosiaris: depool codfw for mathoid in preparation for kubernetes cluster update
11:56 zeljkof: EU SWAT finished
11:56 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add importsources to ru.wikinews (T199045) (duration: 00m 50s)
11:51 moritzm: installing chromium 67.0.3396.87-1~deb9u1 security updates on proton* (tested compatability of new release in deployment-prep)
11:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use new wordmark for bnwikivoyage (T196680) (duration: 00m 51s)
11:35 zfilipin@deploy1001: Synchronized static/images/mobile/copyright/wikivoyage-wordmark-bn.svg: SWAT: Upload wordmark for bnwikivoyage (T196680) (duration: 00m 50s)
11:33 moritzm: installing glibc updates from stretch 9.4 point release
11:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create group eventparticipant on zhwiki (T198167) (duration: 00m 51s)
10:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 50s)
10:35 marostegui: Deploy schema change on db1083 T146591 T197891 T196379
10:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 50s)
10:31 godog: upgrade hp raid firmware on ms-be1017 - T141756
10:12 godog: powercycle ms-be1017 - can't login on ssh/console
10:07 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:06 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
09:22 ppchelko@deploy1001: Finished deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689 (duration: 15m 32s)
09:07 ppchelko@deploy1001: Started deploy [restbase/deploy@794d6ee]: Enable language variants for HTML T190689
09:02 vgutierrez: Bump AES128-SHA traffic redirection to 100% - T192555
08:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 49s)
08:35 jynus: updating password for labdb1004/5 for admin users
08:31 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on db1074 with replication, this will generate lag on s2 on labshosts - T187521
08:26 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labs hosts for s1 T146591 T197891 T196379
08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 50s)
08:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 00m 51s)
08:08 marostegui: Deploy schema change on db1105:3311 T146591 T197891 T196379
08:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 49s)
08:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 50s)
07:54 marostegui: Deploy schema change on db1099:3311 T146591 T197891 T196379
07:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 50s)
07:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 00m 50s)
07:40 marostegui: Deploy schema change on db1067 T146591 T197891 T196379
07:40 ema: reboot lvs canaries for microcode updates: lvs4007 lvs3004 lvs2006 lvs2010 lvs1006 T127825
07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 50s)
07:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 49s)
07:18 marostegui: Deploy schema change on db1119 - T146591 T197891 T196379
07:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 50s)
07:18 elukey: update filter analytics-in4 on cr1/cr2 eqiad
07:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 50s)
06:54 marostegui: Deploy schema change on db1089 - T146591 T197891 T196379
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 50s)
06:41 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1066 (s2 primary master) T197459
06:30 marostegui: Deploy schema change on dbstore1002:s1 - T146591 T197891 T196379
06:25 elukey: restart hue on thorium to pick up new smtp changes - T196920
06:13 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1076 T197459
06:12 marostegui: Deploy schema change on dbstore1001:s1 - T146591 T197891 T196379
06:03 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on dbstore1002:s2, db1122 and db1105 - T187521
06:01 marostegui: Deploy schema change on s1 codfw master (db2048) with replication, this will generate lag on s1 codfw - T146591 T197891 T196379
05:55 marostegui: Deploy schema change on s2 primary master (db1066) - T146591 T197891 T196379
05:53 marostegui: Deploy schema change on db1076 - T146591 T197891 T196379
05:41 marostegui: Optimize bgwiki itwiki svwiki zhwiki wbc_entity_usage on s2 codfw master with replication - lag will happen on s2 codfw - T187521
05:31 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1090 T197459
05:31 marostegui: Deploy schema change on db1090 - T146591 T197891 T196379
05:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 51s)
05:12 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labs hosts - T146591 T197891 T196379
05:04 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1074 with replication, this will generate lag on labs hosts T197459
05:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 52s)
02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 9 02:44:19 UTC 2018 (duration 10m 23s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 33s)
20:37 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I81d566ba860
20:30 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Id94ca5f55c2c6
19:10 chasemp: disable puppet on labstore1007 to turn down ratelimits for nginx (load is rising and rising)
11:27 elukey: restart rsyslog on lithium - in:imtcp thread stuck at 99% cpu usage
23:25 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Icbbccf495899
23:23 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: I001f68a1b39
23:18 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I1b42c260b5ee3
13:02 _joe_: rebooting lvs2001, panic logs from the ethernet card module
18:08 foks: reset 2FA for User:Howcheng
16:06 chasemp: labcontrol1003:~# /sbin/reboot T198950
13:36 moritzm: installing openldap security updates on trusty
13:22 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1103:3312 - T197459
13:22 marostegui: Deploy schema change on db1103:3312 - T146591 T197891 T196379
13:05 moritzm: installing bouncycastle security updates
13:00 moritzm: installing mercurial security updates
12:57 hashar: Restarting Zuul
12:45 mobrovac@deploy1001: Finished deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0 (duration: 15m 32s)
12:38 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db1105 and db1122 - T197459
12:33 marostegui: Deploy schema change on db1105:3312 - T146591 T197891 T196379
12:30 marostegui: Deploy schema change on dbstore1002:s2 and db1122 T146591 T197891 T196379
12:29 mobrovac@deploy1001: Started deploy [restbase/deploy@c42c048]: Update restbase-mod-table-cassandra to v1.1.0
12:18 marostegui: Deploy schema change on s2 codfw master (db2035) with replication, this will generate lag on s2 codfw T146591 T197891 T196379
11:51 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on dbstore1002:s2 - T197459
11:15 marostegui: Optimize {itwiki,enwiktionary,nlwiki}.logging on db2035 (codfw s2 master), this will generate lag on s2 codfw - T197459
11:09 marostegui: Optimize shwiki.logging on s3 eqiad hosts (one by one) T197459
11:08 marostegui: Optimize shwiki.logging on db2043 (codfw s3 master), this will generate lag on s3 codfw - T197459
11:04 marostegui: Optimize frwiki.logging on db1061 (s6 primary master) - T197459
11:02 marostegui: Deploy schema change on db1061 (s6 primary master) T146591 T197891 T196379
11:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 after alter table (duration: 00m 50s)
10:49 marostegui: Optimize frwiki.logging on db1088 - T197459
10:48 marostegui: Deploy schema change on db1088 T146591 T197891 T196379
10:48 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 50s)
10:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 51s)
10:02 marostegui: Optimize frwiki.logging on db1093 - T197459
10:01 mobrovac: restbase depool restbase2001 to test the cassandra node driver v3.5.0 - T169009
10:00 marostegui: Deploy schema change on db1093 T146591 T197891 T196379
10:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 51s)
09:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 51s)
09:46 marostegui: Deploy schema change on db1085 with replication, this will generate lag on s6 labsdb T146591 T197891 T196379
09:08 jynus: restart db1115 mariadb instance (this will cause temporary downtime if tendril and dbtree)
08:53 marostegui: Optimize frwiki.logging on db11085 T197459
08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 51s)
08:18 marostegui: Optimize frwiki.logging on db1113:3316 - T197459
08:15 marostegui: Deploy schema change on db1113:3316 T146591 T197891 T196379
08:12 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3318 after alter table (duration: 00m 50s)
07:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 50s)
07:47 marostegui: Optimize frwiki.logging on db1098:3316 - T197459
07:46 marostegui@deploy1001: sync-file aborted: Depool db1098:3315 for alter table (duration: 00m 01s)
07:45 marostegui: Deploy schema change on db1098:3316 T146591 T197891 T196379
07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 50s)
07:22 moritzm: re-enabled puppet on mw2246
07:12 moritzm: upgrading remaining video scalers to vp9-row-mt enabled ffmpeg build (T190333 )
07:11 marostegui: Optimize frwiki.logging on db1096:3316 - T197459
07:07 marostegui: Deploy schema change on db1096:3316 T146591 T197891 T196379
07:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 51s)
06:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 52s)
06:49 marostegui: Optimize frwiki.logging on dbstore1002 - T197459
06:34 marostegui: Deploy schema change on dbstore1002:s6 T146591 T197891 T196379
06:31 marostegui: Optimize frwiki.logging on db2039 (s6 codfw master), with replication, this will generate lag on s6 codfw - T197459
06:25 marostegui: Deploy schema change on s6 codfw master (db2039) with replication, this will generate lag on s6 codfw T146591 T197891 T196379
06:22 marostegui: Optimize dewiki.logging on db1070 (s5 primary master) - T197459
05:03 kartik@deploy1001: Finished deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 T198830 ) (duration: 03m 26s)
05:01 marostegui: Optimize dewiki.logging on db1110 - T197459
05:00 kartik@deploy1001: Started deploy [cxserver/deploy@6f9fcce]: Update cxserver to bfc9c84 T198830 )
05:00 marostegui: Deploy schema change on s5 primary master (db1070) T146591 T197891 T196379
04:59 marostegui: Deploy schema change on db1110 T146591 T197891 T196379
04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 50s)
04:51 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
04:42 marostegui: Deploy schema change on db1082 with replication T146591 T197891 T196379
00:08 ejegg: updated CiviCRM from e2c8ddd70e d24c2bc08b
23:47 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/Echo/: Handle missing presentation model (T195253 ) (duration: 00m 52s)
23:39 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: Ic87af972e1
23:32 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I8165473c49
23:19 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Disable static maps on bgwiki (duration: 00m 51s)
23:13 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Use require rather than include_once for $wgInterwikiCache (duration: 00m 52s)
21:55 twentyafterfour: whitelist office IPs in phabricator throttle
21:30 bstorm_: disabled unused P440ar RAID controller on labvirt1020
21:15 bstorm_: disabled unused RAID controller on labvirt1019
19:30 gehel: repool wdqs1003, lag almost completely recovered
19:07 ebernhardson: T194678 un-pause cirrussearch writes to codfw
18:58 gehel: depooling wdqs1003 to help it recover update lag
18:55 ebernhardson: T194678 pause cirrussearch writes to codfw to check how kafka+mirrormaker responds
18:48 gehel: restarting blazegraph on wdqs1003 (updater lag)
18:21 thcipriani@deploy1001: Synchronized wmf-config/Wikibase-production.php: SWAT: Set dispatchLagToMaxLagFactor to 60 for wikidata T194950 (duration: 00m 51s)
18:14 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "L" as an alias of Lexeme namespace in wikidata T195493 (duration: 00m 59s)
18:10 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only) (duration: 00m 28s)
18:09 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1010 only)
16:13 jynus: stop and reimage db2045
15:34 mobrovac@deploy1001: Finished scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186 (duration: 30m 50s)
15:26 elukey: upgrade (without jvm restart) prometheus-jmx-exporter on the analytics node listed in debmonitor still not running the last version
15:03 mobrovac@deploy1001: Started scap: Set the Accept-Language header explicitly when making requests to the REST API - T198186
15:02 jynus: stop and reimage db2040
15:00 marostegui: Optimize dewiki.logging on db1082 with replication, this will generate lag on s5 on labsdb hosts/script load irssinotifier - T197459
14:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 50s)
14:50 moritzm: installing php security updates on netmon1002
14:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 50s)
14:15 moritzm: installing dbus updates from stretch point release
14:09 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/EventBus/: SWAT: Dont specify the comment if it is an empty string (duration: 00m 51s)
14:03 jynus: stop and reimage db2052
13:55 moritzm: installing glibc updates from stretch point release
13:55 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Revert "Change logo for eswiki temporarily" (T198846) (duration: 00m 50s)
13:50 moritzm: installing postgresql security updates on maps cluster
13:41 aaron@deploy1001: Synchronized wmf-config/mc.php: Make test wikis just write to both nutcracker and mcrouter (duration: 00m 50s)
13:32 zfilipin@deploy1001: Synchronized wmf-config: SWAT: Enable ORES wp10, draftquality on draft ns (118) for enwiki (T198768) (duration: 00m 51s)
13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Replace Tidy with RemexHtml everywhere (T175706) (duration: 00m 50s)
13:18 marostegui: Optimize dewiki.logging on db1113:3315 - T197459
13:17 marostegui: Deploy schema change on db1113:3315 T146591 T197891 T196379
13:12 ladsgroup@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgChangeTagsSchemaMigrationStage to write both for Wikivoyage, Wikibooks (T194165) (duration: 00m 52s)
11:30 moritzm: installing python-mimeparse updates from jessie point release
11:06 moritzm: rebooting contint1001 for kernel update/enabling microcode
11:05 hashar: Stopping Jenkins CI for kernel upgrade on contint1001
10:31 moritzm: installing ncurses security updates on jessie
09:50 marostegui: Optimize dewiki.logging on db1097:3315 - T197459
09:50 marostegui: Deploy schema change on db1097:3315 T146591 T197891 T196379
09:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 50s)
09:42 moritzm: rebooting ms-be1013/1016/1028/1040 as microcode update canaries
09:40 mobrovac@deploy1001: Finished deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023 (duration: 00m 08s)
09:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 after alter table (duration: 00m 52s)
09:40 mobrovac@deploy1001: Started deploy [zotero/translators@d6702bc]: String.includes() polyfill for Sveriges radio translator - T187023
09:25 moritzm: rebooting ms-fe1005 as microcode update canary
09:11 godog: drain restbase200[234] and restart cassandra to pick up updated certificates
08:57 moritzm: draining restbase1007 for reboot to pick up Intel microcode update
08:42 moritzm: draining restbase2010 for reboot to pick up Intel microcode update
08:26 godog: add graphite2003 to carbon-c-relay frontend on graphite1001 - T196483
08:12 moritzm: draining restbase2001 for reboot to pick up Intel microcode update
07:40 marostegui: Optimize dewiki.logging on db1096:3315 - T197459
07:39 marostegui: Deploy schema change on db1096:3315 T146591 T197891 T196379
07:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 for alter table (duration: 00m 50s)
07:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 52s)
07:13 elukey: stop mariadb on analytics1003 to apply https://gerrit.wikimedia.org/r/443893 and enable auth via unix socket
06:39 kartik@deploy1001: Finished deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 T191124 , T198779 ) (duration: 03m 52s)
06:35 kartik@deploy1001: Started deploy [cxserver/deploy@3cb9a21]: Update cxserver to f8c71a1 T191124 , T198779 )
06:11 marostegui: Deploy schema change on db1100 T146591 T197891 T196379
06:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 52s)
06:09 marostegui: Optimize dewiki.logging on db1100 and dbstore1002:s5 - T197459
05:50 marostegui: Deploy schema change on dbstore1002:s5 T146591 T197891 T196379
05:29 marostegui: Deploy schema change on s3 primary master (db1075) T191316 T192926 T195193
04:57 marostegui: Deploy schema change on s5 codfw host by host T146591 T197891 T196379
04:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1089 original traffic (duration: 00m 51s)
04:42 marostegui: Deploy schema change on s7 primary master (db1062) T191316 T192926 T195193
02:40 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: 370242d13 45f4f61c2
02:32 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jul 5 02:32:29 UTC 2018 (duration 10m 21s)
02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 08m 46s)
00:46 krinkle@deploy1001: Synchronized wmf-config/: Ia3bef874a
22:28 krinkle@deploy1001: Synchronized wmf-config/FeaturedFeedsWMF.php: I004bc9c3e71
22:06 krinkle@deploy1001: Synchronized wmf-config/CommonSettings.php: I5a6e28
21:53 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Iab176f3d205
21:49 krinkle@deploy1001: Synchronized wmf-config/: I6f8cfa8f
21:43 krinkle@deploy1001: Synchronized wmf-config/: Ia8deabc5d2625
21:42 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ia8deabc5d2625
21:36 krinkle@deploy1001: Synchronized wmf-config/: If7da2a26bbf
21:34 krinkle@deploy1001: Synchronized wmf-config/: I1e78ba4365
21:30 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ie30aeecbe
15:55 marostegui: Optimize dewiki.logging on s5 codfw master with replication, this will generate lag on s5 codfw - T197459
15:52 ema: cp300[3-6]: puppet node clean/deactivate T167376
15:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 51s)
14:52 moritzm: installing libipc-run-perl updates from jessie point release
14:36 moritzm: installing perl security updates on trusty (Debian already fixed)
14:25 akosiaris: upgrade kubernetes staging API server to 1.8.14
14:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
14:10 moritzm: installing file/libmagic security updates on trusty (Debian already fixed)
13:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
13:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1089 for after maintenance (duration: 00m 50s)
13:34 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2038, db2047 (duration: 02m 56s)
12:56 marostegui: Stop MySQL and reboot db1089 to upgrade+change it to statement - T197069
12:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for maintenance - T197069 (duration: 02m 57s)
12:39 ema: cp3034 repooled after hw maintenance T189305
12:32 volans: shutting down bast3002 for disk replacement
12:04 moritzm: installing ruby 1.9 security updates on trusty
11:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1077 after alter table (duration: 00m 52s)
11:54 ema: repool cp3043 after hardware maintenance T179953
11:30 ema: shutdown cp3048 and cp3034 (both already depooled) for hardware maintenance T190607 T189305
11:28 moritzm: rolling restart of cassandra on restbase hosts in eqiad completed
11:27 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad completed
11:17 mark: cp3043: mdadm /dev/md0 --add /dev/sdc1 (sdc is former cp3048:sdb)
11:09 mark: cp3043: mdadm /dev/md0 -- fail /dev/sdb1
11:03 ema: depool cp3043 (cache_upload) for hardware maintenance T179953
10:58 godog: update compiler facts
10:53 jynus: stop db2038 and db2047
10:46 marostegui: Deploy schema change on db1077 with replication, this will generate lag on labs s3 T191316 T192926 T89737 T195193
10:42 marostegui: Stop replication on db1077 to drop triggers on db1124:3313 - T192926
10:39 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 00m 50s)
10:28 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 00m 50s)
10:01 moritzm: rolling reboot of sca* for "lazy fpu" kernel updates
09:44 _joe_: stopping all cronjobs via a puppet run on terbium, T192092
09:29 akosiaris: upload kubernetes 1.8.14 to apt.wikimedia.org/stretch-wikimedia/main
09:16 moritzm: uploaded linux-meta 1.18 for jessie-wikimedia to apt.wikimedia.org
09:15 elukey: reimage aqs1009 to Debian Stretch
09:09 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2038, db2047 (duration: 00m 50s)
09:03 marostegui: Optimize recentchanges table on s2 codfw - this will generate lag on codfw s2 - T178290
09:00 akosiaris: manually rebalance the mathoid kubernetes production cluster namespaces pods wise
08:56 moritzm: uploaded linux 4.9.107~wmf1 for jessie-wikimedia to apt.wikimedia.org
08:54 marostegui: Deploy schema change on db1123 T191316 T192926 T89737 T195193
08:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 00m 50s)
08:53 elukey: update analytics-in4 filter rules on cr1/cr2 eqiad - T198623
08:47 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 00m 50s)
08:19 marostegui: Optimize recentchanges table on s6 codfw - this will generate lag on codfw s6 - T178290
08:09 moritzm: rebooting multatuli for kernel update to 4.9.107~wmf1
08:00 marostegui: Deploy schema change on db1078 T191316 T192926 T89737 T195193
07:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 00m 50s)
07:58 ema: install misc VCL on all text hosts T164609
07:53 volans: reimaging silver (spare host, to-be-decomm'ed) as testing host for the reimage script
07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 after alter table (duration: 00m 53s)
07:46 ema: install misc VCL on a text host (cp3030) T164609
07:43 moritzm: resuming rolling restart of cassandra on restbase hosts in eqiad to pick up OpenJDK security update
07:38 hashar: rebased /srv/mediawiki-staging on deploy1001 for beta cluster only change https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/443475/
07:22 moritzm: installing imagemagick security updates
07:18 akosiaris: reimage kubernetes100{1,2}.eqiad.wmnet kubernetes200{1,2}.codfw.wmnet without swap
06:43 _joe_: restarting tilerator on maps-test2004, to check if it can recover
06:15 elukey: reimage aqs1008 to Debian Stretch
06:03 marostegui: Optimize recentchanges table on s7 codfw - this will generate lag on codfw s7 - T178290
05:34 marostegui: Optimize recentchanges table on s3 eqiad, host by host T178290
05:28 marostegui: Optimize recentchanges table on s3 codfw - this will generate lag on codfw s3 - T178290
04:59 marostegui: Deploy schema change on db1101:3317 T191316 T192926 T89737 T195193
04:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 for alter table (duration: 00m 52s)
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jul 4 02:45:51 UTC 2018 (duration 10m 16s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 14m 57s)
00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-2x.png
00:38 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki-1.5x.png
00:37 Krinkle: Purge https://en.wikipedia.org/static/images/project-logos/eswiki.png
00:36 krinkle@deploy1001: Synchronized static/images/project-logos/: T198761 - Update eswiki logo (duration: 00m 51s)
23:08 maxsem@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/CodeMirror: SWAT https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CodeMirror/+/443743/ (duration: 00m 52s)
18:21 anomie@deploy1001: Synchronized wmf-config/CommonSettings.php: Fix for T197475 (gerrit:440543 ) (duration: 00m 52s)
18:09 herron: cobalt:~# systemctl restart gerrit
16:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase es1017,db1100 weights (duration: 00m 50s)
16:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 with low weight (duration: 00m 50s)
16:00 addshore: WikibaseLexeme for Wikidata clients, SE01EP05 deploy slot complete
16:00 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY (duration: 00m 51s)
15:46 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: all wikidata clients (duration: 00m 50s)
15:42 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group1 (duration: 00m 50s)
15:37 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: group0 (duration: 00m 51s)
15:17 godog: reset-failed on ms-be1024 for two user sessions failed when the host was in trouble
15:01 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 51s)
14:59 addshore@deploy1001: sync-file aborted: T197454 EP5 Enable WikibaseLexeme on clients: testwiki (duration: 00m 09s)
14:42 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1017 with low weight (duration: 00m 50s)
14:12 kart_: Finished ContentTranslation draft purge script
14:11 papaul: OS install on graphite2003
14:11 kart_: Running ContentTranslation draft purge script
14:08 moritzm: rolling restart of cassandra on restbase in eqiad to pick up Java security updates
14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Change es1017 IP and rack T197072 (duration: 00m 50s)
14:06 zeljkof: EU SWAT finished
14:05 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Some wikis bureacurats are able to grant non-grantable groups (T197026) (duration: 00m 51s)
13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Clean legacy AddGroups/RemoveGroups (T197024) (duration: 00m 50s)
13:38 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow bcts on private&fishbowl wikis advanced privilege manipulation (T197024) (duration: 00m 50s)
13:31 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add namespace alias on pswikivoyage (T197507) (duration: 00m 49s)
13:24 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable SandboxLink on eswikiversity (T198335) (duration: 00m 51s)
13:19 marostegui: Deploy schema change on dbstore1002:s3 T191316 T192926 T89737 T195193
13:14 zfilipin@deploy1001: Synchronized wmf-config/throttle.php: SWAT: New throttle rule for Wikimania 2018 (T198288) (duration: 00m 53s)
12:49 jynus: shutting down es1017
11:39 elukey: reimage aqs1007 to Debian Stretch
11:02 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 after alter table (duration: 00m 52s)
10:46 jynus: stop and reimage to stretch db2051
10:13 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462 (duration: 00m 52s)
10:12 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@ba672a3]: Decrease checker job concurrency T198462
09:59 marostegui: Deploy schema change on s3 codfw primary master (db2043) with replication, this will generate lag on s3 codfw T191316 T192926 T89737 T195193 T197459
09:56 marostegui: Stop replication on db2094:3313 to replace triggers - T192926
09:44 jynus: reimage db1100 for upgrade to stretch
09:13 marostegui: Deploy schema change on db1098:3317 T191316 T192926 T89737 T195193 T197459
09:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for alter table (duration: 00m 50s)
09:11 elukey: reimage aqs1006 to Debian Stretch
09:09 jynus: deploying sys schema into db1072 (phabricator master)
09:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 after alter table (duration: 00m 50s)
09:04 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 (duration: 00m 51s)
08:32 jynus: stop es1017 mysql for upgrade and maintenance
08:06 moritzm: resuming rolling restart of cassandra on restbase in codfw to pick up Java security updates
07:38 elukey: reimage aqs1005 to debian stretch
07:33 akosiaris: upload apertium-separable_0.3.1-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main T191404
07:18 moritzm: upgrading ffmpeg on mw1307 (T190333 )
07:17 akosiaris: reimage kubestage1001
07:13 marostegui: Deploy schema change on db1090:3317 T191316 T192926 T89737 T195193 T197459
07:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 for alter table (duration: 00m 51s)
07:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 00m 50s)
07:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1017 (duration: 00m 51s)
06:16 marostegui: Stop replication on db1079 to drop triggers on db1125:s7 - T192926
06:14 marostegui: Deploy schema change on db1079 with replication, this will generate lag on s7 on labsdb hosts T191316 T192926 T89737 T195193 T197459
06:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 00m 50s)
06:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 00m 50s)
05:26 marostegui: Deploy schema change on db1094 T191316 T192926 T89737 T195193 T197459
05:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 00m 49s)
05:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 00m 50s)
05:04 marostegui: Deploy schema change on s8 primary master db1071 T191316 T192926 T195193
04:40 marostegui: Deploy schema change on db1086 T191316 T192926 T89737 T195193 T197459
04:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 00m 52s)
04:27 marostegui: Deploy schema change on s2 primary master db1066 T191316 T192926 T195193
02:43 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jul 3 02:43:03 UTC 2018 (duration 10m 19s)
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 07s)
23:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: Watchlist performance fixes (T198359 , T198399 ) (duration: 00m 51s)
23:09 catrope@deploy1001: Synchronized wmf-config/CommonSettings.php: Add chapter wikis to CORS domain list (T181165 ) (duration: 00m 52s)
20:53 urandom: Bringing down Cassandra for hardware testing, restbase2010 - T197477
19:25 urandom: Bringing down Cassandra for hardware testing, restbase2001 - T197477
19:00 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Mark rollbacking revision as patrolled T198449 ; WikiPage: Do not set undid revision ID for rollbacks T190374 (duration: 00m 50s)
18:47 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/: RCFilters: Hide highlight containers when RCFilters is disabled T198440 (duration: 00m 51s)
18:41 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Fix incorrect arguments to prepareContent() call in WikiPage T198483 (duration: 00m 51s)
18:16 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Fix setting of reviewed state based on autopatrol permission T198497 (duration: 00m 52s)
18:11 Niharika: Tried updating interwiki cache but it failed with "<AttributeError> 'Namespace' object has no attribute 'force'"
17:16 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (duration: 09m 04s)
17:07 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph
17:05 gehel@deploy1001: Finished deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only) (duration: 00m 28s)
17:04 gehel@deploy1001: Started deploy [wdqs/wdqs@228f9c5]: new version of wdqs GUI and blazegraph (wdqs1009 only)
15:55 _joe_: stopping jobrunner, jobchron across all jobrunners T198220
15:38 marostegui: Deploy schema change on dbstore1002:s7 T191316 T192926 T89737 T195193 T197459
15:31 andrewbogott: rebooting labnodepool1002 as part of a paging test
15:06 moritzm: installing file/libmagic security updates
15:06 addshore: Lexeme deploy window done...
15:04 twentyafterfour: restarted apache2 on phab1001: whitelist WMDE
14:29 elukey: copy cassandra-tools-wmf 1.0.2-1 from jessie-wikimedia to stretch-wikimedia
14:23 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1018 (duration: 00m 52s)
14:16 kart_: Finished: Dry run: ContentTranslation Draft Purge
14:09 kart_: Dry run: ContentTranslation Draft Purge
13:40 moritzm: installing libgcrypt11 updates on trusty
13:34 elukey: reimage aqs1004 to Debian Stretch
13:29 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/styles/: RCFilters: Fix highlight container selector in Watchlist overrides - T198445 (duration: 00m 51s)
13:28 hashar@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/ContentTranslation: Allow non-Main namespace source titles iff given in URL - T198178 (duration: 00m 52s)
13:21 marostegui: Optimize frwiktionary'.logging on codfw - T197459
13:15 marostegui: Optimize viwiki.logging on codfw - T197459
12:59 ppchelko@deploy1001: Finished deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621 (duration: 01m 26s)
12:57 ppchelko@deploy1001: Started deploy [changeprop/deploy@82fd280]: Revert: Move static blacklisting to change-prop T198386 Can't deploy yet due to scap bug T198621
12:50 marostegui: Stop replication on db2077 to remove triggers from db2095 - T192926
12:49 marostegui: Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on s7 codfw T191316 T192926 T89737 T195193
12:49 ppchelko@deploy1001: Finished deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2 (duration: 00m 24s)
12:48 ppchelko@deploy1001: Started deploy [changeprop/deploy@f20916f]: Move static blacklisting to change-prop T198386 take 2
12:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 00m 52s)
12:24 ppchelko@deploy1001: Finished deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386 (duration: 00m 39s)
12:24 ppchelko@deploy1001: Started deploy [changeprop/deploy@a5a57ff]: Move static blacklisting to change-prop T198386
12:10 akosiaris: T197559 upload lttoolbox_3.4.2-2+wmf2 to apt.wikimedia.org/jessie-wikimedia/main
11:28 ema: repool cp3037 T196974
11:02 moritzm: installing libgcrypt20 security updates
10:06 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:05 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
09:53 elukey: reboot ms-be1039 (bad disk, spike in I/O and load, not reachable via ssh or mgmt console)
09:37 jynus: SET GLOBAL expire_logs_days = 30 on db1067
09:24 marostegui: Deploy schema change on db1087 with replication, this will generate lag on s8 on labs T191316 T192926 T89737 T195193
09:17 marostegui: Stop replication on db1087 to drop triggers on db1124 - T192926
09:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 00m 51s)
09:00 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462 (duration: 00m 50s)
08:59 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@dfdd362]: Disable delayed execution for cirrusSearchCheckerJob T198462
08:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 00m 50s)
08:47 volans@deploy1001: Finished deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command (duration: 00m 24s)
08:46 volans@deploy1001: Started deploy [debmonitor/deploy@15a0776]: GC fix and maintenance command
08:24 moritzm: installing jasper security updates
08:09 jynus: stop and reimage es1018
08:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1018 (duration: 00m 50s)
08:04 gehel: powercycle maps-test2003
04:59 marostegui: Deploy schema change on db1109 T191316 T192926 T89737 T195193
04:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 00m 56s)
04:25 twentyafterfour: Bsadowski1 The throttle is averaged over 5 minutes, you should be unblocked shortly after being blocked
04:24 twentyafterfour: applied security patch on phab1001, source in /srv/patches
02:44 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jul 2 02:44:40 UTC 2018 (duration 10m 21s)
02:34 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 39s)
02:44 andrewbogott: forcing reboot of labservices1001 via mgmt
21:35 ariel@deploy1001: Finished deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063 (duration: 02m 27s)
21:32 ariel@deploy1001: Started deploy [dumps/dumps@a1bc510]: generate temp stubs smarter, T196063
16:28 chasemp: labstore1006:~# service nfs-kernel-server restart
21:37 chasemp: (late log) of VLAN creations for T184209 cloud-instances2-b-eqiad and cloud-instance-transport1-b-eqiad
19:11 herron: stat1005:~# killall -u ironholds T197895
17:28 XioNoX: Re-activating v6 BGP session to Telia on cr2-eqiad - T198502
17:21 XioNoX: deactivating v6 BGP session to Telia on cr2-eqiad - T198502
14:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 00m 50s)
12:30 moritzm: uploaded ffmpeg 3.2.10-1~deb9u1+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (linked against libvpx 1.7 and with backported row-mt support) (T190333 )
11:11 moritzm: installing zsh updates from jessie 8.11 point release
11:01 moritzm: installing lame security updates
10:34 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2058 after reimage (duration: 00m 51s)
10:21 mobrovac@deploy1001: Finished deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 T165105 T197853 (duration: 03m 26s)
10:20 marostegui: Deploy schema change on db1092 T191316 T192926 T89737 T195193
10:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 00m 50s)
10:18 mobrovac@deploy1001: Started deploy [citoid/deploy@40cdff7]: Update citoid to fd77117 T165105 T197853
10:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 00m 50s)
09:23 marostegui: Stop MySQL on db2058 to reimage it
09:23 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2058 for reimage (duration: 00m 50s)
09:18 moritzm: uploaded libvpx 1.7.0-3+wmf1 to apt.wikimedia.org/stretch-wikimedia (component/vp9) (T190333 )
09:08 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2054 after reimage (duration: 00m 51s)
08:39 vgutierrez: Apply new TLS varnishkafka settings in cache::text nodes - T182993
08:27 akosiaris: upgrade php on phab1001 to 5.6.33+dfsg-0+deb8u1
08:03 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1016 (duration: 00m 50s)
07:53 marostegui: Stop MySQL on db2054 for reimage
07:53 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2054 for reimage (duration: 00m 50s)
06:21 jynus: stop and reimage es1016
05:40 elukey: force umount of dumps labstore nfs mountpoints on stat100[56]/notebook100[34] to reduce load (also too many open files)
05:02 marostegui: Deploy schema change on db1104 T191316 T192926 T89737 T195193
05:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 00m 50s)
04:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 00m 50s)
04:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool es1015 (duration: 00m 52s)
01:53 catrope@deploy1001: Finished scap: Full scap to fix i18n issues on testwiki (duration: 32m 42s)
01:20 catrope@deploy1001: Started scap: Full scap to fix i18n issues on testwiki
01:19 krinkle@deploy1001: Synchronized vendor/: I39e592d837
23:54 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources/src/mediawiki.rcfilters/: Watchlist perf fixes (T198359 , T198399 ) (duration: 00m 52s)
23:18 catrope@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Increase Schema:CitationUsage sampling rate to 100% (T191086 ) (duration: 00m 51s)
21:55 niharika29@deploy1001: Synchronized php-1.32.0-wmf.10/extensions/PageTriage/: Update extension directory (duration: 00m 51s)
21:52 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable Draft namespace and AfC mode for PageTriage on testwiki T198143 (duration: 00m 53s)
20:53 dduvall@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.10
20:46 marxarelli: Rolling 1.32.0-wmf.10 to group2 following fix and successful re-deploy to group1
20:29 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10 otra vez
20:07 dduvall@deploy1001: Synchronized php-1.32.0-wmf.10/includes/page/WikiPage.php: Syncing table locking fix (T198350 ) (duration: 00m 57s)
20:04 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Rollback commonswiki to 1.32.0-wmf.8 (resync following ssh hang)
20:02 dduvall@deploy1001: sync-wikiversions aborted: Rollback commonswiki to 1.32.0-wmf.8 (duration: 15m 24s)
19:34 dduvall@deploy1001: rebuilt and synchronized wikiversions files: commonswiki to 1.32.0-wmf.10
19:17 dduvall@deploy1001: Synchronized php: Group1 (less commons) to 1.32.0-wmf.10 (duration: 00m 57s)
19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 (less commons) to 1.32.0-wmf.10
16:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1015 - crashed (duration: 00m 57s)
16:09 moritzm: restarting Cassandra instances on restbase2005 to pick up Java security update
15:01 marostegui: Deploy schema change on db1101:3318 T191316 T192926 T89737 T195193
15:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 00m 57s)
14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 00m 58s)
14:50 raynor: EU SWAT finished
14:49 pmiazga@deploy1001: Synchronized php-1.32.0-wmf.10/skins/Vector/components/watchstar.less: SWAT: Use exactly calculated value to work around a Chrome bug (T196610) (duration: 01m 00s)
14:46 elukey: upgrade piwik 3.2.1 to matomo (new name/package) 3.5.1 - T192298
14:29 moritzm: installing ghostscript security updates
14:26 hashar: CI jobs running npm might suffer from a 10 minutes delay since June 27th | T198348
14:06 gehel: downgrading cassandra to 2.2.6-wmf3 on maps-test2001 (it should never have been upgraded)
13:49 dcausse@deploy1001: Synchronized ./php-1.32.0-wmf.10/includes/htmlform/: Allow overloading of getLabel() with return ' ' (duration: 00m 59s)
13:49 elukey: downgrade cassadra and cassandra-tools from 2.2.6-wmf5 to 2.2.6-wmf3 in jessie-wikimedia component/cassandra22 - T197062
13:25 dcausse@deploy1001: Synchronized ./wmf-config/Wikibase-production.php: Add cirrussearch settings for wikibase (3/3) (duration: 00m 56s)
13:18 dcausse@deploy1001: Synchronized ./wmf-config/: Add cirrussearch settings for wikibase (2/3) (take 2) (duration: 00m 58s)
13:14 vgutierrez: Apply new TLS varnishkafka settings in cache::upload nodes - T182993
13:13 dcausse@deploy1001: Synchronized ./wmf-config/WikibaseSearchSettings.php: Add cirrussearch settings for wikibase (1.5/3) (duration: 00m 56s)
13:01 elukey: upload matomo (new Piwik) 3.5.1-1 to jessie-wikimedia
13:00 moritzm: installing bwm-ng update from jessie 8.11 point release
12:53 moritzm: installing blktrace update from jessie 8.11 point release
12:49 elukey: stop hadoop daemons on analytics1032 + shutdown to swap BBU -T194234
12:37 akosiaris: repool scb1002 T196901
12:37 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
12:08 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=.*,cluster=scb,name=scb1002
12:04 moritzm: installing patch security updates
10:48 twentyafterfour: deployed fix for PhabricatorDataNotAttachedException - https://phabricator.wikimedia.org/rPHEX03971ea8965d3613df69833a766d1502b6d8dabb
10:19 twentyafterfour: hotfixing phabricator DataNotAttached bug
10:18 joal@deploy1001: Finished deploy [analytics/refinery@4fc20a5]: Regular weekly deploy (duration: 08m 21s)
10:11 twentyafterfour: phabricator database migration complete, service restored and appears stable.
10:10 joal@deploy1001: Started deploy [analytics/refinery@4fc20a5]: Regular weekly deploy
10:07 twentyafterfour: running phabricator database migration
10:00 moritzm: installing reportbug update from jessie 8.11 point release
09:58 twentyafterfour: Resuming deployment of phabricator upgrade tagged release/2018-06-27/1 - details: https://phabricator.wikimedia.org/project/profile/3439/ )
09:48 joal@deploy1001: Finished deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected (duration: 02m 48s)
09:45 joal@deploy1001: Started deploy [analytics/aqs/deploy@8eef2a9]: Deploying AQS pageviews-per-country ceiling-value glue code - Corrected
09:29 joal@deploy1001: Finished deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code (duration: 01m 03s)
09:28 joal@deploy1001: Started deploy [analytics/aqs/deploy@194ca96]: Deploying AQS pageviews-per-country ceiling-value glue code
09:20 arturo: T198377 stop nova-spiceproxy daemon in labcontrol1002.wikimedia.org
09:14 mobrovac@deploy1001: Finished deploy [proton/deploy@8a887b5]: Update to dceaf80 T186748 (duration: 00m 28s)
09:13 mobrovac@deploy1001: Started deploy [proton/deploy@8a887b5]: Update to dceaf80 T186748
09:10 vgutierrez: Apply new TLS varnishkafka settings in cache::misc nodes - T182993
08:46 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool es1016 (duration: 01m 04s)
08:46 arturo: aborrero@labtestnet2001:~ 7s 130 $ sudo service nova-spiceproxy stop # daemon in infinite respawning loop
08:15 elukey: restart-hhvm on mw1227 (some threads stuck in jit-related operations, causing high load)
08:09 vgutierrez: updating librdkafka1 && restart varnishkafka instances in cache::text nodes - T182993
07:12 elukey: upload piwik 3.2.1 to jessie-wikimedia
04:46 marostegui: Deploy schema change on db1099:3318 T191316 T192926 T89737 T195193
04:45 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 00m 59s)
03:03 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 28 03:03:34 UTC 2018 (duration 10m 25s)
02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 13m 47s)
02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 54s)
00:33 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/resources: Watchlist perf patches for SWAT, part 2 (T197168 , T198140 , T198142 ) (duration: 00m 57s)
00:32 catrope@deploy1001: Synchronized php-1.32.0-wmf.10/includes: Watchlist perf patches for SWAT, part 1 (T197168 , T198140 , T198142 ) (duration: 01m 13s)
00:13 twentyafterfour: rolled back and restored service to previous state
00:12 twentyafterfour: phabricator update failed. unable to apply database migrations: mysql access denied
00:05 twentyafterfour: taking apache offline momentarily on phab1001
23:57 twentyafterfour: phabricator deployment is coming up in just a couple of minutes. There will be downtime while I run database migrations.
23:10 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turning on page creation log for most wikis T196400 (duration: 00m 58s)
21:50 elukey: piwik maintenance on bohrium completed
21:42 XioNoX: setting BFD of the Zayo eqiad-codfw link to standard of 300
20:02 dduvall@deploy1001: Synchronized php: (no justification provided) (duration: 00m 57s)
20:02 SMalyshev: applied fix for T197447 to eqiad wdqs cluster, which involved restart of the services
19:54 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group1 rolled back to 1.32.0-wmf.8
19:52 marxarelli: Rolling back group1 due to rise in error rate (T198350 )
19:22 marxarelli: errors seem due to "INSERT INTO `revision_comment_temp`" statements and lock wait timeout
19:18 marxarelli: seeing rising "Wikimedia\Rdbms\DBQueryError from line 1443 of /srv/mediawiki/php-1.32.0-wmf.10/includes/libs/rdbms/database/Database.php: A database query error has occurred. Did you forget to run your application's database schema update..." errors
19:16 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.10 (duration: 00m 58s)
19:15 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.10
17:57 XioNoX: updating NTP servers on network devices
17:53 mobrovac@deploy1001: Finished deploy [proton/deploy@cd6ed94]: Update proton to 491e966 T186748 T197856 (duration: 00m 35s)
17:52 mobrovac@deploy1001: Started deploy [proton/deploy@cd6ed94]: Update proton to 491e966 T186748 T197856
16:39 marostegui: Deploy schema change on dbstore1002:s8 T191316 T192926 T89737 T195193
15:38 thcipriani@deploy1001: Synchronized README: Scap 3.8.3-1 noop test sync-file (duration: 00m 56s)
15:36 marostegui: Stop replication on db2094:3318 to update triggers on archive table
15:30 godog: upload scap 3.8.3 - T198277
14:51 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 (duration: 00m 56s)
13:57 dcausse: EU swat done
13:42 dcausse@deploy1001: Finished scap: wmf-config Add cirrussearch settings for wikibase (1/3) (duration: 05m 41s)
13:40 moritzm: uploaded debmonitor 0.1.5 to apt.wikimedia.org
13:36 dcausse@deploy1001: Started scap: wmf-config Add cirrussearch settings for wikibase (1/3)
13:24 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: Change FileImporter config data location (T198050) (duration: 00m 57s)
13:07 elukey: piwik upgraded to 3.2.1 on bohrium + started the db migration procedure (will last 2/3h probably)
12:49 vgutierrez: Upgrade librdkafka1 and restart varnishkafka-webrequest in cache::upload nodes - T182993
11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 after alter table (duration: 00m 57s)
11:26 marostegui: Deploy schema change on s8 codfw master (db2045) with replication, this will generate lag on s8 codfw T191316 T192926 T89737 T195193
10:38 gehel: removing maps-test2003 from cluster for reimage - T198290
10:36 volans@deploy1001: Finished deploy [debmonitor/deploy@9536ebf]: CSP header hotfix (duration: 00m 22s)
10:36 volans@deploy1001: Started deploy [debmonitor/deploy@9536ebf]: CSP header hotfix
10:24 marostegui: Deploy schema change on db1076 T191316 T192926 T89737 T195193
10:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for alter table (duration: 00m 56s)
10:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1074 after alter table (duration: 00m 57s)
10:11 jynus: stopping db1067 and reimage it
09:38 volans@deploy1001: Finished deploy [debmonitor/deploy@052a9ea]: Release v0.1.5 (duration: 00m 24s)
09:38 volans@deploy1001: Started deploy [debmonitor/deploy@052a9ea]: Release v0.1.5
09:14 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 (duration: 00m 56s)
08:35 marostegui: Deploy schema change on db1074 with replication, this will generate lag on s2 on labsdb T191316 T192926 T89737 T195193
08:33 marostegui: Stop replication on db1074 to remove triggers from db1125 - T192926
08:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1074 for alter table (duration: 00m 57s)
08:26 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1090:3312 after alter table (duration: 00m 57s)
07:58 vgutierrez: Reinstall acamar & achernar as spare systems
07:50 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns4001.wikimedia.org
07:38 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns4001.wikimedia.org
07:37 vgutierrez: Depool dns4001 for server restart - T198215
05:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1090:3312 for alter table (duration: 01m 06s)
05:08 marostegui: Deploy schema change on db1090:3312 T191316 T192926 T89737 T195193
04:47 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba
04:42 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@2207b66]: Update mobileapps to d7221ba
03:04 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 27 03:04:07 UTC 2018 (duration 10m 32s)
02:53 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.10) (duration: 15m 53s)
02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 33s)
21:26 dduvall@deploy1001: rebuilt and synchronized wikiversions files: Group0 to 1.32.0-wmf.10
21:18 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 22m 50s)
20:56 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
20:47 marxarelli: removing orphaned /srv/mediawiki/php-1.31.0-wmf.28 dir on deploy1001
20:38 volans: regenerated debian installer for jessie 8.11 point release (released 3 days ago)
20:34 marxarelli: removed /srv/mediawiki/php-1.32.0-wmf.4 on mwdebug2002 to free disk space
20:22 marxarelli: manually cleaning up on mwdebug2002 due to full disk and scap failures
20:21 dduvall@deploy1001: Pruned MediaWiki: 1.32.0-wmf.4 [keeping static files] (duration: 03m 21s)
19:43 marxarelli: cdb-rebuild failed on mwdebug2002 due to lack of disk space
19:41 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache (duration: 39m 11s)
19:02 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
18:39 marxarelli: setting $wmgUseMwEmbedSupport = false in php-1.32.0-wmf.10/LocalSettings.php to extension registry exception (see T197918 and T191056 )
18:14 marxarelli: scap sync for testwiki is currently failing on l10update due to MwEmbedSupport's deprecation but no change yet to mediawiki/extensions to remove the extension. blocking train on T197918
18:01 dduvall@deploy1001: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2905568126" --threads=30 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 23s)
17:59 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.10 and rebuild l10n cache
17:16 marxarelli: starting branch cut for 1.32-wmf.10
16:21 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3312 after alter table (duration: 00m 57s)
14:43 elukey: rm syslog.1.gz puppet.log.1.gz on tegment to fix cronspam
14:19 ottomata: Re-enable job and change-prop Kafka topic mirroring from main-eqiad -> main-codfw
13:37 hashar: European swat completed
13:36 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable zh-my variant on zhwiki - T193983 (duration: 00m 57s)
13:31 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logos for ruwikiquote - T197508 (duration: 00m 57s)
13:29 hashar: purgeList.php for the various ruwikiquote logos
13:27 hashar@deploy1001: Synchronized static/images/project-logos: Change logo resources for ruwikiquote - T197508 (duration: 00m 56s)
13:25 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable TemplateStyles on ruwikiquote - T197526 (duration: 00m 57s)
13:22 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on wikidatawiki_content
13:21 dcausse: elastic@eqiad deleting stale index viwiki_general_1525301649
13:19 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Create a few of namespace aliases for ruwiktionary - T197565 (duration: 00m 57s)
13:06 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add some namespace aliases to ruwikiquote - T197058 (duration: 00m 56s)
12:54 chasemp: labcontrol1003 aptitude install ldap-utils python-ldappool
12:15 marostegui: Deploy schema change on db1105:3312 T191316 T192926 T89737 T195193
12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3312 for alter table (duration: 00m 57s)
12:07 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3312 after alter table (duration: 00m 57s)
11:09 godog: roll-upgrade thumbor in codfw/eqiad
11:08 godog: upload thumbor 2.0-2 to apt.wikimedia.org
10:40 godog: upgrade thumbor to 2.0-1 on thumbor1001
10:10 godog: test upgrade thumbor 2.0 on thumbor2001
10:10 akosiaris: proceed with apertium-apy upgrade on all scb hosts. T194342
10:01 akosiaris: downgrade erroneously upgraded apertium-fra-cat on scb1001, scb1002 from 1.3.0~r84327-1+wmf1 to 1.2.0~r78602-1+wmf2 T194342
09:40 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=codfw
09:39 akosiaris: re-enable icinga-wm that I 've previously silenced
09:37 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=api_appserver,dc=eqiad
09:36 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=appserver
09:33 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327 (duration: 05m 07s)
09:31 akosiaris: upgrade apertium-apy, apertium-fra-cat, python3-streamparser, enable puppet, run puppet on scb1002 as a last test before full cluster upgrade. T194342
09:31 vgutierrez: updating librdkafka1 and restarting varnishkafka-webrequest on cache::misc nodes - T182993
09:29 dcausse: elastic@eqiad: forcemerge (only_expunge_deletes=true) on commonswiki_file
09:28 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync with HHVM restarts, take #3 - T190327
09:25 akosiaris: manually install python3-streamparser on scb1001 for testing. T194342
09:19 mobrovac@deploy1001: Finished deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626 (duration: 01m 14s)
09:18 mobrovac@deploy1001: Started deploy [changeprop/deploy@5d8eaf1]: Bug fix: Add restore to WD description checks - T197626
09:11 mobrovac@deploy1001: Finished scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327 (duration: 08m 34s)
09:09 vgutierrez: upload librdkafka_0.11.3-1~bpo8+1+wikimedia2 to apt.w.o jessie-wikimedia - T182993
09:08 akosiaris: upgrade scb1001 apertium-apy, apertium-fra-cat, enable puppet and run puppet
09:06 akosiaris: T194342 . Disable puppet on scb hosts for apertium-apy upgrade
09:04 dcausse: unbanning elastic1042
09:03 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync, take #2 - T190327
08:52 mobrovac@deploy1001: scap failed: RuntimeError scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details) (duration: 04m 27s)
08:52 mobrovac@deploy1001: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
08:48 mobrovac@deploy1001: Started scap: Switch the last remaining jobs to EventBus, full scap sync for clean-up - T190327
08:47 mobrovac@deploy1001: Synchronized wmf-config/CommonSettings.php: Switch the last remaining jobs to EventBus, only CommonSettings.php now - T190327 (duration: 00m 58s)
08:47 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327 (duration: 00m 58s)
08:46 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@7d9a1aa]: Enable all jobs in kafka queue T190327
07:58 vgutierrez: Replaced acamar and achernar with dns200[12] as main lvs name servers in codfw - T196493
07:13 _joe_: restarting ircecho, was not reporting alerts
07:05 marostegui: Deploy schema change on db1103:3312 T191316 T192926 T89737 T195193
07:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3312 for alter table (duration: 00m 56s)
06:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1122 after alter table (duration: 00m 57s)
05:58 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 55s)
05:57 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove db1054, it is going to be decommissioned T197063 (duration: 00m 57s)
05:52 marostegui: Stop MySQL on db1054 as it is going to be decommissioned - T197063
05:28 SMalyshev: testing fix for T197447 on wdqs1009
05:15 marostegui: Deploy schema change on db1122 T191316 T192926 T89737 T195193
05:15 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1122 for alter table (duration: 00m 59s)
03:05 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 12m 59s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 26s)
23:40 thcipriani: restarting jenkins for updates
23:30 twentyafterfour: After monitoring logstash, everything appears stable. Evening SWAT is complete.
23:23 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440878/ for SWAT: no-op (duration: 00m 56s)
23:10 twentyafterfour@deploy1001: Synchronized wmf-config/Wikibase-production.php: sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440420/ for SWAT (duration: 00m 57s)
21:04 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52
20:58 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@770cdb0]: Update mobileapps to 8c76d52
20:09 mdholloway: deploying mobileapps to canary (scb2001) failed, rolling back to investigate
20:07 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0
20:06 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@fb70d4f]: Update mobileapps to 362f2a0
20:02 cscott: Completed deploy of Parsoid to version b068bb51 T197949 , T197702 , T196799 )
19:53 cscott@deploy1001: Finished deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
19:46 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
19:22 cscott@deploy1001: Started deploy [parsoid/deploy@5925200]: Updating Parsoid to b068bb51
18:33 thcipriani@deploy1001: Synchronized portals: SWAT: Bumping portals to master T128546 (duration: 00m 56s)
18:32 thcipriani@deploy1001: Synchronized portals/wikipedia.org/assets: SWAT: Bumping portals to master T128546 (duration: 00m 57s)
18:25 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add import sources to ta.wiktionary T196445 (duration: 00m 58s)
18:13 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Enable logging for Schema:CitationUsage T191086 (noop beta-only sync) (duration: 00m 57s)
18:07 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (duration: 08m 19s)
17:58 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater
17:56 gehel@deploy1001: Finished deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 24s)
17:56 gehel@deploy1001: Started deploy [wdqs/wdqs@e9a1e13]: new version of wdqs GUI and updater (wdqs1009 only)
17:36 ottomata: Re-enable job and change-prop kafka topic mirroring from main-codfw -> main-eqiad
16:45 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: Beta only gerrit:441909 (duration: 00m 57s)
16:19 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=achernar.wikimedia.org
16:13 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=acamar.wikimedia.org
16:04 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2002.wikimedia.org
16:00 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns2001.wikimedia.org
15:21 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: REVERT: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 57s)
15:16 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on testwiki (again again) T197454 (duration: 00m 55s)
15:05 addshore: FileImporter slot done, moving onto Lexeme slot
15:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileExpoter on ar-, de- and fa-wiki T196969 T197066 (duration: 00m 57s)
14:58 marostegui: Deploy schema change on dbstore1002:s2 T191316 T192926 T89737 T195193
14:57 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable FileImporter on Wikimedia Commons T196969 (duration: 00m 56s)
14:40 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 56s)
14:37 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Ignore namespace string when checking templates and categories (duration: 00m 58s)
14:32 addshore@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 57s)
14:30 addshore@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/FileImporter: Add a temp config to allow interwiki linking T196976 (duration: 00m 58s)
14:23 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Enable license filters for the FileImporter in production T194502 (duration: 00m 55s)
14:19 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: Add ar, de and fa wikipedia to FileImporter interwiki config T196969 T196976 (duration: 00m 56s)
14:17 hashar@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 59s)
14:14 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 2 (moving them back from environments/production)
14:13 gehel: banning and depooling elastic1042 (high response times)
14:01 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config, part II - T197633 (duration: 00m 57s)
13:59 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Whitelist two Indian government websites - T197944 (duration: 00m 57s)
13:57 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
13:54 hashar: mwscript namespaceDupes.php --wiki=zhwikivoyage --fix | T198007
13:53 elukey: merging jmxtrans and kafkatee's submodules to operations/puppet - part 1 (moving them under environments/production)
13:51 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Autoconfirmed should require 10 edits&4 days on zhwikivoyage - T198006 (duration: 00m 56s)
13:50 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add namespace aliases to zhwikivoyage - T198007 (duration: 00m 56s)
13:37 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Remove BN alias for NS_USER on dewikivoyage - T196905 (duration: 00m 57s)
13:34 hashar@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Echo: Remove masterPos from the job specification - T192945 (duration: 00m 58s)
13:32 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix ORES config - T197633 (duration: 00m 58s)
13:27 hashar@deploy1001: Synchronized dblists/commonsuploads.dblist: Add wikimania2018wiki to commonsupload.dblist | T197714 (duration: 00m 56s)
13:13 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable logging for Schema:CitationUsage at 1% | T191086 (duration: 00m 57s)
13:05 marostegui: Deploy schema change on db2035 (codfw primary master for s2) with replication, this will generate lag on s2 codfw T191316 T192926 T89737 T195193
13:05 godog: refresh cassandra certificates for 'restbase' cluster
13:00 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 00m 57s)
12:29 kartik@deploy1001: Finished deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a T191874 , T196354 , T195768 , T195768 ) (duration: 03m 33s)
12:26 kartik@deploy1001: Started deploy [cxserver/deploy@cc6dc61]: Update cxserver to ece5e7a T191874 , T196354 , T195768 , T195768 )
12:04 mobrovac: Restbase: Add the "headers" field to Cassandra schemae in production for T197789
12:00 mobrovac@deploy1001: Finished deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions (duration: 04m 05s)
12:00 gehel: repooling wdqs1005 it has catched up on updates - T198042
11:56 mobrovac@deploy1001: Started deploy [restbase/deploy@f521e7e] (dev-cluster): Dev cluster: Include pagelanguage in title_revisions
10:32 akosiaris: increase CPU count for proton machines from 2 to 10. T197862
08:22 marostegui: Deploy schema change on db1106 with replication, this will generate lag on s1 labs T191316 T192926 T89737 T195193
08:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 00m 56s)
08:16 marostegui: Stop replication on db1106 to change triggers for db1124 and db1095 - T192926
08:11 gehel: rolling restart of wdqs to enable async logging - T198051
08:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 58s)
07:46 gehel: depooling wdqs1005 to allow it to catch up on updates - T198042
07:33 moritzm: slow rollout of most of the remaining debmonitor client installations
05:43 marostegui: Deploy schema change on db1080 T191316 T192926 T89737 T195193
05:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 17s)
02:39 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 37s)
02:21 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 40s)
22:26 gehel: restarting wdqs1004 - T198042
21:51 gehel: restarting wdqs1005 after taking a few thread dumps - T198042
19:41 gehel: restart blazegraph on wdqs1004
16:09 godog: restart wdqs-updater on wdqs1005
15:40 godog: restart wdqs-blazegraph on wdqs1005
15:40 jynus: restart graphite2001
15:35 godog: systemctl restart wdqs-blazegraph on wdqs1004
15:33 godog: restart-wdqs on wdqs1004
19:54 jynus: applying transaction manually on dbstore1002 due to weird bug with savepoint on tokudb+image table
21:38 ebernhardson: banned elastic1036 from search cluster, waited for all load to shift away, and unbanned
16:38 ejegg: updated CiviCRM from 349d43eb8b e2c8ddd70e
13:28 vgutierrez: Bump AES128-SHA pageview replacement to 4%
12:49 chasemp: remove labvirt1019 canary to start debug of network
11:08 hashar: Refreshed operations-puppet-tests-docker jenkins job to a new Docker container build that includes isc-dhcp-server | https://gerrit.wikimedia.org/r/c/integration/config/+/441367
02:51 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 07m 40s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 12s)
21:18 ejegg: updated 'domain' settings in CiviCRM so name is now 'Wikimedia Foundation' and description is 'donate.wikimedia.org'
20:08 bearND: rolled back "Update mobileapps to 9d856ec" (was just on canary)
20:07 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec
20:03 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@3420e67]: Update mobileapps to 9d856ec
19:54 ottomata: removed Kafka MirrorMaker from kafka10(12|13|14)
16:01 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 03m 08s)
15:58 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
15:57 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN (duration: 00m 27s)
15:56 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: Kartotherian: make Pl fallback to EN
15:52 sbisson@deploy1001: Finished deploy [kartotherian/deploy@9af9191]: (no justification provided) (duration: 03m 36s)
15:49 sbisson@deploy1001: Started deploy [kartotherian/deploy@9af9191]: (no justification provided)
14:59 dcausse: disk space issue on elastic1020 is due to shard rebalancing (currently receiving 2 enwiki_general shards but removing one wikidatawiki_content)
14:07 imarlier@deploy1001: Finished deploy [performance/navtiming@742edb0]: (no justification provided) (duration: 00m 04s)
14:07 imarlier@deploy1001: Started deploy [performance/navtiming@742edb0]: (no justification provided)
14:03 imarlier@deploy1001: Finished deploy [performance/navtiming@8914e26]: (no justification provided) (duration: 00m 05s)
14:03 imarlier@deploy1001: Started deploy [performance/navtiming@8914e26]: (no justification provided)
13:43 imarlier@deploy1001: Finished deploy [performance/navtiming@995cb0f]: (no justification provided) (duration: 00m 05s)
13:43 imarlier@deploy1001: Started deploy [performance/navtiming@995cb0f]: (no justification provided)
13:38 anomie: Re-running populateExternallinksIndex60.php on plwiki and ptwiki for phab:T59176 (initial run collided with the s2 master switch).
08:57 _joe_: removing user.log as well on tegmen
08:54 _joe_: running logrotate on tegmen
08:54 _joe_: killall nsca on tegmen
08:51 _joe_: restarting nsca daemon on tegmen (gone wild, hundreds of subprocesses
08:47 _joe_: removing user.log.1 and messages.log.1 on tegmen to save some space
07:40 _joe_: powercycled mw1272, down since yesterday
07:29 hashar: deploy1001: rebased php-1.32.0-wmf.8/extensions/Translate to catch up with a non production merged change ( https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Translate/+/441127 ).
02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 08m 17s)
16:10 mobrovac@deploy1001: Finished deploy [proton/deploy@43af7d9]: (no justification provided) (duration: 00m 27s)
16:09 mobrovac@deploy1001: Started deploy [proton/deploy@43af7d9]: (no justification provided)
14:13 herron: re-enabling puppet agents
14:11 herron: increased client_max_body_size on puppetdb nginx frontends from 30m to 60m https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
14:06 herron: temporarily disabling puppet agents for deploy of https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/441044/
12:41 _joe_: hard reboot of ms-be1019 - unable to ssh, console showing i/o errors only
04:56 ebernhardson: unban elastic1035
04:45 ebernhardson: ban elastic1035 from cluster to allow it to recover
02:36 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 06m 53s)
02:19 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 07m 39s)
19:49 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@bcb2904]: GUI update (duration: 18m 52s)
19:30 smalyshev@deploy1001: Started deploy [wdqs/wdqs@bcb2904]: GUI update
19:30 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: GUI update (duration: 00m 56s)
19:29 smalyshev@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: GUI update
18:41 Trey314159: reindexing Serbian wikis on elastic@eqiad (T196404 )
16:39 urandom: DROP unused Cassandra keyspaces - T197080
10:52 _joe_: removing wtp1043 from all pybal configuration until the disk is replaced T196886
10:04 _joe_: initialize namespace "ci" on the kubernetes staging cluster T196654
09:56 joal@deploy1001: Finished deploy [analytics/refinery@e9dbe79]: Regular weekly deploy (duration: 06m 54s)
09:49 joal@deploy1001: Started deploy [analytics/refinery@e9dbe79]: Regular weekly deploy
03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.999) (duration: 14m 30s)
02:33 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 13m 12s)
02:27 Trey314159: reindexing Serbian wikis on elastic@codfw (T196404 )
22:24 no_justification: gerrit: started back up, nvm
22:23 no_justification: gerrit: stopping services for a few minutes
13:05 Trey314159: reindexing Serbo-Croatian wikis on elastic@eqiad (T196658 )
03:46 Trey314159: reindexing Serbo-Croatian wikis on elastic@codfw (T196658 )
20:42 mdholloway: disabled puppet on maps-test2004 for testing new map styles setup
19:24 Trey314159: reindexing Croatian wikis on elastic@eqiad (T196658 )
13:19 Trey314159: reindexing Croatian wikis on elastic@codfw (T196658 )
00:15 krinkle@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Ica4cb6644
18:48 Trey314159: reindexing Bosnian wikis on elastic@eqiad (T196658 )
18:10 Trey314159: reindexing Bosnian wikis on elastic@codfw (T196658 )
17:15 krinkle@deploy1001: Synchronized wmf-config/mc.php: I619a2ff5db611
15:49 mutante: install2002 - disabling puppet temp, live hackking DHCP config for debugging backup2001 install issue
15:37 mutante: switching noc.wikimedia.org site from terbium to mwamiant1001 backend, running puppet on all cache::misc cp servers (T192092 )
15:19 ottomata: bouncing kafka broker on kafka-jumbo1001 to test https://gerrit.wikimedia.org/r/#/c/440520/
15:19 gehel: rolling restart of elasticsearch eqiad for plugin upgrade completed - T194245
14:49 elukey: restart varnishkafka-eventlogging on cp4028, errors logged
14:43 elukey: restart varnishkafka-eventlogging on cp5012 as attempt to clear out the errors (not needed but logging it anyway)
14:12 papaul: OS install on backup2001
13:41 mepps: updated process control to 3b9f51e0fd
12:56 legoktm@deploy1001: Synchronized wmf-config/mc.php: Make sure that mcrouter BagOStuff goes through ObjectCache::newFromParams() - T197450 (duration: 00m 57s)
12:38 jynus: killing refresh counts on commonswiki
12:30 jynus: reenabling db2040 consistency after slaves caught up
12:13 papaul: OS install on bast2002
10:32 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 00m 58s)
09:48 godog: delete cpjobqueue metrics older than 10d - T196067
09:45 jynus: reenabling db2048 consistency after slaves caught up
09:43 jynus: reducing temp. db2040 consistency to speed up slave lag catch up
09:20 godog: fully remove ms-be1036 from swift due to hw failure - T196873
09:03 bawolff: deploy patch T197279
08:05 gehel: rolling restart of elasticsearch eqiad for plugin upgrade - T194245
05:50 moritzm: slow rollout of debmonitor
05:34 moritzm: installing gnupg security updates on trusty (Debian already fixed)
05:13 marostegui: Deploy schema change on db1119 T191316 T192926 T89737 T195193
05:13 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 00m 57s)
05:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 07s)
23:40 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.3 (duration: 04m 38s)
23:31 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.2 (duration: 04m 50s)
23:24 jforrester@deploy1001: Synchronized php-1.32.0-wmf.8/resources/src/mediawiki.rcfilters/styles/mw.rcfilters.less: T195903 SWAT for Roan (duration: 00m 59s)
23:12 jforrester@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196400 for SWAT (duration: 01m 00s)
21:56 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.999/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 57s)
21:47 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.999 T196585
21:44 thcipriani@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/PageTriage/includes/Hooks.php: Fix event presentation class names T197262 (duration: 00m 52s)
21:27 thcipriani@deploy1001: Pruned MediaWiki: 1.32.0-wmf.1 (duration: 06m 54s)
21:18 thcipriani@deploy1001: Finished scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585 ) and rebuild l10n cache (duration: 65m 50s)
21:12 bblack: re-enable and run puppet on cache_upload - T192555
20:45 bblack: re-enable and run puppet on rest of cache_text (eqiad, eqsin, esams) - T192555
20:21 bblack: re-enable and run puppet on text@ulsfo - T192555
20:12 thcipriani@deploy1001: Started scap: testwiki to 1.32.0-wmf.999 (multi-content-revisions T196585 ) and rebuild l10n cache
20:11 bblack: re-enable and run puppet on text@codfw - T192555
19:21 vgutierrez: Reenable puppet in cache:misc nodes - T192555
19:13 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.8
18:17 niharika29@deploy1001: Synchronized wmf-config/CommonSettings.php: Bump ExtensionDistributor default to REL1_31 (duration: 00m 57s)
18:14 niharika29@deploy1001: Synchronized langlist-labs: beta: declare beta sr.wikipedia and beta crh.wikipedia to langlist-labs (duration: 00m 58s)
18:10 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on a few more wikis - T195263 (duration: 01m 00s)
17:34 gehel: rolling restart of elasticsearch codfw completed - T194245
17:10 ottomata: bouncing kafka broker on kafka2003 to make sure ACLs are okk
17:08 ottomata: applying ACLs to Kafka main-codfw and main-eqiad - T196081
16:41 vgutierrez: disable puppet on cache nodes before merging gerrit/440114 - T192555
15:39 ottomata: switching EventStreams service to be backed by main-eqiad - T185225
14:48 ejegg: disabled donations and refund queue consumers
14:45 ejegg: updated CiviCRM from 69091d8 f54f981
14:40 mutante: moving wikidata query dispatcher from terbium to mwmaint1001 - scheduled downtime - check turned into a WARN - disabling puppet on mwmaint1001, removing crons on terbium, waiting a couple minutes for them to finish, re-enabling puppet on mwmaint1001 (T192092 )
14:32 moritzm: (slow) initial rollout of debmonitor-client
13:47 zeljkof: EU SWAT finished
13:46 zfilipin@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/Wikibase: SWAT: Revert "Statement transclusion: when entity of unknown type in statement, display ID as string" (T195615) (duration: 01m 21s)
13:35 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set a few of namespace aliases on ruwikisource (T196719) (duration: 00m 59s)
13:11 volans@deploy1001: Finished deploy [debmonitor/deploy@476fd8b]: Release v0.1.4 (duration: 00m 19s)
13:10 volans@deploy1001: Started deploy [debmonitor/deploy@476fd8b]: Release v0.1.4
12:54 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
12:29 ema: cp3037: run update-ocsp-all
12:23 gehel: rolling restart of elasticsearch codfw for plugin upgrade - T194245
11:38 marostegui: Deploy schema change on db1067 T191316 T192926 T89737 T195193
11:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 00m 57s)
11:16 elukey: upgrade cassandra on aqs* to 2.2.6-wmf5
11:16 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 01s)
10:04 addshore: @terbium: for i in {2001..3000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
09:59 addshore: @terbium: for i in {1001..2000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
09:57 addshore: @terbium: for i in {1..1000}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki // T197222
09:15 elukey: add debmonitor term to analytics-in4 on cr1/cr2 eqiad
08:59 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes (duration: 01m 18s)
08:58 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@2680ba8]: Decrease the checkerJob delays recheck to 10 minutes
08:31 elukey: restart hadoop hdfs master nodes to pick up the new journal node settings
08:29 gehel: restart of elasticsearch / relforge for plugin updates
08:08 mutante: switch backend for dbtree.wikimedia.org away from terbium to mwmaint1001 (T192092 )
08:07 elukey: roll restart of hadoop journal nodes to pick up the new configuration (two more journal nodes added)
08:03 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings-labs.php: BETAONLY: senses for beta wikidatas (duration: 00m 58s)
07:59 moritzm: resuming rolling restart of cassandra on restbase2* to pick up OpenJDK security update
07:41 marostegui: Deploy schema change on db1114 T191316 T192926 T89737 T195193
07:41 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 00m 58s)
07:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 00m 58s)
05:47 mutante: LDAP - added user mepps to wmf group (T192472 )
05:15 marostegui: Deploy schema change on db1083 T191316 T192926 T89737 T195193
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 00m 58s)
05:10 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 01s)
05:03 marostegui: Deploy schema change on s4 primary master (db1068) T191316 T192926 T195193
03:06 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 14 03:06:19 UTC 2018 (duration 10m 30s)
02:55 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 14m 42s)
02:22 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 09m 41s)
23:15 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/432093/ (duration: 00m 58s)
23:07 maxsem@deploy1001: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/440178/ (duration: 01m 00s)
21:20 awight@deploy1001: Finished deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468 (duration: 72m 56s)
20:07 awight@deploy1001: Started deploy [ores/deploy@36037b6]: New badwords for ORES in English: T196468
19:26 dduvall@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.8 (duration: 00m 57s)
19:25 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.8
18:28 twentyafterfour: finished swat (28m overtime! :P)
18:22 twentyafterfour: ran namespaceDupes for urwiktionary
18:20 twentyafterfour@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: sync 5b47244 14ca2ba 2569a77 T196488 , T196744 , T196727 , T196763 (duration: 00m 57s)
17:55 twentyafterfour@deploy1001: Synchronized static/images/project-logos/: sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
17:53 twentyafterfour@deploy1001: Synchronized static/images/project-logos/bnwikivoyage-1.5x.png: static/images/project-logos/bnwikivoyage-2x.png static/images/project-logos/bnwikivoyage.png sync bnwikivoyage logos refs T196803 (duration: 00m 58s)
17:49 twentyafterfour@deploy1001: Synchronized wmf-config: Sync https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/436430/ for SWAT refs T184244 (duration: 01m 00s)
17:08 mutante: terbium - closing unusued screen sessions for all Amir users (2)
16:53 ppchelko@deploy1001: Finished deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082 (duration: 15m 44s)
16:38 ppchelko@deploy1001: Started deploy [restbase/deploy@f521e7e]: Add page_language to title_revision table T197082
16:31 XioNoX: moving mr1-eqiad interfaces to new router
16:14 mutante: rsyncing /home dirs from terbium to mwmaint1001, they will appear later in a subdir "home-terbium" like it was done for tin->deploy1001 (T192092 )
16:13 urandom: ALTERing Cassandra schema - T197082
16:11 moritzm: installing plexus-archiver security updates
16:07 moritzm: installing imagemagick security updates on trusty (Debian already fixed)
15:55 elukey: rolling restart of aqs on aqs100[4-9] to pick up the new config changes
15:25 jynus: stopping db1053 and db1059 in preparation for decomm T194634 T196606
15:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore original weight for db1076 (duration: 00m 58s)
14:19 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1076 (duration: 00m 57s)
14:15 anomie@deploy1001: Synchronized php-1.32.0-wmf.8/includes/Category.php: Backporting fix for T195397 (gerrit:440053 ) (duration: 01m 00s)
14:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1076 with low weight (duration: 00m 58s)
14:01 zeljkof: EU SWAT finished
14:00 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Implementing Patroller User Rights for azwiki (T196488) (duration: 00m 57s)
13:54 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: English aliases for extra namespaces on urwiktionary (T196614) (duration: 00m 58s)
13:48 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix wrong language in ur.wiktionary namespace (T196614) (duration: 00m 58s)
13:28 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@160206f]: (no justification provided) (duration: 04m 11s)
13:26 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make ProofreadPage operate on correct namespaces in pmswikisource (T197033) (duration: 00m 57s)
13:24 elukey@deploy1001: Started deploy [analytics/aqs/deploy@160206f]: (no justification provided)
13:12 zfilipin@deploy1001: Synchronized wmf-config/: SWAT: Enable FileImporter monolog channel in production (T195370) (duration: 01m 00s)
13:02 moritzm: rolling restart of cassandra in codfw to pick up OpenJDK security update
13:00 elukey@deploy1001: Finished deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213 (duration: 02m 38s)
12:57 elukey@deploy1001: Started deploy [analytics/aqs/deploy@84fab89]: Update AQS for T190213
12:46 elukey: restart mirror maker on kafka1012->1014 to pick up new openjdk-7 upgrades
12:28 elukey: rolling restart of kafka on kafka1012->23 for openjdk-7 upgrades
12:16 arturo: T196633 extend downtime for labcontrol1003
11:21 moritzm: installing perl security updates
10:59 marostegui: Deploy schema change on db1105:3311 T191316 T192926 T89737 T195193
10:59 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 00m 58s)
10:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 00m 59s)
10:10 akosiaris: upload apertium-apy_0.11.3-1+wmf1 to apt.wikimedia.org/jessie-wikimedia/main
09:00 marostegui: Restart mysql on dbstore1002 for maintenance
08:46 ema: depool cp1053 T165252
08:04 marostegui: Stop MySQL and reboot db1076 - T197063
08:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1076 for binlog change - T197063 (duration: 00m 57s)
07:59 oblivian@deploy1001: Synchronized wmf-config/ProductionServices.php: Remove unused redis shards from the jobqueue T197003 (duration: 00m 58s)
07:57 Pchelolo: restart cpjobqueue on scb1001 cause, it's lost all it's workers
07:55 marostegui: Deploy schema change on db1089 T191316 T192926 T89737 T195193
07:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 00m 58s)
07:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore s2 default read-only message (duration: 00m 57s)
07:41 marostegui: Stop MySQL on db1054 for socket update and binlog change
07:33 godog: start removing ms-be1036 from swift rings - T196873
06:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 00m 59s)
06:08 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 33s)
06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Remove read only from s2 - T194870 (duration: 00m 34s)
06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Set s2 on read-only for primary db master maintnance - T194870 (duration: 01m 08s)
06:00 marostegui: Starting s2 failover from db1054 to db1066 - T194870
05:11 marostegui: Starting topology changes in order to get ready for s2 failover - T194870
05:09 marostegui: Disable gtid on db1066
05:03 marostegui: Deploy schema change on dbstore1001:s1 T191316 T192926 T89737 T195193
03:14 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 13 03:14:53 UTC 2018 (duration 10m 19s)
03:04 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.8) (duration: 15m 45s)
02:31 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 12m 21s)
01:12 aaron@deploy1001: Synchronized wmf-config/mc.php: Add "memcached-mcrouter" to $wgObjectCaches as default for testwiki (duration: 00m 58s)
01:01 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 57s)
01:00 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/lbfactory/LBFactory.php: f83bad6
00:55 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/tests/phpunit/includes/db/LBFactoryTest.php: (no justification provided) (duration: 00m 58s)
00:53 aaron@deploy1001: Synchronized php-1.32.0-wmf.8/includes/libs/rdbms/lbfactory/LBFactory.php: c2df9668d13
00:46 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5
00:44 bawolff@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/CentralAuth/extension.json: Deploy I5f25c5
00:40 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/AntiSpoof/CentralAuthAntiSpoofHooks.php: Deploy I5f25c5
00:39 bawolff@deploy1001: Synchronized php-1.32.0-wmf.8/extensions/CentralAuth/extension.json: Deploy I5f25c5
00:37 bawolff@deploy1001: Synchronized wmf-config/CommonSettings.php: Deploy I5f25c5
23:47 niharika29@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Add sites to the wgCopyUploadsDomains whitelist of Wikimedia Commons T195270 , T195928 (duration: 00m 59s)
23:05 tzatziki: resetting passwords for compromised accounts (T197046 )
23:00 bblack: cp3043 - done, reimaged, in live service for cache_upload
22:59 bblack@neodymium: conftool action : set/pooled=yes; selector: name=cp3043.esams.wmnet
22:37 tzatziki: (from yesterday) resetting passwords for compromised accounts (T197046 )
22:24 twentyafterfour: phabricator: I scheduled a 24 hour downtime in icinga for the phd service, to give me time to work on this issue. See T196840
22:23 twentyafterfour: phabricator: taking phd offline to relieve the load on the m3 database cluster
21:46 bblack: cp3046 - restart varnish backend for mbox lag
21:40 bblack@neodymium: conftool action : set/pooled=no; selector: name=cp3043.esams.wmnet
21:39 bblack: cp3043 - starting process to move to reimage into cache_upload
19:16 dduvall@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.8
18:57 herron: restarted icinga service on einsteinium
18:42 dduvall@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache (duration: 39m 39s)
18:03 dduvall@deploy1001: Started scap: testwiki to php-1.32.0-wmf.8 and rebuild l10n cache
17:38 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 04s)
17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
17:37 ariel@deploy1001: Finished deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install (duration: 00m 07s)
17:37 ariel@deploy1001: Started deploy [dumps/dumps@038c8b3]: sync after snapshot1009 install
16:54 marxarelli: starting branch cut for 1.32.0-wmf.8
16:11 volans@deploy1001: Finished deploy [debmonitor/deploy@0eca14a]: Release v0.1.3 (duration: 00m 22s)
16:11 volans@deploy1001: Started deploy [debmonitor/deploy@0eca14a]: Release v0.1.3
15:40 bblack: cp3034 - nevermind, doing different approach later in the day, still pooled in text for now!
15:29 bblack: cp3043 switching from text to upload shortly, downtimed in icinga for 2h - https://gerrit.wikimedia.org/r/c/operations/puppet/+/439936
15:07 ema: cp3039: restart varnish-backend
14:38 addshore: file exporter importer slot done
14:38 addshore@deploy1001: Synchronized wmf-config/InitialiseSettings.php: FileImporter/Exporter Enable FileExporter/Importer on group0 wikis T195370 (duration: 00m 51s)
14:20 addshore@deploy1001: Synchronized wmf-config/CommonSettings.php: FileImporter/Exporter Allow setting of export target for FileExporter T195370 (duration: 00m 50s)
14:09 addshore@deploy1001: Finished scap: FileExporter backport - Pre deployment backport (extension not yet deployed) (duration: 30m 37s)
13:38 addshore@deploy1001: Started scap: FileExporter backport - Pre deployment backport (extension not yet deployed)
13:16 moritzm: installing openjdk-8 security updates on restbase-dev along with cassandra restarts
12:38 ema: cp3035: restart varnish-be, mbox lag
12:34 _joe_: repooling mw1230 after reimaging T196881
12:14 marostegui: Deploy schema change on db1099:3311 T191316 T192926 T89737 T195193
12:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 00m 52s)
12:11 marostegui: Deploy schema change on dbstore1002:s1 T191316 T192926 T89737 T195193
12:05 akosiaris@puppetmaster1001: conftool action : set/weight=10; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
11:47 moritzm: updated component/cassandra311 on apt.wikimedia.org to 3.11.2
10:26 jynus: setting expire_log_days on db1066 as 30
10:21 godog: bounce stuck rsyslog on lithium / wezen - T136312
09:41 vgutierrez: cp3037 has been depooled due to unknown hardware issues T196974
08:48 marostegui: Stop replication on db2094 to change triggers for archive table
08:36 volans: running puppet on failed hosts post small puppet outage and puppetdb reboot
08:35 akosiaris: rebalance ganeti codfw cluster
08:35 ema@neodymium: conftool action : set/pooled=no; selector: name=cp3037.esams.wmnet
08:33 akosiaris: reboot puppetdb1001 for spec-ctrl enable. Bundling it with a minor puppet outage to only have a torrent of harmless puppet failures once
08:15 akosiaris: ganeti2002 reboot for microcode update
08:04 akosiaris: ganeti2006 reboot for microcode update
08:03 marostegui: Deploy schema change on s1 codfw primary master (db2048) with replication, this will generate lag on codfw T191316 T192926 T89737 T195193
07:53 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 00m 50s)
07:43 akosiaris: ganeti2007 reboot for microcode update
07:41 akosiaris: ganeti2003 reboot for microcode update
07:31 mutante: closing idle screen session on tin (about to be decomed, dont use anymore)
06:37 marostegui: Deploy schema change on db1121 with replication, this will generate lag on labsdb:s4 T191316 T192926 T89737 T195193
06:37 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 for alter table (duration: 00m 50s)
06:31 marostegui: Stop replication on db1095, db1102, db1125 to change triggers - T192926
06:30 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 00m 51s)
05:09 marostegui: Deploy schema change on db1091 T191316 T192926 T89737 T195193
05:09 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 00m 52s)
02:45 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 12 02:45:53 UTC 2018 (duration 10m 18s)
02:35 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 14m 10s)
00:35 legoktm: remove non-deployers from wmf-deployment Gerrit group (T196959 )
00:16 ejegg: updated CiviCRM from 69091d8b5f f54f9810ab
23:45 ebernhardson@deploy1001: Synchronized wmf-config/CirrusSearch-common.php: SWAT: Lower CirrusSearch delayed job drop timeout (duration: 00m 50s)
23:26 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Tune CirrusSearch slow logging (duration: 00m 48s)
23:18 ebernhardson@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Promote Cirrus MLR models from AB test to prod (duration: 00m 51s)
23:02 twentyafterfour: phabricator: restarting apache2 on phab1001 to free up apache workers
22:17 awight@deploy1001: Finished deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models (duration: 33m 58s)
21:43 awight@deploy1001: Started deploy [ores/deploy@6ee8775]: ORES: bswiki, euwiki, srwiki models
20:57 arlolra: Updated Parsoid to 06b74d2 T191843 )
20:49 arlolra@deploy1001: Finished deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2
20:32 arlolra@deploy1001: Started deploy [parsoid/deploy@97cdab8]: Updating Parsoid to 06b74d2
20:27 otto@deploy1001: Finished deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418 (duration: 09m 52s)
20:17 otto@deploy1001: Started deploy [eventstreams/deploy@6b013f9]: Enable composite stream and timestamp since param - T196009 , T187418
20:02 ottomata: bouncing varnishkafka-webrequest on cp3039,cp3047,cp2007,cp3010
19:56 ottomata: bouncing varnishkafka on cp3032
19:29 aaron@deploy1001: Synchronized php-1.32.0-wmf.7/includes/libs/rdbms/ChronologyProtector.php: 11e596776f940
18:50 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Restore full config (duration: 00m 16s)
18:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Restore full config
18:15 urandom: convert timeline indices to time-windowed compaction - T196024
17:59 twentyafterfour: phabricator: taking phd offline while I clear out the queue backlog (downtime is logged in icinga) see T196840
17:51 twentyafterfour: phabricator: rebuilding git parent caches
17:33 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 13m 38s)
17:27 twentyafterfour: phabricator: restarting phd for D1067
17:25 twentyafterfour: Phabricator: deploying hotfix (D1067) refs T196840 T196860 T196855
17:20 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
17:19 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (duration: 00m 03s)
17:19 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater
17:17 gehel@deploy1001: Finished deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only) (duration: 00m 26s)
17:16 gehel@deploy1001: Started deploy [wdqs/wdqs@37f6f32]: new version of wdqs GUI and updater (wdqs1009 only)
17:00 akosiaris: ganeti2008 reboot for microcode update
16:42 marostegui: Set disk 32:1 offline on db1065 to get a new one - T196806
16:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 01m 32s)
16:00 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
15:44 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 33s)
15:43 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
15:43 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 50s)
15:33 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 00m 22s)
15:33 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
15:31 marostegui: Set offline disk 32:3 on db1063 - T196806
15:31 marostegui: Set offline disk 32:1 on db1065 - T196806
15:02 akosiaris@deploy1001: Finished deploy [proton/deploy@97ec4bf]: (no justification provided) (duration: 35m 33s)
14:55 marostegui: Deploy schema change on db1084 T191316 T192926 T89737 T195193
14:55 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 00m 50s)
14:53 akosiaris: reboot bohrium for kernel upgrades and spec-ctrl enabling. Manually stopped mysql behorehand
14:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 00m 50s)
14:35 akosiaris: reboot mx1001, poolcounter1001 for kernel upgrades and spec-ctrl enabling
14:26 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
14:25 godog: upload scap 3.8.2-1 - T196710
14:21 hoo: Updated operations/dumps/dcat (536bd5b..559dee3) on snapshot1008
13:58 marostegui: Deploy schema change on db1081 T191316 T192926 T89737 T195193
13:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 00m 48s)
13:56 otto@deploy1001: Finished deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 00m 04s)
13:56 otto@deploy1001: Started deploy [eventlogging/analytics@08a1dff]: Producing events with kafka timestamp set to event time - T196407
13:54 otto@deploy1001: Finished deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407 (duration: 01m 55s)
13:52 otto@deploy1001: Started deploy [eventlogging/eventbus@08a1dff]: Producing events with kafka timestamp set to event time - T196407
13:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 00m 50s)
13:47 hashar: European SWAT completed
13:35 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 49s)
13:29 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Fix wgMetaNamespace for pswikivoyage - T196837 (duration: 00m 50s)
13:21 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 00m 56s)
13:20 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
13:20 hashar@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Use uploaded HD logo for bewikiquote - T196134 (duration: 00m 50s)
13:17 hashar@deploy1001: Synchronized static/images/project-logos: Change logo files for bewikiquote - T196134 (duration: 00m 50s)
13:13 hashar@deploy1001: Synchronized static/images/project-logos: Revert "Change bewikiquote logo" - T196134 (duration: 00m 51s)
13:01 volans@deploy1001: Finished deploy [debmonitor/deploy@81d7333]: Release v0.1.2 (duration: 07m 16s)
12:54 volans@deploy1001: Started deploy [debmonitor/deploy@81d7333]: Release v0.1.2
11:45 arturo: T196633 downtime labcontrol100[3,4] due to unexpected puppet errors on installation of keystone
11:38 arturo: T196633 deploy keystone to labcontrol100[3,4].wikimedia.org. Dormant daemon, no DB yet
11:30 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies (duration: 00m 42s)
11:30 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@b5396cd]: Tune cirrus jobs concurrencies
11:23 marostegui: Deploy schema change on db1103:3314 T191316 T192926 T89737 T195193
11:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 50s)
11:20 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 00m 51s)
10:52 mutante: phab1002 - editing cached scap config /srv/deployment/phabricator/deployment-cache/.config to replace tin.eqiad with deploy1001.eqiad deployment server, run puppet. other options: run scap with --refresh-config, delet cached .config file (T196019 ) (T175288 )
10:29 _joe_: depooling permantently mw1230 for disk replacement, T196881
10:11 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 50s)
10:10 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
09:07 marostegui: Deploy schema change on db1097:3314 T191316 T192926 T89737 T195193
09:06 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 00m 52s)
08:32 moritzm: installing gnupg2 security updates
08:27 gehel: restart elastic1020 to enable G1 GC - T156137
08:25 moritzm: installing gnupg1 security updates
07:52 moritzm: installing gnupg security updates
07:37 marostegui: Deploy schema change on dbstore1002:s4 T191316 T192926 T89737 T195193
07:29 moritzm: installing openjdk-7 security updates
06:39 elukey: restart pdfrender on scb1002
06:14 marostegui: Deploy schema change on s4 codfw master (db2051) this will generate lag on codfw - T191316 T192926 T89737 T195193
06:10 marostegui: Stop replication on db2095 to update triggers - T192926
05:32 marostegui: Restart mysql on codfw sanitariums (db1095, db1102, db1124, db1125) to pick up new replication filters - T196748
05:28 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool pc2005 - T196339 (duration: 00m 52s)
05:25 marostegui: Restart mysql on codfw sanitariums (db2094, db2095) to pick up new replication filters - T196748
05:17 marostegui: Stop MySQL and reboot pc2005 for intel-microcode update and final HW check - T196339
05:15 marostegui: Deploy schema change on s6 primary master (db1061) - T191316 T192926 T195193
02:42 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 11 02:42:42 UTC 2018 (duration 10m 16s)
02:32 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 13m 06s)
12:30 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: CSP in report mode for group0 (duration: 00m 55s)
12:01 bawolff: disable some botpasswords (T194204 )
11:08 _joe_: restarting gerrit on cobalt as the web interface is unresponsive
22:41 no_justification: gerrit: restarting
19:36 urandom: upgrade Cassandra to 3.11.2, restbase1015 & restbase1018 - T178905
18:49 urandom: upgrade Cassandra to 3.11.2, restbase1009 & restbase1014 - T178905
18:02 urandom: upgrade Cassandra to 3.11.2, restbase1017 & restbase1013 - T178905
17:33 no_justification: gerrit: up mostly, but will see some errors about "wip" label for a bit until reindexing completes.
17:28 cmjohnson: powering flerovium down to move to a different space in the rack
17:14 demon@deploy1001: Finished deploy [gerrit/gerrit@7324140]: 2.15.2 (duration: 00m 11s)
17:14 demon@deploy1001: Started deploy [gerrit/gerrit@7324140]: 2.15.2
17:12 no_justification: gerrit: taking offline for 2.14 -> 2.15 upgrade
15:35 urandom: upgrade Cassandra to 3.11.2, restbase1012 - T178905
15:26 marostegui: Restart MySQL on codfw sanitarium hosts db2094, db2095 - https://phabricator.wikimedia.org/T196748
15:00 urandom: upgrade Cassandra to 3.11.2, restbase1008 - T178905
14:17 urandom: upgrade Cassandra to 3.11.2, restbase1011 & restbase1016 - T178905
12:14 twentyafterfour: running phabricator public_task_dump script manually to confirm that it's working as expected.
09:31 arturo: merging https://gerrit.wikimedia.org/r/#/c/436337/ for the eqia1 openstack deployment (labcontrol1003/labcontrol1004)
09:19 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
09:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Unify and update sanitarium comments - T190704 (duration: 00m 50s)
09:06 akosiaris@deploy1001: Started deploy [proton/deploy@97ec4bf]: (no justification provided)
08:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 after reboot (duration: 00m 50s)
08:18 marostegui: Stop MySQL and reboot db1066 for intel-microcode install - T194870
08:18 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 for reboot (duration: 00m 50s)
07:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1113:3316 after alter table (duration: 00m 51s)
07:42 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Restore db1091 original weight (duration: 00m 50s)
07:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase weight for db1091 (duration: 00m 50s)
07:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Increse weight for db1091 (duration: 00m 50s)
06:47 gilles@deploy1001: Synchronized wmf-config/InitialiseSettings.php: T196630 Remove unneeded description from performance survey definition (duration: 00m 52s)
06:44 elukey: bounce kafka mirror maker main-eqiad-to-main-codfw (kafka200*) due to errors in the logs (also lag metrics not displaying)
06:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after reimage (duration: 00m 50s)
05:34 marostegui: Deploy sanitarium events on db1124 - T190704
05:23 marostegui: Stop MySQL on db1091 for reimage
05:22 marostegui: Deploy schema change on db1113:3316 - T191316 T192926 T195193 T89737
05:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1113:3316 for alter table (duration: 00m 52s)
02:12 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
02:11 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
23:58 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2) (duration: 00m 14s)
23:58 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (x2)
23:57 demon@deploy1001: Finished deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state (duration: 00m 06s)
23:57 demon@deploy1001: Started deploy [gerrit/gerrit@a07d943]: No-op of current deployed version, want to sync repo state
21:49 mobrovac@deploy1001: Finished deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748 (duration: 02m 19s)
21:47 mobrovac@deploy1001: Started deploy [proton/deploy@97ec4bf]: Initial deploy to production - T186748
20:37 herron: ircecho restarted
19:47 urandom: rolling Cassandra restart, restbase2005, restbase2006, restbase2012 -- T178905
19:36 XenoRyet: updated payments-wiki config from 3197c729ee b11d120362
19:29 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.7
19:16 herron: stopped ircecho
18:28 urandom: rolling Cassandra restart, restbase2004, restbase2008, restbase2011 -- T178905
18:20 urandom: restarting Cassandra, restbase2003-c -- T178905
18:13 subbu: Updated Parsoid (T183706 , T192726 , T194879 , T196357 , T196360 , T43716 )
18:10 ssastry@deploy1001: Finished deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7
17:59 ssastry@deploy1001: Started deploy [parsoid/deploy@2f80639]: Updating Parsoid to 7819c9e7
17:21 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2) (duration: 04m 34s)
17:16 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (take 2)
17:16 awight@deploy1001: Finished deploy [ores/deploy@65ce165]: New home page for ORES; T196580 (duration: 03m 19s)
17:16 reedy@deploy1001: Synchronized wmf-config/interwiki-labs.php: labs! (duration: 00m 57s)
17:13 awight@deploy1001: Started deploy [ores/deploy@65ce165]: New home page for ORES; T196580
16:45 urandom: rolling Cassandra restart, restbase2001, restbase2002, restbase2007 - T178905
16:21 urandom: rolling Cassandra restart, restbase1007 - T178905
15:59 hoo: Finished running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
{{safesubst:SAL entry|1=15:38 urandom: upgrade Cassandra to 3.11.2, restbase1010-{a,b,c} - T178905}}
15:30 hoo: Running "foreachwikiindblist wikidataclient extensions/Wikibase/lib/maintenance/populateSitesTable.php --force-protocol https" T196360 T183706 T195014 T196357
15:23 hoo: Emptied out the sites and site_identifiers tables on pswikivoyage, pmswikisource, bnwikivoyage and sahwikiquote for T122520 ,
15:20 hoo@deploy1001: Synchronized dblists/wikidataclient.dblist: Enable WikidataClient on sahwikiquote and pswikivoyage - T183706 , T196360 (duration: 00m 57s)
15:08 marostegui: Sanitize wikis on db1124 (current sanitarium for s3) - T196362 T196358 T196359 T195008 T193187
15:04 reedy@deploy1001: Synchronized wmf-config/interwiki.php: (no justification provided) (duration: 00m 56s)
14:56 ottomata: beginning rolling restarts of all cluster kafka brokers to apply log.message.timestamp.type=CreateTime - T196407
14:53 reedy@deploy1001: rebuilt and synchronized wikiversions files: new wikis
14:53 marostegui: Sanitize wikis on db1095 (old sanitarium) - T196362 T196358 T196359 T195008 T193187
14:53 reedy@deploy1001: Synchronized wmf-config/InitialiseSettings.php: new wikis (duration: 00m 57s)
14:50 reedy@deploy1001: Synchronized static/images/: new wikis (duration: 00m 57s)
14:48 reedy@deploy1001: Synchronized multiversion/MWMultiVersion.php: idwikimedia (duration: 00m 57s)
14:47 reedy@deploy1001: Synchronized dblists/: 5 new wikis (duration: 00m 55s)
14:45 Reedy: created new wiki databases T183706 T192726 T194879 T196357 T196360
14:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1088 (duration: 00m 57s)
14:11 reedy@deploy1001: Synchronized wmf-config/jobqueue-labs.php: labs (duration: 00m 56s)
14:10 reedy@deploy1001: Synchronized wmf-config/LabsServices.php: labs (duration: 00m 57s)
14:02 reedy@deploy1001: Synchronized docroot/noc/: Add some more symlinked configs (duration: 00m 57s)
13:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1066 (duration: 00m 57s)
13:31 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 56s)
13:29 reedy@deploy1001: Synchronized php-1.32.0-wmf.6/extensions/WikimediaMaintenance/addWiki.php: add all the wikis (duration: 00m 58s)
13:26 reedy@deploy1001: Synchronized dblists/all-labs.dblist: labs (duration: 00m 54s)
13:24 reedy@deploy1001: Synchronized wikiversions-labs.json: labs (duration: 00m 57s)
13:22 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1066 (duration: 00m 56s)
13:17 zeljkof: EU SWAT finished
13:16 zfilipin@deploy1001: Synchronized wmf-config/CommonSettings.php: SWAT: When using the FileExporter set it as BeatFeature by default (T195370) (duration: 00m 56s)
13:10 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add FileExporter to BetaFeaturesWhiteList (T195370) (duration: 00m 57s)
12:59 marostegui: Deploy schema change on db1088 - T191316 T192926 T195193 T89737
12:58 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1088 for alter table (duration: 00m 57s)
12:54 jynus: stop, clone and reimage db1091
12:52 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 57s)
12:46 mutante: planet1001/puppetmaster: revoke old cert, sign new cert request, initial puppet run, reinstalled, will turn service active-active again once done (T168490 )
12:26 mutante: planet1001 - schedule downtime, boot to PXE, reinstall with stretch (ganeti) (T168490 )
12:15 moritzm: repooled mw1280 after hardware maintenance (T195734 )
11:31 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1085 after alter table (duration: 00m 57s)
10:29 moritzm: installing openssl security updates
08:50 marostegui: Deploy schema change on db1085 with replication, this will generate lag on labsdb for s6 section - T191316 T192926 T195193 T89737
08:49 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1085 for alter table (duration: 00m 56s)
08:49 akosiaris: slow reboot of all ganeti eqiad VMs (except bohrium, puppetdb1001, poolcounter1001, mx1001) for kernel upgrades and picking up spec-ctrl cpu flag
08:38 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1093 after alter table (duration: 00m 55s)
08:34 marostegui: Stop replication on db1102:3316 and db1125:3316 to update triggers for archive table - T192926
08:15 jynus: running ANALYZE on db2091 T196526
07:46 marostegui: Deploy schema change on db1093 - T191316 T192926 T195193 T89737
07:46 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1093 for alter table (duration: 00m 56s)
07:40 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1098:3316 after alter table (duration: 00m 57s)
07:09 moritzm: installing git security updates on trusty (Debian already fixed)
06:55 jynus: relaxing write consistency on db2048 due to ongoing maintenance (sync_binlog,flush_log)
06:26 marostegui: Deploy sanitarium events on db1125 - T190704
06:24 jynus: phabricator maintenance finished
06:10 jynus: start database maintenance on phabricator- brief interruptions could happen
05:15 marostegui: Deploy event_sanitarium on codfw sanitariums - T190704
05:12 marostegui: Deploy schema change on db1098:3316 - T191316 T192926 T195193 T89737
05:11 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1098:3316 for alter table (duration: 00m 56s)
05:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1096:3316 after alter table (duration: 00m 58s)
04:01 eileen: civicrm revision changed from 1aaa798e9b 69091d8b5f c60375be8d
03:05 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Thu Jun 7 03:05:05 UTC 2018 (duration 10m 19s)
02:54 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 07m 02s)
02:30 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 11m 24s)
02:25 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: log
02:00 paravoid: starting exim4 and reenabling puppet on mx1001, due to T196598
01:12 ejegg: disabled fundraising mailing statistics fetch jobs
01:00 ejegg: updated CiviCRM from 7a06a6a387 1aaa798e9b
00:51 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: (no justification provided) (duration: 01m 02s)
00:50 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@a07af40]: (no justification provided)
00:33 ejegg: restarted fundraising jobs
00:25 legoktm@deploy1001: Finished scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875 , gerrit:437814 ) (duration: 64m 01s)
{{safesubst:SAL entry|1=23:51 urandom: upgrade Cassandra to 3.11.2, restbase2012-{a,b,c} - T178905}}
23:49 eileen: civicrm revision changed from ce960c0642 7a06a6a387 b7bc5dc31f
23:43 eileen: civicrm revision changed from ac13914d03 ce960c0642 b7bc5dc31f
{{safesubst:SAL entry|1=23:23 urandom: upgrade Cassandra to 3.11.2, restbase2009-{a,b,c} - T178905}}
23:21 legoktm@deploy1001: Started scap: Preference for responsive MonoBook, plus set mobile width cutoff to 550px (gerrit:437875 , gerrit:437814 )
23:19 eileen: civicrm revision changed from 0b97f1f5b2 ac13914d03
23:07 cwd: disabled process-control for civi update
22:22 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
22:16 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@0346959]: Update mobileapps to 5ea008c
20:47 ebernhardson: sighup logstash on logstash100[789] to reload config for gerrit.wikimedia.org/r/437657
{{safesubst:SAL entry|1=20:42 urandom: upgrade Cassandra to 3.11.2, restbase2005-{a,b,c} - T178905}}
20:19 bearND: rolled back mobileapps deploy
20:18 bsitzmann@deploy1001: Finished deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 T196402 T195948 ) (duration: 08m 37s)
{{safesubst:SAL entry|1=20:17 urandom: upgrade Cassandra to 3.11.2, restbase2011-{a,b,c} - T178905}}
20:10 bsitzmann@deploy1001: Started deploy [mobileapps/deploy@a07af40]: Update mobileapps to 3bf9be5 T196402 T195948 )
{{safesubst:SAL entry|1=19:38 urandom: upgrade Cassandra to 3.11.2, restbase2008-{a,b,c} - T178905}}
19:19 thcipriani@deploy1001: Synchronized php: group1 wikis to 1.32.0-wmf.7 (duration: 00m 56s)
19:18 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.7
18:55 herron: stopped exim on mx1001 in prep for upgrade to stretch
{{safesubst:SAL entry|1=18:54 urandom: upgrade Cassandra to 3.11.2, restbase2004-{a,b,c} - T178905}}
18:38 andrewbogott: rebooting labvirt1019, 1020
18:36 andrewbogott: rebooting labvirt1018, 1021, 1022
18:29 andrewbogott: rebooting labvirt1017
18:23 andrewbogott: rebooting labvirt1016
18:21 andrewbogott: rebooting labvirt1015
{{safesubst:SAL entry|1=18:19 urandom: upgrade Cassandra to 3.11.2, restbase2003-{a,b,c} - T178905}}
18:14 andrewbogott: rebooting labvirt1013
18:06 andrewbogott: rebooting labvirt1012
17:59 andrewbogott: rebooting labvirt1011
{{safesubst:SAL entry|1=17:59 urandom: upgrade Cassandra to 3.11.2, restbase2010-{a,b,c} - T178905}}
17:44 andrewbogott: rebooting labvirt1010
{{safesubst:SAL entry|1=17:01 urandom: upgrade Cassandra to 3.11.2, restbase2007-{a,b,c} - T178905}}
16:59 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 fully (duration: 00m 56s)
16:51 andrewbogott: rebooting labvirt1008
16:33 andrewbogott: rebooting labvirt1007
16:25 jynus: stop mysql @ db1051 in preparation for decom
16:25 andrewbogott: rebooting labvirt1006
16:16 andrewbogott: rebooting labvirt1005
16:09 andrewbogott: rebooting labvirt1004
16:05 XioNoX: lvs1002 repooled
16:04 andrewbogott: rebooting labvirt1002
15:57 twentyafterfour: reloading apache on phab1001 to free up some resources
15:56 andrewbogott: rebooting labvirt1014
{{safesubst:SAL entry|1=15:54 urandom: upgrade Cassandra to 3.11.2, restbase2001-{a,b,c} - T178905}}
15:51 XioNoX: disable pybal on lvs1002 - T187962
15:39 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies (duration: 00m 40s)
15:38 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@402d729]: Adjust the cirrus concurrencies
15:32 _joe_: cross enabling videoscalers,jobrunners in their respective pools
{{safesubst:SAL entry|1=15:27 urandom: upgrade Cassandra to 3.11.2, restbase2001-{b,c} - T178905}}
15:26 XioNoX: stop pybal on lvs1001
15:26 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336 (duration: 25m 37s)
15:13 _joe_: adding jobrunners, videoscalers to both pools with equal weight in codfw
15:06 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,dc=codfw,service=nginx
{{safesubst:SAL entry|1=15:06 urandom: upgrade Cassandra to 3.11.2, restbase1007-{b,c} - T178905}}
15:01 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES: new draft topic model; T176336
14:39 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw22(4[1-5]|5[3-8]|6[1-9]).*
14:35 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,dc=codfw,service=nginx,name=mw21(5[3-9]|6).*
14:22 andrewbogott: rebooting labvirt1009
14:21 jynus: setting m2 on read write
14:18 jynus: setting gerrit on read only
14:14 jynus: starting s2-master switchover from db1051 to db1065
14:06 andrewbogott: rebooting labvirt1003
13:58 jynus: disabling puppet on db1051, db1065
13:35 marostegui: Deploy schema change on db1096:3316 - T191316 T192926 T195193 T89737
13:35 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1096:3316 for alter table (duration: 00m 57s)
13:06 moritzm: installing elfutils security updates
13:01 akosiaris: starting slow rolling restart of all VMs on ganeti01.svc.codfw.wmnet
12:59 akosiaris: add +spec_ctrl to ganeti01.svc.codfw.wmnet cluster default cpu_type
12:50 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1084 with low load (duration: 00m 56s)
12:24 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs to EventBus for all wikis, file 2/2 - T189137 (duration: 00m 56s)
12:23 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327 (duration: 00m 47s)
12:23 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch CirrusSearch jobs to EventBus for all wikis - T189137 (duration: 00m 57s)
12:22 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@c8d62da]: Enable cirrus for everything T190327
11:38 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Increase s4 weight for db1097 and db1103 (duration: 00m 56s)
11:35 jynus: stop and reimage db1084
11:27 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1084 (duration: 00m 58s)
10:19 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 03m 44s)
10:16 marostegui: Deploy schema change on dbstore1002:s6 - T191316 T192926 T195193 T89737
10:15 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ORES canary deployment to ores2002.codfw.wmnet; T176336
10:15 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336 (duration: 00m 06s)
10:15 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: ORES canary deployment to ores2002.codfw.wmnet; T176336
08:29 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011 - T190704
08:25 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 56s)
08:24 addshore@deploy1001: Synchronized wmf-config/Wikibase.php: wikidatawiki dispatching: dispatchMaxTime 720 (4 dispatchers at once) (duration: 00m 56s)
08:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 57s)
07:48 marostegui: Stop replication on all sanitarium masters to move labsdb1011 - T190704
07:30 marostegui: Stop MySQL on labsdb1011 to install intel-microcode and reboot
06:32 marostegui: Deploy schema change on s6 codfw master (db2039), this will generate lag on s6 codfw - T191316 T192926 T195193 T89737
06:14 marostegui: Stop slave on db2095:3316 to rebuild archive_insert and archive_update triggers - T192926
06:14 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 , take 2 (duration: 07m 13s)
06:06 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 , take 2
06:05 ppchelko@deploy1001: Finished deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402 (duration: 11m 45s)
06:04 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 01m 09s)
05:53 ppchelko@deploy1001: Started deploy [restbase/deploy@baa70b7]: Public release of feed availability endpoint T196402
05:46 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - T190704
05:31 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
05:25 marostegui: Restart MySQL on labsdb1010
05:24 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - T190704
05:17 marostegui: Deploy schema change on db1070 s5 primary master - T191316 T192926 T195193
04:44 kartik@deploy1001: Finished deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 T196462 ) (duration: 03m 06s)
04:41 kartik@deploy1001: Started deploy [cxserver/deploy@8ce20ba]: Update cxserver to 391d7b6 T196462 )
03:10 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Wed Jun 6 03:10:14 UTC 2018 (duration 10m 15s)
02:59 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.7) (duration: 15m 31s)
02:27 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 23s)
01:07 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/composer.json: I13dbdba2b9d T196496 (duration: 00m 57s)
01:06 krinkle@deploy1001: Synchronized php-1.32.0-wmf.7/vendor/: I5a5d7d T196496 (duration: 01m 35s)
00:15 reedy@deploy1001: Synchronized php-1.32.0-wmf.7/extensions/WikimediaMessages/: respect watchlist preference feature flag (duration: 00m 58s)
23:54 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
23:42 reedy@deploy1001: Synchronized wmf-config/: Config! (duration: 00m 57s)
23:41 reedy@deploy1001: Synchronized rpc/: code updates (duration: 00m 56s)
23:26 reedy@deploy1001: Synchronized multiversion/submodules.json: minus unicodeconverter (duration: 00m 56s)
23:24 reedy@deploy1001: Synchronized wmf-config/: bye to UnicodeConverter (duration: 00m 57s)
23:21 reedy@deploy1001: Synchronized wmf-config/: page creation log on Beta Labs (duration: 00m 56s)
23:07 reedy@deploy1001: Synchronized wmf-config/: Updates (duration: 00m 58s)
23:05 reedy@deploy1001: Synchronized composer.json: minus x! (duration: 00m 56s)
23:04 reedy@deploy1001: Synchronized rpc: 644 (duration: 00m 56s)
23:01 ebernhardson: restore ferm on elastic1018 and logstash1009
22:57 ebernhardson: temporarily disable ferm on elastic1018 and logstash1007 to test theory
22:57 ebernhardson: temporarily disable ferm on elastic1018 to test theory
22:50 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Simplify PHP_SAPI conditionals (duration: 00m 57s)
22:20 maxsem@deploy1001: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/437638/ (duration: 00m 57s)
21:41 thcipriani@deploy1001: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.7
21:29 thcipriani@deploy1001: Finished scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache (duration: 65m 25s)
20:23 thcipriani@deploy1001: Started scap: testwiki to php-1.32.0-wmf.7 and rebuild l10n cache
20:03 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7
19:57 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@7ecc3b6]: Update mobileapps to 66727b7
19:35 urandom: upgrade Cassandra to 3.11.2, restbase1007-a (canary) - T178905
17:45 urandom: upgrade Cassandra to 3.11.2, restbase2001 (canary) - T178905
17:00 urandom: upgrade Cassandra to 3.11.2, restbase-dev1006 - T178905
16:58 Trey314159: reindexing Slovak wikis on elastic@eqiad (T191545 )
16:56 urandom: upgrade Cassandra to 3.11.2, restbase-dev1005 - T178905
16:10 reedy@deploy1001: Synchronized wmf-config/CommonSettings.php: Remove if file_exists for /etc/wikimedia-scaler (duration: 00m 51s)
16:07 thcipriani: starting branch cut for 1.32.0-wmf.7
16:05 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 fully (duration: 00m 49s)
15:58 urandom: upgrade Cassandra to 3.11.2, restbase-dev1004 - T178905
15:42 Trey314159: reindexing Slovak wikis on elastic@codfw (T191545 )
15:41 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 fully (duration: 00m 49s)
15:30 bstorm_: reboot labsdb1005 for upgrades T195313
15:24 marostegui: Stop MySQL on labsdb1005
14:55 marostegui: Downtime labsdb1005 and labsdb1004 for maintenance on labsdb1005 - T195313
14:52 urandom: restarting restbase-dev1004 - T178905
14:51 moritzm: installing fixed kernels/microcode for spectre v2 on labvirt*
14:42 jynus: reenabling puppet on all databases
14:09 jynus: disabling puppet on eqiad dbs for 435751 gerrit deploy
13:55 Amir1: ladsgroup@terbium:~$ mwscript deleteAutoPatrolLogs.php --wiki=commonswiki --sleep 2 --check-old
13:26 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 03m 15s)
13:24 zeljkof: EU SWAT finished
13:24 zfilipin@deploy1001: Synchronized dblists/mobilemainpagelegacy.dblist: SWAT: Remove ruwiki from MFSpecialCaseMainPage (T196223) (duration: 00m 51s)
13:22 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
13:22 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 08m 00s)
13:14 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
13:13 mobrovac@deploy1001: Finished deploy [restbase/deploy@a39743d]: Fix transform monitoring tests (duration: 18m 02s)
12:55 mobrovac@deploy1001: Started deploy [restbase/deploy@a39743d]: Fix transform monitoring tests
12:13 kartik@deploy1001: Finished deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671
12:12 kartik@deploy1001: Started deploy [cxserver/deploy@0a350c3]: Update cxserver to 7fb7671
12:06 XioNoX: disable graceful-switchover on dual-RE routers
11:35 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1081 with low load (duration: 00m 49s)
11:30 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120) on mw* hosts
11:21 mobrovac@deploy1001: Synchronized wmf-config/InitialiseSettings.php: Switch CirrusSearch jobs for all wikis except wp, wd, commons - T189137 (duration: 00m 51s)
11:21 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327 (duration: 00m 41s)
11:21 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@aa5e94b]: Enable cirrus jobs in kafka for everything except wikipedia, wikidata and commons T190327
11:07 elukey: manually set net.netfilter.nf_conntrack_tcp_timeout_time_wait to 65 (was 120)
10:51 jynus: stop db1081 for reimage
10:45 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1081 (duration: 00m 50s)
10:38 akosiaris: reboot acrab for spec_ctrl CPU flag addition
10:20 akosiaris: reboot acrab for kernel upgrade and qemu upgrade
10:18 akosiaris: upgrade qemu to 1:2.8+dfsg-6+deb9u4 on ganeti01.svc.codfw.wmnet
10:14 moritzm: installing batik security updates
10:08 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1080 with low load (duration: 00m 50s)
09:54 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1110 after alter table (duration: 00m 50s)
09:22 jynus: stop db1080 for reimage
09:09 jynus@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1080 (duration: 00m 49s)
09:01 marostegui: Deploy schema change on db1110 - T191316 T192926 T89737 T195193
09:01 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1110 for alter table (duration: 00m 51s)
08:52 mobrovac@deploy1001: Synchronized wmf-config/jobqueue.php: Switch video scaling jobs to EventBus - T190327 (duration: 00m 52s)
08:52 ppchelko@deploy1001: Finished deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327 (duration: 00m 49s)
08:51 ppchelko@deploy1001: Started deploy [cpjobqueue/deploy@63b30a6]: Enable videoscaler jobs in kafka T190327
08:16 ema: libvmod-re2 1.3.1-1 uploaded to apt.w.o T196355
08:10 gehel: rebooting elastic10(41|43) for plugin update - T193734
06:42 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059 after reimage (duration: 00m 50s)
05:52 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1100 after alter table (duration: 00m 50s)
05:44 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Depool db2059 for reimage (duration: 00m 51s)
05:14 marostegui: Deploy schema change on db1100 - T191316 T192926 T89737 T195193
05:14 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1100 for alter table (duration: 00m 51s)
03:25 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
03:19 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
03:12 pnorman@deploy1001: Finished deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
03:07 pnorman@deploy1001: Started deploy [tilerator/deploy@9e40702] (cleartables): Re-try 074d01a
02:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): enable v3view (duration: 00m 06s)
02:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
02:49 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): enable v3view
02:45 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources (duration: 00m 26s)
02:45 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable more sources
02:43 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels (duration: 02m 22s)
02:41 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable style without labels
02:30 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Tue Jun 5 02:30:57 UTC 2018 (duration 10m 15s)
02:20 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 08m 21s)
00:08 thcipriani: restarting jenkins to finalize updates
23:57 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable page creation log on Test Wikipedia" T196400 (duration: 00m 49s)
23:50 thcipriani@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable page creation log on Test Wikipedia T196400 (duration: 00m 50s)
23:21 smalyshev@deploy1001: Finished deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325 (duration: 09m 39s)
23:11 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
22:57 smalyshev@deploy1001: Started deploy [wdqs/wdqs@825863f]: Potential mitigation for T194325
22:35 pnorman@deploy1001: Finished deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters (duration: 00m 21s)
22:35 pnorman@deploy1001: Started deploy [kartotherian/deploy@a588bf4] (cleartables): Deploy var name parameters
21:08 pnorman@deploy1001: Finished deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test (duration: 00m 21s)
21:07 pnorman@deploy1001: Started deploy [kartotherian/deploy@d8dcba3] (cleartables): Redeploy kartotherian to test
20:58 awight@deploy1001: Finished deploy [ores/deploy@bf182e2]: roll back ores2001 (duration: 01m 11s)
20:57 awight@deploy1001: Started deploy [ores/deploy@bf182e2]: roll back ores2001
20:50 awight@deploy1001: Finished deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision) (duration: 01m 47s)
20:48 awight@deploy1001: Started deploy [ores/deploy@65e979f]: ores2001 canary of drafttopic; T176336 (take 3 after bumping revision)
20:40 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f (duration: 00m 22s)
20:40 mholloway-shell@deploy1001: Finished deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d
20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)f
20:40 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: l ores2001.codfw.wmnet ores2001 canary of drafttopic; T176336 (take 2 after init'ing LFS)
20:34 mholloway-shell@deploy1001: Started deploy [mobileapps/deploy@276ea43]: Update mobileapps to f579f0d
20:34 awight@deploy1001: Finished deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336 (duration: 02m 35s)
20:34 arlolra@deploy1001: Finished deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840
20:31 awight@deploy1001: Started deploy [ores/deploy@d77e52c]: ores2001 canary of drafttopic; T176336
20:30 andrew@deploy1001: Finished deploy [horizon/deploy@12aa2d3]: fix for T192179 (duration: 03m 35s)
20:27 andrew@deploy1001: Started deploy [horizon/deploy@12aa2d3]: fix for T192179
20:23 arlolra@deploy1001: Started deploy [parsoid/deploy@828034c]: Updating Parsoid to bd5a840
19:10 gehel: elasticsearch cluster restart on eqiad completed - T193734
19:06 ottomata: bouncing kafka2003 one more time for T196077
19:04 otto@deploy1001: Started restart [eventlogging/eventbus@3a5c395]: bouncing eventbus after upgrading to python-kafka 1.4.3 for T196077
19:00 ottomata: bouncing kafka2003 again to test T196077 with python-kafka 1.4.3
18:52 thcipriani@deploy1001: Synchronized docroot/noc/index.html: SWAT: Fix wrong link to Server Admin Log on noc.wikimedia.org T193848 (duration: 00m 50s)
18:44 ottomata: bouncing kafka on kafka2003 to test T196077
18:36 otto@deploy1001: Finished deploy [eventlogging/eventbus@3a5c395]: T196077 (duration: 04m 07s)
18:35 ottomata: deploying eventlogging-service-eventbus for T196077
18:32 otto@deploy1001: Started deploy [eventlogging/eventbus@3a5c395]: T196077
18:31 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test (duration: 00m 25s)
18:30 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Redeploy to test
18:01 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source (duration: 00m 25s)
18:00 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm source
17:56 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source (duration: 00m 25s)
17:55 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-intl source
17:53 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source (duration: 00m 25s)
17:52 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Enable osm-pbf source
17:51 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 25s)
17:51 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
17:48 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 24s)
17:47 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
17:44 tgr: disabled SUL+wikitech 2FA for MarkAHershberger (T196370 )
17:37 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view (duration: 00m 26s)
17:37 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Disable configurations after v3view
17:30 gehel@deploy1001: Finished deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions (duration: 08m 25s)
17:30 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 02m 54s)
17:27 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
17:22 gehel@deploy1001: Started deploy [wdqs/wdqs@fd534fa]: WDQS: new GUI and blazegraph versions
17:22 pnorman@deploy1001: Finished deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test (duration: 03m 24s)
17:18 pnorman@deploy1001: Started deploy [tilerator/deploy@074d01a] (cleartables): Deploy new meddo to test
15:09 _joe_: performing rolling restart of authdns servers to pick up ip change for the videoscalers
15:05 herron: gzipped large rotated log files in analytics1003:/var/log/hive to clear icinga disk space warning
14:49 _joe_: restarting low-traffic pybals in eqiad,codfw for adding the videoscaler VIP
14:38 ppchelko@deploy1001: Started restart [cpjobqueue/deploy@c6dc83d]: (no justification provided)
14:35 Amir1: ladsgroup@terbium:~$ foreachwikiindblist large deleteAutoPatrolLogs.php --sleep 2 --check-old
14:34 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - T190704
14:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool all sanitariums masters - T190704 (duration: 00m 49s)
14:28 moritzm: installing wireshark security updates
14:27 oblivian@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=videoscaler,name=eqiad
14:21 oblivian@puppetmaster1001: conftool action : set/pooled=inactive; selector: cluster=videoscaler,service=nginx,dc=codfw,name=mw211.*
14:21 oblivian@puppetmaster1001: conftool action : set/pooled=yes:weight=10; selector: cluster=videoscaler,service=nginx,dc=codfw
14:20 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: dc=.*,service=mathoid,cluster=kubernetes,name=.*
14:19 oblivian@puppetmaster1001: conftool action : set/pooled=yes; selector: cluster=videoscaler,service=nginx,dc=eqiad
14:10 herron: lithium:~# systemctl restart rsyslog.service
14:10 marostegui: Stop replication on all sanitarium masters to move labsdb1010 to another sanitarium host - T190704
14:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool all sanitariums masters - T190704 (duration: 00m 49s)
14:02 moritzm: installing wireshark security updates
14:02 zeljkof: EU SWAT finished
13:49 zfilipin@deploy1001: Synchronized static/images/project-logos: SWAT: Change bewikiquote logo (T196134) (duration: 00m 49s)
13:44 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgMetaNamespace to "Вікіцытатнік" on bewikiquote (T196230) (duration: 00m 49s)
13:39 anomie: Running populateExternallinksIndex60.php on group 2 for T59176 . FYI: this will probably take until next Friday to complete.
13:34 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string on zhwikisource (T194875) (duration: 00m 49s)
13:27 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgProofreadPagePageSeparator to empty string for jawikisource (T195873) (duration: 00m 49s)
13:16 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Temporarily enable MFMobileMainPageCss in ruwiki (T195905) (duration: 00m 50s)
13:11 zfilipin@deploy1001: Synchronized wmf-config/InitialiseSettings.php: SWAT: Assign movefile to autoreviewrs and patrollers on zhwiki (T195247) (duration: 00m 52s)
12:49 gehel: restart elastic2001 to enable G1 GC - T156137
12:18 krinkle@deploy1001: Finished deploy [performance/navtiming@b229f75]: (no justification provided) (duration: 00m 05s)
12:18 krinkle@deploy1001: Started deploy [performance/navtiming@b229f75]: (no justification provided)
11:38 akosiaris: rebalance row_A, row_C nodegroups in ganeti01.svc.eqiad.wmnet cluster
10:51 akosiaris: reimage ganeti1004, ganeti1008 to stretch
10:39 _joe_: rolling restart of apache on the jobrunners to pick the changed privatetmp setting, rotating logs
10:14 jdrewniak@deploy1001: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 49s)
10:13 jdrewniak@deploy1001: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 51s)
09:39 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T190704
09:34 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1097:3315 after alter table (duration: 00m 49s)
09:04 addshore: addshore@terbium:~$ for i in {1..2500}; do echo Lexeme:L$i; done | mwscript purgePage.php --wiki wikidatawiki
08:56 marostegui: Deploy schema change on db1097:3315 - T191316 T192926 T89737 T195193
08:56 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1097:3315 for alter table (duration: 00m 49s)
08:53 jynus@deploy1001: Synchronized wmf-config/db-codfw.php: Depool pc2005 (duration: 00m 50s)
08:10 jynus: restarting icinga due to ongoing check/downtime issues
07:57 marostegui: Stop replication on db2094:3315 for testing
07:29 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1082 after alter table (duration: 00m 51s)
07:11 gehel: starting elasticsearch cluster restart on eqiad - T193734
06:18 marostegui@deploy1001: Synchronized wmf-config/db-codfw.php: Repool db2059, db2075 - T190704 (duration: 00m 49s)
06:05 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Repool db1121 - T190704 (duration: 00m 49s)
05:52 marostegui: Stop replication in sync on db1121 and db2051 - T190704
05:50 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1121 - T190704 (duration: 00m 49s)
05:29 marostegui: Deploy schema change on db1082 with replication (this will generate lag on labs for s5) - T191316 T192926 T89737 T195193
05:24 marostegui@deploy1001: Synchronized wmf-config/db-eqiad.php: Depool db1082 for alter table (duration: 00m 53s)
02:53 l10nupdate@deploy1001: ResourceLoader cache refresh completed at Mon Jun 4 02:53:16 UTC 2018 (duration 10m 14s)
02:43 l10nupdate@deploy1001: scap sync-l10n completed (1.32.0-wmf.6) (duration: 14m 33s)
02:18 andrewbogott: rebooting labservices1001; it seems to have crashed
07:36 legoktm@deploy1001: Synchronized php-1.32.0-wmf.6/skins/MonoBook/: Temporarily revert responsive MonoBook (T195625 ) (duration: 00m 58s)
19:21 ebernhar1son: enable query phase slow logging and increase thresholds for fetch phase slow logging for content/general indices on eqiad and codfw elasticsearch clusters
19:14 mutante: zh.planet - fixed issue with corrupt state file and permissions - updated and using new design as well now
17:34 mutante: deployment.eqiad/codfw DNS names switched from tin to deploy1001
17:06 thcipriani@deploy1001: Synchronized README: noop test of new deployment server (duration: 00m 53s)
16:39 mutante: deploy2001 - also fixing file permissions. files owned by 996 -> mwdeploy, files owned by 997 -> trebuchet
16:21 mutante: deployment server has switched away from tin to deploy1001. set global scap lock on deploy1001, re-enabled puppet and ran puppet, disabled tin as deployment server (T175288 )
16:13 herron: enabled new logstash tcp input with TLS enabled for syslogs on port 16514 T193766
15:51 gehel: elasticsearch cluster restart on codfw completed - T193734
15:47 mutante: @deploy1001:/srv/deployment# find . -uid 997 -exec chown trebuchet {} \;
15:41 mutante: root@deploy1001:/srv/mediawiki-staging# find . -uid 996 -exec chown mwdeploy {} \;
15:17 mutante: [deploy1001:~] $ scap pull-master tin.eqiad.wmnet
15:12 mutante: tin umask 022 && echo 'switching deploy servers' > /var/lock/scap-global-lock
15:05 mutante: rsyncing /srv/mediawiki-staging to /srv/mediawiki-staging-before-backup/ on tin as a backup
14:52 mutante: deploy1001 - scap pull
14:30 elukey: killed pt-heartbear-wikimedia after https://gerrit.wikimedia.org/r/436748 on db1107
14:24 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 fully (duration: 01m 02s)
14:08 reedy@tin: Synchronized php-1.32.0-wmf.6/extensions/FlaggedRevs: T196139 (duration: 01m 08s)
13:32 marostegui: Deploy schema change on dbstore1002:s5 - T191316 T192926 T89737 T195193
13:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1096:3315 (duration: 01m 03s)
10:59 _joe_: disabling puppet on all hosts with role::mediawiki::common while installing mcrouter everywhere
10:35 marostegui: Deploy schema change on db1096:3315 - T191316 T192926 T89737 T195193
10:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1096:3315 (duration: 01m 03s)
10:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 db1096:3315 (duration: 01m 02s)
10:04 Amir1: ladsgroup@terbium:~$ foreachwikiindblist medium deleteAutoPatrolLogs.php --sleep 2 --check-old
09:53 volans@tin: Finished deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1 (duration: 00m 33s)
09:53 volans@tin: Started deploy [debmonitor/deploy@fe8df6e]: Release v0.1.1
09:37 marostegui: Stop replication in sync on db1113:3315 and db1096:3315 for data checks
09:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 db1096:3315 (duration: 01m 03s)
09:30 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 with low load (duration: 01m 03s)
08:38 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004 (duration: 04m 53s)
08:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): reenable v3view on 2004
08:30 jynus: reimage db1083
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1113:3315 after alter table (duration: 01m 03s)
08:24 jynus: temporarily reducing s7-codfw-master consistency to aliviate lag (binlog_sync, flush_log)
08:22 joal@tin: Finished deploy [analytics/refinery@7a72241]: Regular weekly deploy (duration: 12m 31s)
08:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 (duration: 01m 05s)
08:10 joal@tin: Started deploy [analytics/refinery@7a72241]: Regular weekly deploy
06:15 marostegui: Stop MySQL on db2059 to clone db2075 - T190704
06:15 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 (duration: 00m 56s)
05:36 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2092 and db2062 in s1 (duration: 00m 59s)
05:27 marostegui: Deploy schema change on db1113:3315 - T191316 T192926 T89737 T195193
05:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1113:3315 for alter table (duration: 00m 57s)
01:34 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 00m 33s)
01:33 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
01:27 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 34s)
01:24 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
01:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 24s)
01:00 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
00:59 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 52s)
00:56 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
00:53 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 26s)
00:51 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
00:48 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 03m 11s)
00:45 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
00:03 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 04m 26s)
23:59 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
23:38 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 22s)
23:36 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
23:35 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 06m 57s)
23:28 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
23:02 pnorman@tin: Finished deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 22s)
22:59 pnorman@tin: Started deploy [tilerator/deploy@709ca69] (cleartables): Redeploy to 2004 to try to reproduce error
22:51 pnorman@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): Redeploy to 2004 to try to reproduce error (duration: 02m 14s)
22:48 pnorman@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): Redeploy to 2004 to try to reproduce error
22:11 mholloway-shell@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided) (duration: 12m 46s)
21:58 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
21:55 jynus: temporarily reducing s4-codfw-master consistency to aliviate lag (binlog_sync, flush_log)
21:00 mutante: dzahn@neodymium:~$ sudo wmf-auto-reimage-host --new phab1002.eqiad.wmnet (T196019 )
20:57 mholloway-shell@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided) (duration: 00m 07s)
20:57 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
19:47 thcipriani@tin: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.6
19:42 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
19:40 mholloway-shell@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): (no justification provided)
19:31 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
19:29 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 06s)
19:29 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
19:27 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
19:25 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 05s)
19:25 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
19:11 mholloway-shell@tin: Finished deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided) (duration: 00m 06s)
19:11 mholloway-shell@tin: Started deploy [tilerator/deploy@UNKNOWN] (cleartables): (no justification provided)
19:04 ariel@tin: Finished deploy [dumps/dumps@038c8b3]: tempdir split into subdirs (duration: 00m 04s)
19:04 ariel@tin: Started deploy [dumps/dumps@038c8b3]: tempdir split into subdirs
16:07 addshore: WikibaseLexeme slot done (7 min overrun)
16:06 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: REVERT Load WikibaseLexeme on testwiki (again) T195615 (duration: 01m 21s)
16:03 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: REVERT Load WikibaseLexeme on group0 T195615 (duration: 01m 21s)
15:59 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on group0 T195615 (duration: 01m 18s)
15:52 andrewbogott: rebooting labtestservices2001 to troubleshoot unknown load problems
15:50 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Load WikibaseLexeme on testwiki (again) T195615 (duration: 01m 21s)
15:46 herron: enabling localhost:25 exim smtp listeners in production realm T175361
15:44 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: Only add repo-specific entity type definition elements in Repo context T195615 (duration: 01m 32s)
15:42 addshore@tin: Synchronized php-1.32.0-wmf.6/extensions/WikibaseLexeme: Only add repo-specific entity type definition elements in Repo context T195615 (duration: 01m 32s)
15:31 jynus: reimage db2079
14:52 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: NOOP patch and revert Load WikibaseLexeme on testwiki (sanity) (duration: 01m 22s)
14:38 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme/src/WikibaseLexemeHooks.php: T195615 Dont run repo only hooks on clients (duration: 01m 20s)
14:36 addshore@tin: Synchronized php-1.32.0-wmf.6/extensions/WikibaseLexeme/src/WikibaseLexemeHooks.php: T195615 Dont run repo only hooks on clients (duration: 01m 24s)
14:27 addshore@tin: Synchronized wmf-config/Wikibase.php: Wikibase.php shift around the loading of WikibaseLexeme (duration: 01m 22s)
14:21 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: BETA ONLY gerrit (duration: 01m 21s)
14:20 addshore@tin: Synchronized wmf-config/Wikibase-labs.php: BETA ONLY gerrit (duration: 01m 21s)
14:08 ottomata: beginning restarts of Kafka main-eqiad to enable SSL port - T193778
13:41 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitarium masters - T190704 (duration: 01m 21s)
13:23 hashar@tin: Synchronized php-1.32.0-wmf.6/vendor: WikibaseLexeme: Encoding problems in labels - T195359 (duration: 03m 12s)
12:54 akosiaris: reimage kubernetes200{3,4}.codfw.wmnet
12:30 marostegui: Stop replication on all sanitarium masters - T190704
11:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitarium masters - T190704 (duration: 01m 22s)
10:45 elukey: removed Pivot from thorium (pivot.wikimedia.org now simply redirects to Turnilo)
09:45 marostegui: Stop MySQL on db2062 to clone db2092
09:45 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2062 for cloning db2092 (duration: 01m 22s)
09:40 volans: restarting Icinga, issues processing the command file
09:35 jynus: testing icinga alerting
09:28 elukey: reimage druid1006 to Debian Stretch
09:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1082 fully (duration: 01m 21s)
08:42 gehel: power off elastic2018 - T196045
08:39 Amir1: ladsgroup@terbium:~$ mwscript deleteAutoPatrolLogs.php --wiki=commonswiki --sleep 2 --check-old
08:09 gehel: power reset elastic2018
08:08 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=codfw,service=mathoid,cluster=scb,name=scb.*
08:07 akosiaris@puppetmaster1001: conftool action : set/pooled=inactive; selector: dc=eqiad,service=mathoid,cluster=scb,name=scb.*
07:54 akosiaris: reimage kubernetes1004 without swap
07:48 ema: power-cycle elastic2018
07:41 marostegui: Stop Replication on db2066
07:40 akosiaris: re-enable puppet across the fleet
07:30 akosiaris: disable puppet for https://gerrit.wikimedia.org/r/#/c/436468/ merge cross fleet
07:27 marostegui: Deploy schema change on s5 codfw master (db2052) this will generate lag on codfw - T191316 T192926 T89737 T195193
07:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2066 after alter table (duration: 01m 22s)
06:38 marostegui: Deploy schema change on db2066 - T191316 T192926 T89737 T195193
06:38 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2066 for alter table (duration: 01m 21s)
06:30 marostegui: Deploy schema change on db2092:3315 and db2094:3315 - T191316 T192926 T89737 T195193
06:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2084:3315 after alter table (duration: 01m 28s)
06:09 akosiaris: reimage ganeti1003, ganeti1007 to stretch
05:59 elukey: reimage druid1005 to Debian Stretch
05:51 elukey: delete /tmp/scap_l10n_1501525840,scap_l10n_1501525840,l10nstuff,l10nstuff3 from tin to free some space in the root partition (1.9G left)
05:40 elukey: restart pdfrender on scb1001
05:25 marostegui: Deploy schema change on db2084:3315 - T191316 T192926 T89737 T195193
05:24 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2084:3315 for alter table (duration: 01m 27s)
05:13 moritzm: installing python-crypto security updates on trusty
04:19 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 31 04:19:57 UTC 2018 (duration 14m 48s)
04:18 moritzm: rebooting wtp2001/wtp2015 for microcode updates
04:05 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.6) (duration: 18m 15s)
03:03 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 18s)
02:09 pnorman@tin: Finished deploy [tilerator/deploy@2a26f1e] (cleartables): Deploy style with fewer fonts (duration: 00m 24s)
02:08 pnorman@tin: Started deploy [tilerator/deploy@2a26f1e] (cleartables): Deploy style with fewer fonts
02:02 pnorman@tin: Finished deploy [tilerator/deploy@78448de] (cleartables): Deploy style with fewer fonts (duration: 00m 25s)
02:01 pnorman@tin: Started deploy [tilerator/deploy@78448de] (cleartables): Deploy style with fewer fonts
01:39 pnorman@tin: Finished deploy [tilerator/deploy@fad9969] (cleartables): Deploy updated stylesheet (duration: 00m 24s)
01:39 pnorman@tin: Started deploy [tilerator/deploy@fad9969] (cleartables): Deploy updated stylesheet
01:38 pnorman@tin: Finished deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet (duration: 00m 05s)
01:38 pnorman@tin: Started deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet
01:30 pnorman@tin: Finished deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet (duration: 00m 25s)
01:30 pnorman@tin: Started deploy [tilerator/deploy@78d1b82] (cleartables): Deploy updated stylesheet
01:05 mutante: $lang.planet.wikimedia.org is changing software from planet-venus to rawdog
22:40 thcipriani@tin: Synchronized php-1.32.0-wmf.6/skins/Timeless/includes/TimelessTemplate.php: Fix condition for "emptyPortlet" class T196026 (duration: 01m 21s)
22:22 thcipriani@tin: Synchronized php-1.32.0-wmf.6/extensions/GlobalPreferences/includes/Hooks.php: SWAT: Do not type hint PreferencesFormPreSave hook against PreferencesForm T196023 (duration: 01m 22s)
21:47 thcipriani@tin: rebuilt and synchronized wikiversions files: Group1 to 1.32.0-wmf.6
20:58 thcipriani@tin: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.6
20:45 thcipriani@tin: Finished scap: testwiki to php-1.32.0-wmf.6 and rebuild l10n cache (duration: 114m 02s)
19:38 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Restore test2001 test2002 (duration: 00m 27s)
19:37 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Restore test2001 test2002
19:33 pnorman@tin: Started deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004
19:31 pnorman@tin: Finished deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004 (duration: 00m 13s)
19:31 pnorman@tin: Started deploy [tilerator/deploy@bc35971] (cleartables): Use parameterized dbname on test2004
18:51 thcipriani@tin: Started scap: testwiki to php-1.32.0-wmf.6 and rebuild l10n cache
18:05 Amir1: Morning SWAT is done
18:04 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Update Wikidata wgPropertySuggesterDeprecatedIds (duration: 01m 01s)
17:54 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on a bunch of additional wikis (T195263 ) (duration: 01m 02s)
17:47 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable the datetime selector on Sp:Block on all Wikimedia wikis (T193785 ) (duration: 01m 03s)
17:42 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Enable $wgCookieSetOnIpBlock on test wiki (T195930 ) (duration: 01m 03s)
16:38 moritzm: upgrading openjdk-7 on conf100[1-3]
16:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with full weight, repool db1089 with low (duration: 01m 02s)
15:49 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2038 after alter table (duration: 01m 01s)
15:43 moritzm: installing spice security updates
15:22 jynus: stop and reimage db1082
15:19 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with higher load, depool db1082 (duration: 01m 01s)
15:09 marostegui: Deploy schema change on db2038 - T191316 T192926 T89737 T195193
15:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2038 for alter table (duration: 01m 01s)
14:21 herron: changing logstash elasticsearch index prefix for syslogs to 'logstash-syslog' https://gerrit.wikimedia.org/r/431860
14:11 ottomata: enabling SSL port for Kafka main-codfw cluster (take 2 :) ) T193778
13:48 raynor: EU SWAT finished
13:46 pmiazga@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Remove unused PopupsAnonsExperimentalGroupSize config variable (T173952) (duration: 01m 01s)
13:44 pmiazga@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove unused PopupsAnonsExperimentalGroupSize config variable (T173952) (duration: 01m 02s)
13:42 ppchelko@tin: Finished deploy [changeprop/deploy@4503987]: Fix a bug with no content in ores response (duration: 01m 38s)
13:40 ppchelko@tin: Started deploy [changeprop/deploy@4503987]: Fix a bug with no content in ores response
13:39 ppchelko@tin: Finished deploy [eventstreams/deploy@14e0b03]: Recreate config with new puppet and restart service T167180 (duration: 02m 07s)
13:37 ppchelko@tin: Started deploy [eventstreams/deploy@14e0b03]: Recreate config with new puppet and restart service T167180
13:32 elukey: reboot analytics1002 (Hadoop master node standby) to pick up new cpu microcode
13:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 with low load (duration: 00m 59s)
13:27 ppchelko@tin: Finished deploy [changeprop/deploy@43310d4]: Emit revision-score event T167180 (duration: 01m 32s)
13:26 ppchelko@tin: Started deploy [changeprop/deploy@43310d4]: Emit revision-score event T167180
13:24 akosiaris: emptying ganeti1003, ganeti1007 for stretch upgrade
13:22 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add WMDS support question feed to mediawikiwiki RSS whitelist (T185087) (duration: 01m 01s)
13:12 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable detection of changes in moved paragraphs on most wikis (T195375) (duration: 01m 02s)
13:10 marostegui: Deploy schema change on dbstore2001:3315 - T191316 T192926 T89737 T195193
13:00 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2089:3315 after alter table (duration: 01m 02s)
12:40 akosiaris: reimage ganeti1002, ganeti1005 as stretch
12:35 elukey: reboot analytics1029,1042,1070 to pick up the new cpu-microcode
12:34 gehel: starting elasticsearch cluster restart on codfw - T193734
11:04 jynus: stop and reimage db1089
10:44 marostegui: Deploy schema change on db2089:3315 - T191316 T192926 T89737 T195193
10:43 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2089:3315 for alter table (duration: 01m 01s)
10:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 (duration: 01m 01s)
10:35 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2059 after alter table (duration: 01m 02s)
09:47 marostegui: Deploy schema change on db2059 - T191316 T192926 T89737 T195193
09:33 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2059 for alter table (duration: 01m 02s)
09:24 XioNoX: peering setup with RIPE RIS in eqsin
08:47 elukey: reimage druid1004 to Debian Stretch
08:38 marostegui: Stop and reboot db2094 and db2095 for testing - T190704
08:22 akosiaris: upgrade python3-tornado on scb1001 and restart apertium-apy. T194883
07:38 Nikerabbit: running refresh-translatable-pages.php for wikis having Translate
07:14 moritzm: installing git security updates
06:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitariums masters for s1, s3, s5, s8 - T190704 (duration: 01m 01s)
06:43 marostegui: Stop db1087 and db2045 in sync - T190704
06:31 marostegui: Stop db1082 and db2052 in sync - T190704
06:25 marostegui: Stop db1077 and db2043 in sync - T190704
06:17 elukey: reimage druid1001 to Debian stretch
06:11 marostegui: Stop db1106 and db2048 in sync - T190704
06:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitariums masters for s1, s3, s5, s8 - T190704 (duration: 01m 00s)
05:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool sanitariums masters for s2, s4, s6, s7 - T190704 (duration: 01m 00s)
05:50 elukey: restart Kafka mirror maker on kafka10[12-23] - failures to consume after rebalance
05:41 marostegui: Stop db1079 and db2040 in sync - T190704
05:31 marostegui: Stop db1085 and db2039 in sync - T190704
05:23 marostegui: Stop db1074 and db2035 in sync - T190704
05:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool sanitariums masters for s2, s4, s6, s7 - T190704 (duration: 01m 03s)
05:06 marostegui: Deploy schema change on s1 primary master db1052 - T188299
03:15 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 30 03:15:44 UTC 2018 (duration 14m 3s)
03:01 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 12m 57s)
01:08 mutante: built added rawdog_2.22-1-wmf1 to apt.wikimedia.org, upgraded rawdog on planet2001. Unpacking rawdog (2.22-1-wmf1) over (2.22-1) (T180498 )
00:45 pnorman@tin: Finished deploy [tilerator/deploy@bc35971]: Use parameterized dbname on test2004 (duration: 00m 21s)
00:45 pnorman@tin: Started deploy [tilerator/deploy@bc35971]: Use parameterized dbname on test2004
00:41 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 12s)
00:41 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
00:28 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 05s)
00:28 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
00:25 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 02m 17s)
00:22 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
00:22 pnorman@tin: Finished deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004 (duration: 00m 21s)
00:21 pnorman@tin: Started deploy [tilerator/deploy@a194185]: Deploy test of stretch build of tilerator to test2004
23:10 pnorman@tin: Finished deploy [kartotherian/deploy@2b75c93]: Deploy test of stretch build of Kartotherian to test2004 (duration: 00m 23s)
23:10 pnorman@tin: Started deploy [kartotherian/deploy@2b75c93]: Deploy test of stretch build of Kartotherian to test2004
22:09 mutante: boron - apt-get build-dep rawdog (installed libtidy5 python-feedparser python-tidylib
21:31 thcipriani@tin: rebuilt and synchronized wikiversions files: All wikis to 1.32.0-wmf.5
21:06 thcipriani@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/lib/ve: SWAT: Update VE core submodule to wmf/1.32.0-wmf.5 HEAD (9032a90ca) T195514 (duration: 01m 23s)
20:38 volans@tin: Finished deploy [debmonitor/deploy@361c94a]: Initial sync (3) (duration: 50m 13s)
19:48 volans@tin: Started deploy [debmonitor/deploy@361c94a]: Initial sync (3)
19:16 thcipriani: cutting branch for wmf.6, will not deploy wmf.6 as wmf.5 is not currently on group2 as the train is blocked on T195514 which is blocked on T195868 which is blocked on T195906
18:56 volans@tin: Finished deploy [debmonitor/deploy@fd06bd3]: Initial sync (2) (duration: 01m 42s)
18:55 volans@tin: Started deploy [debmonitor/deploy@fd06bd3]: Initial sync (2)
18:22 volans@tin: Finished deploy [debmonitor/deploy@e2efb6b]: Initial sync (duration: 00m 35s)
18:21 volans@tin: Started deploy [debmonitor/deploy@e2efb6b]: Initial sync
18:04 arlolra@tin: Finished deploy [parsoid/deploy@e87f54d]: Reverting Parsoid to fd49ab4
18:01 arlolra@tin: Started deploy [parsoid/deploy@e87f54d]: Reverting Parsoid to fd49ab4
17:59 ottomata: beginning rolling restarts of kafka main-codfw to enable SSL listener
17:36 arlolra@tin: Finished deploy [parsoid/deploy@e87f54d]: Updating Parsoid to bf3a2fd2
17:32 moritzm: repooled mw2182 (was down for hardware maintenance)
17:26 arlolra@tin: Started deploy [parsoid/deploy@e87f54d]: Updating Parsoid to bf3a2fd2
17:25 foks: Removing 2FA - T187312
17:12 bsitzmann@tin: Finished deploy [mobileapps/deploy@ac4c6be]: Update mobileapps to b2fb793 T192664 ) (duration: 06m 28s)
17:05 bsitzmann@tin: Started deploy [mobileapps/deploy@ac4c6be]: Update mobileapps to b2fb793 T192664 )
16:59 elukey: roll restart of kafka mirror maker on kafka-jumbo100* to pick up the new zookeeper settings
16:56 XioNoX: bounced analytics1031 switchport to fix weird issue of that host not being able to receive traffic from analytics1001
16:44 elukey: roll restart of kafka mirror maker on kafka100[1-3] to pick up new zk settings
16:32 thcipriani: upgrading blubber to 0.4.0 for integration machines
15:57 addshore: really done with wb_terms related syncs now
15:52 addshore@tin: Synchronized wmf-config/Wikibase.php: Revert - Dont load PropertySuggester T195520 (duration: 01m 19s)
15:48 elukey: roll restart kafka on kafka-jumbo* to pick up new zookeeper settings
15:45 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/PropertySuggester: Use CirrusSearch for PropertySuggester (duration: 01m 21s)
15:29 addshore: Wikibase - Re enable wb_terms things window - 2 more patches inbound...
15:27 elukey: restart hadoop yarn/hdfs daemons to pick up the new zookeeper settings
15:11 addshore: Wikibase - Re enable wb_terms things window done
14:57 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: TermSqlIndex::getMatchingTerms actually execute select (duration: 02m 19s)
14:57 marostegui: Move s6 topology back to its normal status
14:49 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: TermSqlIndex::getMatchingTerms actually execute select (duration: 02m 18s)
14:32 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: Re add TermSqlIndex::getMatchingTerms select, but dont call (duration: 02m 18s)
14:29 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Re add TermSqlIndex::getMatchingTerms select, but dont call (duration: 02m 13s)
14:24 elukey: roll restart kafka on kafka100[1-3] (job queues) to pick up the new zookeeper settings
14:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool all databases in row C - T187962 (duration: 01m 19s)
14:13 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: track all wb_terms table access via statsd (duration: 02m 19s)
14:13 gehel: comleted rolling restart of relforge for plugin upgrade - T193734
14:10 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: track all wb_terms table access via statsd (duration: 02m 21s)
14:07 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/CentralNotice: Convert numerical URL parameters to numbers for AndyRussG (was left on tin) (duration: 01m 25s)
14:03 elukey: swap zookeeper from conf1003 to conf1006
13:56 XioNoX: rolling back ns0 and ping1001 redirects - T187962
13:47 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create 2 extra namespaces for bdwikimedia (T195700) (duration: 01m 39s)
13:42 moritzm: upgrading remaining job runners in eqiad to hhvm-wikidiff 1.7.0
13:39 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Revert "Temp rate limit for arwiki due to mass vandalism""" (T192668) (duration: 01m 51s)
13:32 volans: restarted ircecho
13:28 volans: puppet run on failed hosts completed
13:25 moritzm: powered down mw2182 for hardware diagnosis
13:19 gehel: rolling restart of relforge for plugin upgrade - T193734
13:16 volans: running puppet on failed only hosts
13:12 volans: stopped ircecho temporarily
12:54 moritzm: installing xdg-utils security updates
11:21 marostegui: Restar db1125 mysql - T195595
11:14 moritzm: upgrading snapshot hosts to hhvm-wikidiff 1.7.0 (HHVM is unused, just for completeness)
11:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Disable read only on s6 T194939 T187962 (duration: 01m 37s)
11:00 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Enable read only on s6 T194939 T187962 (duration: 01m 35s)
10:55 XioNoX: Eqiad row C server move starting - T187962
10:53 XioNoX: Eqiad row C server move starting
10:35 moritzm: upgrading mw1308-mw1311 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
10:09 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus file 2/2 - T190327 T195500 (duration: 01m 47s)
10:06 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus file 1/2 - T190327 T195500 (duration: 01m 39s)
10:05 ppchelko@tin: Finished deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327 (duration: 00m 58s)
10:04 ppchelko@tin: Started deploy [cpjobqueue/deploy@c6dc83d]: Enable all jobs apart from exceptions for everything. T190327
09:20 XioNoX: redirect ns0 to baham - T187962
09:16 XioNoX: disable ping1001 redirect - T187962
09:13 marostegui: Downtime s6 replicas for 4 hours - T195595
09:07 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool all databases in row C - T187962 (duration: 01m 35s)
09:05 moritzm: upgrading labweb servers to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
08:40 jynus: performing topology changes on s6 ahead of a possible failover
08:24 moritzm: upgrading remaining API servers in eqiad to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
07:56 moritzm: upgrading mw1276-mw1290 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
07:49 elukey: reimage druid1002 to debian stretch
07:47 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on ruwiki (duration: 01m 50s)
07:26 moritzm: upgrading remaining app servers in eqiad to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
06:52 elukey: roll restart hadoop master daemons to pick up the new zookeeper settings
05:20 marostegui: Restart MySQL on db2045 (s8 codfw master) - T195598
05:14 marostegui: Stop MySQL on db2094 and db2095 for testing - T190704
04:12 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 29 04:12:10 UTC 2018 (duration 14m 32s)
03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 14m 29s)
02:59 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 13m 18s)
20:14 twentyafterfour: Test failures on https://gerrit.wikimedia.org/r/#/c/435825/ are preventing deployment of the fix for a critical deployment blocker (see T195514 ) 1.32.0-wmf.5 still blocked refs T191051
20:10 twentyafterfour: train still held up by test failures: https://gerrit.wikimedia.org/r/#/c/435825/
20:02 elukey: restart kafka on kafka1003 as attempt to solve the under-replicated partitions warning
19:22 twentyafterfour@tin: Synchronized php-1.32.0-wmf.5/extensions/CentralNotice/: sync wmf.5 CentralNotice for AndyRussG (duration: 01m 25s)
19:12 elukey: roll restart of kafka-mirror maker (main eqiad -> jumbo) on kafka-jumbo* for zookeeper conf updates
19:07 twentyafterfour: attempting to get the wmf.5 train back on track. Deploying a fix for T195514 (https://gerrit.wikimedia.org/r/c/435292/ ) to unblock T191051
18:16 elukey: restart kafka mirror maker on kafka1012->14 - failed after the last round of kafka restarts
17:26 elukey: roll restart of kafka on kafka-jumbo* to pick up the new zookeeper settings
17:20 gehel@tin: Finished deploy [wdqs/wdqs@0e40344]: WDQS updater and GUI (duration: 08m 59s)
17:19 elukey: restart kafka on kafka1012->23 to pick up the new zookeeper settings
17:11 gehel@tin: Started deploy [wdqs/wdqs@0e40344]: WDQS updater and GUI
17:08 moritzm: upgrading mwdebug servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
16:48 moritzm: upgrading codfw video scalers to hhvm-wikidiff 1.7.0
16:31 elukey: roll restart kafka on kafka100[1-3] to pick up new zookeeper settings
16:21 elukey: zookeeper cluster restart completed (main-eqiad / conf1*)
16:18 elukey: stop and mask zookeeper on conf1002
16:16 elukey: restart prometheus-burrow-exporter on kafkamon*
16:12 moritzm: upgrading job runners in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
16:02 marostegui: Stop MySQL on db2095 for testing - T190704
15:59 elukey: swap zookeeper from conf1002 to conf1005
15:56 marostegui: Reboot db1124 and db1125 for more testing - T190704
15:49 _joe_: uploading cergen 0.2.3
14:44 Amir1: EU SWAT is done
14:39 ladsgroup@tin: Synchronized php-1.32.0-wmf.4/extensions/ArticlePlaceholder: Add config variable to disable SearchHookHandler (T195753 ) (duration: 01m 21s)
14:34 ladsgroup@tin: Synchronized wmf-config/InitialiseSettings.php: Disable search integration with Article Placeholder temporarily (T195753 ) (duration: 01m 20s)
14:29 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/ArticlePlaceholder: Add config variable to disable SearchHookHandler (T195753 ) (duration: 01m 18s)
14:23 ladsgroup@tin: Synchronized wmf-config/Wikibase-production.php: Disable Special:ItemDisambiguation in Wikidata (T195756 ) (duration: 01m 20s)
14:21 ema: cp1045,cp2001,cp3007,cp5001: reboot with intel-microcode T127825
14:00 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable "File mover" flag on zh.wikipedia (T195247) (duration: 01m 19s)
13:52 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: New protection level on the Hungarian Wikipedia - trusted (T194568) (duration: 01m 20s)
13:40 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Revert "Enable $wgUseRCPatrol on azwiki"" (T194389) (duration: 01m 20s)
13:34 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Use uploaded HD logos for yiwikisource (T193562) (duration: 01m 19s)
13:28 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Upload new logos for yiwikisource (T193562) (duration: 01m 19s)
13:21 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable template editor group on newiki (T195557) (duration: 01m 21s)
13:13 volans: restarted pdfrender on scb1002
13:03 moritzm: upgrading app servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
12:09 arturo: T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-stretch' update stretch-wikimedia
12:08 arturo: T194665 aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-jessie' update jessie-wikimedia
12:05 arturo: aborrero@install1002:~$ sudo -i reprepro --noskipold -C 'thirdparty/mono-project-trusty' update trusty-wikimedia
11:31 ema: cp1008: reboot with intel-microcode T127825
11:27 moritzm: upgrading API servers in codfw to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
11:10 marostegui: Stop MySQL on db2092 to copy its content to db2094 - T190704
10:36 moritzm: upgrading mw1266-mw1275 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
10:14 moritzm: upgrading eqiad video scalers to hhvm-wikidiff 1.7.0
10:09 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 20s)
10:08 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 21s)
09:07 moritzm: upgrading mw1299-mw1306 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
09:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1083 after alter table (duration: 01m 20s)
09:04 marostegui: Deploy schema change on s1 primary master (db1052) - T191519
09:00 marostegui: Deploy schema change on db1052 (s1 primary master) - T190148
08:27 moritzm: upgrading mw1221-mw1235 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
07:40 XioNoX: enable backup tunnel routing between cr2-ulsfo and cr1-eqdfw - T195584
07:31 marostegui: Deploy schema change on db1083 - T190148 T191519 T188299
07:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1083 for alter table (duration: 01m 20s)
07:29 moritzm: upgrading mw1238-mw1258 to hhvm-wikidiff 1.7.0 (HHVM bytecode cache needs to be pruned during rollout)
07:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 after alter table (duration: 01m 22s)
07:10 moritzm: uploaded hhvm-wikidiff2 1.7.0 (source package name php-wikidiff2) to apt.wikimedia.org
06:44 moritzm: installing ruby-loofah security updates
06:36 elukey: reimage druid1003 to Debian Stretch (Analytics cluster, backend for Pivot/Turnilo)
05:25 marostegui: Stop MySQL on db2075 to copy its content to db2095 - T190704
05:23 marostegui: Deploy schema change on db1106 with replication, this will generate lag on labs - T190148 T191519 T188299
05:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 for alter table (duration: 01m 24s)
03:14 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 14m 30s)
21:47 reedy@tin: Synchronized composer.json: (no justification provided) (duration: 01m 19s)
21:45 reedy@tin: Synchronized multiversion/: multiversion (duration: 01m 21s)
21:42 reedy@tin: Synchronized vendor/: canhasvendor (duration: 01m 46s)
19:01 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 01m 21s)
14:25 marostegui: Add tmp1 index back on db1101:3318 - T194273
14:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 01m 07s)
14:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 01m 20s)
09:56 marostegui: Add tmp1 index back on db1087 (sanitarium master), this will generate lag on labsdb hosts - T194273
09:56 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 01m 20s)
09:47 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 21s)
05:21 marostegui: Add tmp1 index back on db1099:3318 - T194273
05:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 01m 21s)
05:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 22s)
22:57 mutante: apt.wikimedia.org - import jenkins-debian-glue_0.18.4-wmf3 for jessie-wikimedia (T193910 )
21:27 legoktm@tin: Synchronized php-1.32.0-wmf.5/skins/MonoBook/: Temporarily remove responsive support (T195625 ) (duration: 01m 21s)
20:33 mutante: LDAP: added user wmde-leszek to group 'nda' (T195358 )
17:46 XenoRyet: updated civicrm from 4d797fc592 0b97f1f5b2
15:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 20s)
13:47 akosiaris: repool ulsfo, links have been stable for quite a few hours
13:27 marostegui: Deploy schema change on db1119 - https://phabricator.wikimedia.org/T190148 https://phabricator.wikimedia.org/T191519 https://phabricator.wikimedia.org/T188299
13:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1119 for alter table (duration: 01m 20s)
13:15 marostegui: Add indexes back on s8 codfw primary master (db2045) this will generate lag on codfw - T194273
13:14 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:3311 after alter table (duration: 01m 20s)
12:28 moritzm: fixed dpkg installation state on mx2001
11:32 akosiaris: switch to SSH RSA 2048 bit keys for eqiad ganeti intracluster communication
11:22 akosiaris: upgrade eqiad ganeti cluster to ganeti 2.15.2-7+deb9u1~bpo8+1
11:21 akosiaris: rebalance row_B codfw ganeti nodegroup. Cluster is now fully upgraded to stretch
11:18 akosiaris: powercycling ms-be1034, box is unresposive, tons of logs "sd 0:1:0:1: rejecting I/O to offline device"
10:37 XioNoX: test force mtu 1400 between cp1074 and cp3039 - T195365
09:20 marostegui: Deploy schema change on db1105:3311 - T190148 T191519 T188299
09:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 for alter table (duration: 01m 20s)
09:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1119 after alter table (duration: 01m 19s)
09:10 marostegui@tin: scap failed: average error rate on 9/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1114 after alter table (duration: 01m 20s)
08:44 marostegui: Stop MySQL on db1120 to transfer its content to db1125 - T190704
08:39 marostegui: Add tmp1 back on db1092 - https://phabricator.wikimedia.org/T194273
08:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s)
08:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 20s)
08:26 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on frwiki (duration: 01m 22s)
07:05 jynus: stop db1117:m2 to clone it to db1065
06:57 marostegui: Deploy schema change on db1114 - T190148 T191519 T188299
06:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1114 for alter table (duration: 01m 20s)
06:53 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 after alter table (duration: 01m 20s)
06:37 jynus: reimage db1065 after raid rebuild
05:57 marostegui: Add tmp1 index back on dbstore1002 - T194273
05:25 marostegui: Stop MySQL on db1116 to copy its content to db1124 - T190704
05:23 marostegui: Deploy schema change on db1089 - T190148 T191519 T188299
05:23 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 for alter table (duration: 01m 20s)
05:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 after alter table (duration: 01m 21s)
05:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 20s)
05:05 marostegui: Add tmp1 index back to db1109 - T194273
04:59 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 21s)
03:10 papaul: OS install on db209[4-5]
23:13 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: Dumps: Allow several --entity-type arguments (T195420 ) (duration: 02m 25s)
22:21 ladsgroup@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520 ) (duration: 01m 20s)
22:11 ladsgroup@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520 ) (duration: 01m 21s)
20:58 marostegui: Add tmp1 index back on db1104
20:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 20s)
20:56 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/: no-op, sync with git state (duration: 01m 21s)
20:54 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s)
20:41 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: no-op, sync with git state (duration: 01m 20s)
20:17 legoktm@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: wmf.5 this time (duration: 01m 19s)
20:14 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Add debug logging (duration: 01m 19s)
20:11 legoktm@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Disable TermSqlIndex::getMatchingTerms (duration: 01m 20s)
19:49 ejegg: updated payments-wiki from c81e25f8d3 43989ebc96
19:44 reedy@tin: Synchronized wmf-config/Wikibase.php: Disable PropSuggester (duration: 01m 21s)
19:30 twentyafterfour@tin: scap failed: average error rate on 3/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
19:14 twentyafterfour: Starting the MediaWiki train for Thursday May 24, today I will be deploying wmf/1.32.0-wmf.5 to all wikis
18:31 mutante: netmon1002, netmon2001: systemctl mask uwsgi; systemctl reset-failed - to fix Icinga alert about broken DPKG since last netbox deploy and to match existing status on labtestweb2001
18:21 XioNoX: repool esams
18:01 no_justification: gerrit restarting on cobalt, back soon
17:50 mutante: mwdebug1001 - apt-get remove to clean up packages in "non ii" states
16:59 anomie: Running populateExternallinksIndex60.php on group 1 for T59176
16:50 XioNoX: depooled esams - investigating issues
16:31 anomie: Running deduplicateArchiveRevId.php on group 2 for T193180
16:28 jynus: manually failover the backup host for m2 to db1117:3322
16:22 mutante: tin apt-get clean saved 7% disk space on / - fixing disk space alert
16:19 marostegui: Deploy schema change on db1099:3311 - T191519 T188299 T190148
16:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3311 for alter table (duration: 01m 13s)
16:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 after alter table (duration: 01m 17s)
16:15 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase/repo/maintenance/dispatchChanges.php: Dont use WikibaseRepo class in dispatchChanges constructor (duration: 10m 28s)
16:13 mutante: deploy2001 - let mwdeploy own .~tmp~ in /srv/mediawiki-staging
15:48 cmjohnson: swapped failed disk 0 db1065
15:06 moritzm: installing glibc updates from stretch point release
14:57 marostegui: Deploy schema change on dbstore1001:s1 - T191519 T188299 T190148
14:56 papaul: shutting down elastic2020 for maintenance
14:51 moritzm: upgrading mwdebug servers in eqiad to wikidiff 1.7.0
14:09 akosiaris: empty ganeti2004 for stretch reimage
14:00 mutante: deploy2001: scap pull to sync. then add as scap master and host (gerrit:433616)
13:37 akosiaris: rebalance row_A codfw ganeti nodegroup. Fully upgrade to stretch now
13:33 marostegui: Deploy schema change on db1067 - T191519 T188299 T190148
13:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 for alter table (duration: 01m 00s)
13:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 after alter table (duration: 00m 56s)
13:27 anomie: Running deduplicateArchiveRevId.php on group 1 for T193180
13:25 moritzm: upgrading mw1262-mw1265 (canary hosts) to wikidiff 1.7.0
13:13 moritzm: upgrading mw1261 (canary host) to wikidiff 1.7.0
11:31 moritzm: rebooting mw1261, mw1276, mw1319, mw1312, mw1258, mw1221 to use Intel microcode updates
11:19 marostegui: Deploy schema change on db1080 - T191519 T188299 T190148
11:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1080 for alter table (duration: 01m 08s)
11:10 marostegui: Deploy schema change on dbstore1002:s1 - T191519 T188299 T190148
11:05 XioNoX: GTT work on eqiad-esams link starting soon
10:55 krinkle@tin: Synchronized php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s)
10:53 Krinkle: Unexpected dirty git status at tin:/srv/mediawiki-staging/php-1.32.0-wmf.4/extensions/JADE (1 file is locally deleted, but not committed)
10:51 krinkle@tin: Synchronized php-1.32.0-wmf.5/includes/resourceloader/ResourceLoaderUserModule.php: T195380 (duration: 01m 08s)
09:55 mobrovac@tin: Synchronized wmf-config/InitialiseSettings.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 2/2 - T190327 (duration: 01m 06s)
09:55 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327 (duration: 00m 42s)
09:54 ppchelko@tin: Started deploy [cpjobqueue/deploy@b537fa1]: Switch all non-special jobs for everything except wikipedia, commons and wikidata T190327
09:53 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2, take #2 - T190327 (duration: 01m 08s)
09:37 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch all jobs to EventBus for everything except wikipedia, commons and wikidata, file 1/2 - T190327 (duration: 01m 09s)
09:33 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production (duration: 02m 35s)
09:31 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do a fresh deploy of Kartotherian to production
09:27 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 22s)
09:27 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004
09:26 pnorman@tin: Finished deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004 (duration: 00m 23s)
09:25 pnorman@tin: Started deploy [kartotherian/deploy@9fc09ef]: Do test deploy of kartotherian to maps-test2004
09:25 ppchelko@tin: Finished deploy [cpjobqueue/deploy@f66dacb]: Correctly commit offsets for multi-topic rules (duration: 00m 49s)
09:24 ppchelko@tin: Started deploy [cpjobqueue/deploy@f66dacb]: Correctly commit offsets for multi-topic rules
08:55 pnorman@tin: Finished deploy [tilerator/deploy@9e40702] (cleartables): Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 13s)
08:55 pnorman@tin: Started deploy [tilerator/deploy@9e40702] (cleartables): Deploy scap fixes to cleartables map test server in verbose mode
08:55 pnorman@tin: Finished deploy [tilerator/deploy@9e40702] (cleartables--force): Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 05s)
08:55 pnorman@tin: Started deploy [tilerator/deploy@9e40702] (cleartables--force): Deploy scap fixes to cleartables map test server in verbose mode
08:51 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 13s)
08:51 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
08:47 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 05s)
08:47 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
08:45 marostegui: Deploy schema change on s1 codfw primary master (db2048), this will generate lag on codfw - T191519 T188299 T190148
08:45 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode (duration: 00m 01s)
08:45 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server in verbose mode
08:40 ayounsi@tin: Finished deploy [netbox/deploy@ac54feb]: Adding service name in scap.cfg (duration: 00m 33s)
08:39 ayounsi@tin: Started deploy [netbox/deploy@ac54feb]: Adding service name in scap.cfg
08:35 pnorman@tin: Finished deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server go 3 (duration: 00m 23s)
08:35 pnorman@tin: Started deploy [tilerator/deploy@9e40702]: Deploy scap fixes to cleartables map test server go 3
08:32 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Launch performance survey on cawiki and enwikivoyage (duration: 01m 08s)
08:21 gilles: Deployment of cawiki and enwikivoyage performance survey
08:16 jynus: stop db2037 to clone it to db2078 and upgrade
08:05 marostegui: Deploy schema change on s8 primary master (db1071) - T191519 T188299 T190148 T194270
08:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1109 after alter table (duration: 01m 09s)
07:48 jynus: stop db2042 to clone it to db2078 and upgrade
07:36 marostegui: Deploy schema change on db1109 - T191519 T188299 T190148 T194270
07:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1109 for alter table (duration: 01m 08s)
07:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1104 after alter table (duration: 01m 29s)
07:17 moritzm: installing procps security updates
06:47 marostegui: Deploy schema change on db1104 - T191519 T188299 T190148 T194270
06:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1104 for alter table (duration: 01m 08s)
06:39 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1087 after alter table (duration: 01m 08s)
06:24 moritzm: installing remaining curl security updates in eqiad
06:17 marostegui: Deploy schema change on db1087, this will generate lag on labs on s8 - T191519 T188299 T190148 T194270
06:17 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1087 for alter table (duration: 01m 09s)
05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 (duration: 01m 09s)
04:04 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 24 04:04:35 UTC 2018 (duration 7m 16s)
03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 13m 29s)
03:06 mutante: added wmde-fisch to LDAP group nda (T195223 )
03:01 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 12m 48s)
00:15 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/CirrusSearch/includes/Job/CheckerJob.php: Drop cirrus checker job metastore transition check (duration: 01m 08s)
00:14 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/CirrusSearch/includes/Job/CheckerJob.php: Drop cirrus checker job metastore transition check (duration: 01m 08s)
00:10 ebernhardson: upgrade cirrussearch metastore to 1.0 on eqiad and codfw
00:09 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/CirrusSearch/: SWAT: Convert cirrus metastore to single type (duration: 01m 24s)
00:07 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/CirrusSearch/: SWAT: Convert cirrus metastore to single type (duration: 01m 24s)
00:00 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: T195323 : MWSaveDialog: Fix typo in no-categories branch (duration: 01m 07s)
23:49 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/modules/ve-mw/ui/tools/ve.ui.MWPopupTool.js: SWAT: Fix typo in API call for version number help (duration: 01m 08s)
23:45 ebernhardson@tin: Synchronized php-1.32.0-wmf.5/extensions/VisualEditor/modules/ve-mw/ui/dialogs/ve.ui.MWSaveDialog.js: SWAT: T195323 : MWSaveDialog: Fix typo in no-categories branch (duration: 01m 08s)
23:42 ebernhardson@tin: Synchronized php-1.32.0-wmf.4/extensions/MobileFrontend/resources/mobile.editor.common/editor.less: SWAT: T194832 : Fix layout of editor switcher dropdown (duration: 01m 08s)
23:10 ebernhardson@tin: Synchronized wmf-config/InitialiseSettings.php: T152296 : Remove dangerous unused botadmin group at mlwik{tionary|isource} (duration: 01m 10s)
21:29 mutante: arming keyholder on deploy2001
20:57 mutante: rebooting deploy2001
20:56 arlolra@tin: Finished deploy [parsoid/deploy@de18a58]: Reverting Parsoid deploy (duration: 02m 37s)
20:53 arlolra@tin: Started deploy [parsoid/deploy@de18a58]: Reverting Parsoid deploy
20:52 pnorman@tin: Finished deploy [tilerator/deploy@63617a9]: Deploy scap fixes to cleartables map test server go 2 (duration: 00m 23s)
20:52 pnorman@tin: Started deploy [tilerator/deploy@63617a9]: Deploy scap fixes to cleartables map test server go 2
20:43 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: Deploy scap fixes to cleartables map test server (duration: 00m 08s)
20:42 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: Deploy scap fixes to cleartables map test server
20:30 bsitzmann@tin: Finished deploy [mobileapps/deploy@5896151]: Update mobileapps to 29ebe0f T192664 ) (duration: 08m 54s)
20:28 arlolra: Updated Parsoid to dccfeafd T157418 , T194777 , T195317 , T195174 , T194763 , T194658 )
20:21 bsitzmann@tin: Started deploy [mobileapps/deploy@5896151]: Update mobileapps to 29ebe0f T192664 )
20:18 arlolra@tin: Finished deploy [parsoid/deploy@de18a58]: Updating Parsoid to dccfeafd
20:05 arlolra@tin: Started deploy [parsoid/deploy@de18a58]: Updating Parsoid to dccfeafd
19:36 mutante: reinstalling naos as deploy2001, booting to PXE (T193916 )
19:23 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.5 refs T191051 (duration: 01m 08s)
19:21 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.5 refs T191051
18:24 twentyafterfour@tin: Synchronized README: testing sync-masters without naos (duration: 01m 09s)
17:54 twentyafterfour: Finished SWATting, thanks everyone!
17:52 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings.php: sync https://gerrit.wikimedia.org/r/#/c/433546/ for SWAT refs T194871 (duration: 01m 19s)
17:47 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/Translate/MessageCollection.php: syncing https://gerrit.wikimedia.org/r/#/c/434700/ refs T195347 (duration: 01m 19s)
17:45 twentyafterfour@tin: Synchronized php-1.32.0-wmf.5/extensions/Translate/MessageCollection.php: syncing https://gerrit.wikimedia.org/r/#/c/434699/ refs T195347 (duration: 01m 20s)
17:18 krinkle@tin: Synchronized w/robots.php: Ib1c6d676
17:17 krinkle@tin: Synchronized w/extract2.php: Ib1c6d676
17:14 twentyafterfour@tin: Synchronized wmf-config/Wikibase-production.php: SWAT deploying https://gerrit.wikimedia.org/r/#/c/434658/ refs T194273 (duration: 01m 20s)
16:31 SMalyshev: starting wikidata full reindex for T163642
16:11 addshore@tin: Finished scap: WikibaseLexeme: Do not refer to the spelling variant as language, T193603 , Patch 1 , Patch 2 (duration: 103m 38s)
16:10 anomie: Running deduplicateArchiveRevId.php on group 0 for T193180
15:53 anomie: Running populateExternallinksIndex60.php on group 0 for T59176
15:30 jynus: stop db2044 for cloning to db2078 + upgrade
14:52 jynus: restart db2078 for upgrade and to convert it to multiinstance
14:30 moritzm: installing curl security updates on mediawiki canaries along with HHVM restart to pick up new library version
14:28 addshore@tin: Started scap: WikibaseLexeme: Do not refer to the spelling variant as language, T193603 , Patch 1 , Patch 2
14:12 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable HD logos for wikimania2018wiki T194340 (duration: 01m 20s)
14:05 zeljkof: EU SWAT finished
14:01 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Disable wikidiff2 inline moved paragraphs by default (T194271) (duration: 01m 18s)
13:55 jynus: stopping db1065 database to move it to m2
13:47 zfilipin@tin: Synchronized static/images/project-logos/: SWAT: Change logo assets for wikimania2018wiki (T194340) (duration: 01m 20s)
13:32 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable wikidiff2 inline moved paragraphs on production (T194271) (duration: 01m 21s)
13:22 zfilipin@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Revert "Disable wikidiff2 inline moved paragraphs by default" (T194271) (duration: 01m 19s)
13:17 zfilipin@tin: scap failed: average error rate on 7/11 canaries increased by 10x (rerun with --force to override this check, see https://logstash.wikimedia.org/goto/2cc7028226a539553178454fc2f14459 for details)
13:05 jynus: starting logical dump of m3-master
13:00 jynus: starting logical dump of m2-master
12:47 addshore: addshore@terbium:~$ echo 'https://www.wikidata.org/wiki/Special:EntityData/L1.rdf' | mwscript purgeList.php
12:43 addshore@tin: Synchronized wmf-config/Wikibase.php: Fix disabledRdfExportEntityTypes for wikidata T168260 (duration: 01m 20s)
12:16 XioNoX: set normal metric on codfw-eqsin link
12:06 elukey: upgrade druid public to druid 0.11 (druid100[4-6])
11:45 akosiaris: reimage ganeti2002, ganeti2006 as debian stretch
11:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@59674ba]: Correctly commit pending messages for multi-topic rules (duration: 00m 47s)
11:44 ppchelko@tin: Started deploy [cpjobqueue/deploy@59674ba]: Correctly commit pending messages for multi-topic rules
11:33 addshore: addshore@mw1317:~$ scap pull # It seemed to be missing changes.....
11:30 moritzm: installing curl security updates
11:09 addshore: Lexeme deploy window probably done (unless something explodes)
10:42 addshore: addshore@terbium mwscript extensions/WikibaseLexeme/maintenance/createBlacklistedLexemes.php --wiki testwikidatawiki
10:40 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseLexeme on wikidata.org T191457 T168260 https://gerrit.wikimedia.org/r/#/c/434453 (duration: 01m 20s)
10:32 jynus: stop db1065 for cloning (proxys will complain)
10:27 ppchelko@tin: Finished deploy [cpjobqueue/deploy@ffdc19a]: Logging improvements (duration: 00m 47s)
10:26 ppchelko@tin: Started deploy [cpjobqueue/deploy@ffdc19a]: Logging improvements
10:21 jynus: restarting db1117 for setup and upgrade
10:17 dcausse: manually updating mapping on wikidatawiki elastic indices to add new lexeme fields
10:15 addshore: updateSearchIndexConfig.php with --justMapping failed :(
10:15 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3318 after alter table (duration: 01m 20s)
10:14 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki wikidatawiki --justMapping --cluster codfw
09:35 addshore: Finished, forceSearchIndex.php --wiki testwikidatawiki
09:34 moritzm: upgrading remaining job runners in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:30 marostegui: Deploy schema change on db1101:3318 - T191519 T188299 T190148 T194270
09:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3318 for alter table (duration: 01m 20s)
09:03 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1099:3318 after alter table (duration: 01m 19s)
09:02 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/forceSearchIndex.php --wiki testwikidatawiki --queue
09:00 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: [testwikidata] Add Lexeme NS to ContentNamespaces (duration: 01m 20s)
08:31 moritzm: upgrading remaining app/API servers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
08:21 marostegui: Deploy schema change on db1099:3318 - T191519 T188299 T190148 T194270
08:20 addshore: all the updateSearchIndexConfig runs from wasat also had --cluster=codfw
08:20 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1099:3318 for alter table (duration: 01m 20s)
08:18 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
08:18 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
08:17 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: testwikidata: Add Lexeme NS to wgNamespacesToBeSearchedDefault (duration: 01m 19s)
08:06 addshore: addshore@wasat:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now --reindexProcesses=10
08:00 moritzm: upgrading remaining app servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
07:54 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
07:52 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: testwikidata: Add Property NS to wgNamespacesToBeSearchedDefault (duration: 01m 20s)
07:45 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/WikibaseLexeme: Use ISO code und for missing language code (duration: 01m 30s)
07:44 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use ISO code und for missing language code (duration: 01m 31s)
07:38 moritzm: upgrading remaining API servers in eqiad to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
07:29 elukey: upload druid debs 0.11.0-3 to stretch-wikimedia
06:52 moritzm: upgrading mw1299-mw1306 (job runners) to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
06:49 marostegui: Re-add indexes on wb_terms on db1092 - T194273
06:44 elukey: restart zookeeper on druid100[4-6] for openjdk-8 upgrades
06:44 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 20s)
06:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1092 after alter table (duration: 01m 20s)
05:43 marostegui: Deploy schema change on db1092 - T191519 T188299 T190148 T194273 T194270
05:43 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1092 for alter table (duration: 01m 27s)
05:21 marostegui: Deploy schema change on dbstore1002:s8 - T191519 T188299 T190148 T194273 T194270
05:12 marostegui: Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194273
05:06 marostegui: Deploy schema change on s3 primary master (db1075) - T191519 T188299 T190148
04:04 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 23 04:04:43 UTC 2018 (duration 7m 22s)
03:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.5) (duration: 16m 39s)
02:58 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 10m 54s)
23:29 maxsem@tin: Synchronized dblists/categories-rdf.dblist: https://gerrit.wikimedia.org/r/#/c/433740/ (duration: 01m 17s)
22:59 twentyafterfour: train for group0 1.32.0-wmf.5 completed. Tune in tomorrow for more excitement!
22:58 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.5 refs T191051
22:54 twentyafterfour@tin: Finished scap: testwikis wikis to 1.32.0-wmf.5 refs T191051 (duration: 76m 01s)
21:38 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.5 refs T191051
21:13 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_2833418486" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 39m 54s)
21:12 ejegg: updated CiviCRM from 5b8c868a00 4d797fc592
20:33 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.5 refs T191051
19:45 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution (duration: 02m 46s)
19:43 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: Update tilerator, fix variable substitution
19:39 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 29s)
19:38 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: (no justification provided)
19:35 pnorman@tin: Finished deploy [tilerator/deploy@18faaa6]: (no justification provided) (duration: 00m 25s)
19:35 pnorman@tin: Started deploy [tilerator/deploy@18faaa6]: (no justification provided)
19:34 pnorman@tin: Finished deploy [tilerator/deploy@a4a3fc7]: (no justification provided) (duration: 46m 19s)
19:17 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/ContentTranslation/: sync https://gerrit.wikimedia.org/r/#/c/434529/ to fix T194810 (duration: 01m 02s)
19:14 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group2 wikis to 1.32.0-wmf.4 refs T191050
19:03 twentyafterfour: Today's train: Promoting group2 wikis to 1.32.0-wmf.4 followed by group0 to 1.32.0-wmf.5
18:48 pnorman@tin: Started deploy [tilerator/deploy@a4a3fc7]: (no justification provided)
18:44 pnorman: tilerator deploying a4a3fc7
17:03 marostegui: Deploy schema change on s8 codfw primary master (db2045), this will generate lag on codfw - T194270
16:52 elukey: restart zookeeper on druid100[1,3] to complete the openjdk-8 upgrade
16:51 addshore: addshore@terbium:~$ mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki testwikidatawiki --reindexAndRemoveOk --indexIdentifier=now
16:43 elukey: upload druid debs 0.11.0-3 to jessie-wikimedia
16:25 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db2064 from config - T195228 (duration: 01m 18s)
16:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db2064 from config - T195228 (duration: 01m 19s)
16:14 moritzm: upgrading remaining video scalers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
15:20 ppchelko@tin: Finished deploy [cpjobqueue/deploy@4312549]: Increase the concurrency for low traffic topics (duration: 00m 41s)
15:19 ppchelko@tin: Started deploy [cpjobqueue/deploy@4312549]: Increase the concurrency for low traffic topics
14:47 addshore@tin: Synchronized wmf-config/Wikibase.php: Wikidata dispatch, set defaults for dispatchChanges settings (duration: 01m 19s)
14:29 addshore: SWAT done
14:28 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: T193680 Use uploaded logo in wgLogo for liwikibooks (duration: 01m 16s)
14:26 addshore@tin: Synchronized static/images/project-logos/liwikibooks.png: SWAT: T193680 Change liwikibooks logo (duration: 01m 18s)
14:24 addshore: that last also included https://gerrit.wikimedia.org/r/#/c/434495/ - Use correct logo-size for wikimania2018wiki
14:24 addshore@tin: Synchronized static/images/project-logos/wikimania2018wiki.png: SWAT: Change logo for wikimania2018wiki T194340 (duration: 01m 19s)
14:07 elukey: upgrading druid on druid100[123] to 0.11
13:51 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Revert Temp rate limit for arwiki due to mass vandalism (duration: 01m 18s)
13:46 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Revert Enable $wgUseRCPatrol on azwiki (duration: 01m 19s)
13:39 elukey: upload druid 0.11 debs to jessie|stretch wikimedia
13:26 moritzm: upgrading API servers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
13:25 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgUseRCPatrol on azwiki T194389 (duration: 01m 20s)
13:21 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert: Temp rate limit for arwiki due to mass vandalism T192668 (duration: 01m 18s)
13:18 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Add L171081 to clearBlacklistedLexemes (duration: 01m 28s)
13:13 moritzm: upgrading labweb servers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
13:06 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Handle invalid lexemeId in data when using wbeditentity new=form && Lexeme term languages: codes beyond MW default (duration: 01m 28s)
13:01 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Lemma validation: language covered in deserializer (duration: 01m 30s)
12:59 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use the same language validation for representations and lemmas (duration: 01m 25s)
12:57 moritzm: installing xdg-utils security updates on trusty
12:55 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/WikibaseLexeme: Use ChangeOps consistently throughout API (duration: 01m 30s)
12:42 addshore@tin: Synchronized wmf-config: #1 #2 BETA ONLY profiler stuff (duration: 01m 20s)
12:33 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase/repo: API: when validating change op make sure the edited entity is also validated T190928 (duration: 01m 52s)
12:18 gehel: set `unchecked_tombstone_compaction=true` for maps eqiad - T194966
12:15 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Set wgLexemeLanguageCodePropertyId for wikidatawiki T194248 (duration: 01m 19s)
12:11 moritzm: installing imagemagick security updates
12:07 addshore@tin: Finished scap: WikimediaMessages - wikidata-copyright, include the lexeme namespace T169333 (duration: 56m 45s)
11:31 moritzm: upgrading application servers in deployment-prep to wikidiff 1.7.0 (T190717 )
11:10 addshore@tin: Started scap: WikimediaMessages - wikidata-copyright, include the lexeme namespace T169333
11:06 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Add 171081 to wmgWikibaseIdBlacklist for wikibase-lexeme T194248 T187060 (duration: 01m 19s)
11:04 ppchelko@tin: Started restart [cpjobqueue/deploy@b45cd3b]: KafkaConsumer is not connected error
10:32 mobrovac@tin: Synchronized wmf-config/jobqueue.php: Switch cross-wiki posting jobs to EventBus - T190327 (duration: 01m 18s)
10:31 ppchelko@tin: Finished deploy [cpjobqueue/deploy@b45cd3b]: Switch cross-wiki posting jobs for everything T175210 (duration: 01m 03s)
10:30 ppchelko@tin: Started deploy [cpjobqueue/deploy@b45cd3b]: Switch cross-wiki posting jobs for everything T175210
10:24 moritzm: upgrading video scalers in codfw to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
10:23 moritzm: upgrading snapshot hosts to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:56 jynus: stop and reimage db2043
09:47 moritzm: upgrading mw123[8-9], mw1266-mw1275 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:22 moritzm: upgrading mw1280-mw1290 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
07:50 jynus: stop and reimage db2050
07:42 jynus: stop and reimage db2057
07:21 marostegui: Deploy schema change on s8 codfw master (db2045) with replication, this will generate lags on codfw - T191519 T188299 T190148
07:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1078 after alter table (duration: 01m 19s)
05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1067 - T193835 (duration: 01m 19s)
05:10 marostegui: Deploy schema change on db1078 - T191519 T188299 T1901482
05:10 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1078 for alter table (duration: 01m 44s)
03:39 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 22 03:39:38 UTC 2018 (duration 7m 36s)
03:32 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 12m 10s)
02:48 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 12m 02s)
22:50 Krinkle: webperf1001: restart navtiming service, to test with new ipv6 capabilities
22:13 mutante: enabled IPv6 on webperf machines
21:59 foks: adding email to User:Quuxplusone - T194929
21:24 ottomata: granted User:CN=kafka_fundraising_client read permissions for group fundraising* on kafka-jumbo (for kafkatee webrequest consumption: kafka acls --add --allow-principal User:CN=kafka_fundraising_client --consumer --topic '*' --group 'fundraising*'
20:32 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 refs T191050 (duration: 01m 18s)
20:31 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 refs T191050
19:19 gehel: clearing cassandra snaphosts on maps* nodes to regain some space - T194966
17:13 gehel@tin: Finished deploy [wdqs/wdqs@e01dd03]: new WDQS GUI version (duration: 08m 44s)
17:04 gehel@tin: Started deploy [wdqs/wdqs@e01dd03]: new WDQS GUI version
16:53 fdans@tin: Finished deploy [analytics/refinery@16cb3be]: deploying new jar for upgraded ua parsing (duration: 05m 40s)
16:47 fdans@tin: Started deploy [analytics/refinery@16cb3be]: deploying new jar for upgraded ua parsing
15:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 with low load (duration: 01m 18s)
15:23 demon@tin: Pruned MediaWiki: 1.32.0-wmf.2 [keeping static files] (duration: 01m 56s)
15:05 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool fully db1109 (duration: 01m 18s)
14:57 demon@tin: Pruned MediaWiki: 1.31.0-wmf.30 (duration: 03m 19s)
14:50 jynus: stop and reimage db1074
14:46 demon@tin: Pruned MediaWiki: 1.31.0-wmf.29 (duration: 03m 29s)
14:42 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 and pool db1077 with 100% weight (duration: 01m 20s)
14:41 demon@tin: Pruned MediaWiki: 1.31.0-wmf.28 (duration: 03m 54s)
14:08 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1077 with low weight (duration: 01m 20s)
13:37 jynus: stop and reimage db2035
13:22 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: Increase edit rate limits on commons (T194864 ) (duration: 01m 24s)
12:51 jynus: stop and reimage db1077
10:20 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 20s)
10:18 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 22s)
09:57 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105 with full weight (duration: 01m 21s)
09:14 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 with full weight, repool db1105 with low weight (duration: 01m 21s)
08:02 marostegui: Deploy schema change on db1077 with replication, this will generate lags on labs - T191519 T188299 T1901482
08:02 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1077 for alter table (duration: 01m 22s)
07:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1123 after alter table (duration: 01m 21s)
05:54 marostegui: Restart MySQL on db2075 and db2092 for testing
05:50 marostegui: Restart MySQL on db1116 and db1120 for testing
05:40 marostegui: Deploy schema change on db1123 - T191519 T188299 T1901482
05:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1123 for alter table (duration: 01m 20s)
05:27 marostegui: Stop MySQL and reboot db1067 - T194852
05:11 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2064 - T195228 (duration: 01m 44s)
05:09 marostegui: Deploy schema change on s7 primary master (db1062) - T191519 T188299 T1901482
04:14 l10nupdate@tin: ResourceLoader cache refresh completed at Mon May 21 04:14:03 UTC 2018 (duration 7m 40s)
04:06 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 15m 28s)
03:08 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 14m 23s)
01:20 ottomata: bouncing main -> jumbo MirrorMaker with increased max.request.size - T189464
12:21 arturo: reboot labtestneutron2002.codfw.wmnet
10:50 tstarling@tin: Synchronized wmf-config/CommonSettings.php: Scribunto maxLangCacheSize (duration: 01m 23s)
08:24 ariel@tin: Finished deploy [dumps/dumps@5438d41]: sync after reimage of snapshot1007 (duration: 00m 03s)
08:24 ariel@tin: Started deploy [dumps/dumps@5438d41]: sync after reimage of snapshot1007
13:36 mholloway-shell@tin: Finished deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5
13:33 mholloway-shell@tin: Started deploy [kartotherian/deploy@58d2b0a]: Update maps/kartotherian/package to 7520fa5
08:31 volans: restarted pdfrender on scb1001
21:45 tgr: updated some OAuth consumer for a hackathon project: update oauth_registered_consumer set oarc_callback_url = 'http://localhost' where oarc_consumer_key = '2828bd9ca9bdcd81a960721819f25e90';
21:17 demon@tin: Finished deploy [gerrit/gerrit@a07d943]: quota plugin (duration: 00m 11s)
21:16 demon@tin: Started deploy [gerrit/gerrit@a07d943]: quota plugin
19:36 Reedy: tstarling manually loaded Tor Exit Nodes on wikitech
19:01 chasemp: unlock bawolff gerrit account
18:11 mutante: neon, netmon1002 - start ferm service
18:09 mutante: einsteinium: started ferm service
18:03 jynus: stop replication and start schema change on db1105
17:48 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 05s)
17:48 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
17:47 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 05s)
17:47 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
17:45 faidon@tin: Finished deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches (duration: 00m 32s)
17:45 faidon@tin: Started deploy [netbox/deploy@90164b3]: Update netbox to 2.2.4 + WMF patches
16:00 gehel: reduce replication of maps v4 keyspace to 3
15:59 reedy@tin: Synchronized php-1.32.0-wmf.4/extensions/VisualEditor/: Fix dialog (duration: 01m 19s)
15:45 reedy@tin: Synchronized wmf-config/throttle.php: throttling! (duration: 01m 22s)
15:44 marostegui: Stop MySQL and reboot db1067 - T194852
15:38 reedy@tin: Synchronized php-1.32.0-wmf.3/extensions/VisualEditor/: Fix dialog (duration: 01m 25s)
15:31 gehel: rolling restart of cassandra on maps1* (repair was started on each node, instead of sequentially)
15:23 gehel: clear cassandra snapshots on maps1002
13:59 marostegui: Manually fail disk #6 on db1066 - T194870
13:19 bawolff_: reset 2FA for Trizek_(WMF)
12:26 jynus: stop and reimage db2041
10:15 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1085 with low load (duration: 01m 20s)
09:47 marostegui: Stop MySQL on db1116 for testing
09:44 marostegui: Stop MySQL on db2092 for testing
09:43 hoo: Updated the Wikidata property suggester with data from Monday's JSON dump and applied the T132839 workarounds
09:33 gehel: cleared v3 snapshot on maps servers
09:25 jynus: stop and reimage db1085
09:24 gehel: drop v3 keyspace on cassandra maps (unused since migration to i18n)
09:01 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1085 (duration: 01m 20s)
08:32 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1105:s6 and other vslow hosts (duration: 01m 21s)
06:14 XioNoX: bumping eqsin-codfw link OSPF metric to 5000 (due to packet loss on link)
05:37 marostegui: Stop MySQL on db1120 to copy its content to db2075 - T190704
05:33 marostegui: Deploy schema change on dbstore1002:s3 - T191519 T188299 T190148
05:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1066 - T193847 (duration: 01m 22s)
05:24 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011
05:18 marostegui: Stop MySQL and reboot db1067 - T194852
01:34 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4: sync wmf.4 to deploy https://gerrit.wikimedia.org/r/#/c/433673/ (duration: 09m 54s)
01:27 twentyafterfour: syncing wmf.4 again to deploy https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050
00:19 mutante: rdb2004 - down in Icinga since >1d, nothing on console, dont see a SAL entry. powercycling
23:35 twentyafterfour: MediaWiki Train for 1.32.0-wmf.4 remains blocked by critical bugs, see T191050 for a list of blockers.
23:34 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 refs T191050 (duration: 01m 20s)
23:32 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3 refs T191050
23:29 twentyafterfour: rolling back
23:29 twentyafterfour: still seeing Notice: Undefined variable: nonce in /srv/mediawiki/php-1.32.0-wmf.4/includes/resourceloader/ResourceLoaderClientHtml.php on line 272
23:28 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 refs T191050 (duration: 01m 17s)
23:26 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4 refs T191050
23:22 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/: sync https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 (duration: 09m 54s)
22:53 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/433673/ refs T194900 T191050
19:46 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 (duration: 01m 20s)
19:44 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3
19:41 twentyafterfour: rolling back due to spike of undefined variable notices in resourceloader and ApiCSPReport.php
19:39 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.4 (duration: 01m 21s)
19:38 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.4
19:33 twentyafterfour: getting the train back on track. Starting with group1 to 1.32.0-wmf.4 right now, will do all wikis to wmf.4 after verifying that group1 looks stable.
19:28 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/Echo/: unbreak T194848 (duration: 01m 24s)
19:11 twentyafterfour: train is still blocked by T194848
17:23 arlolra: Updated Parsoid to fd49ab4 T194821 , T194687 )
17:15 arlolra@tin: Finished deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4
17:06 arlolra@tin: Started deploy [parsoid/deploy@091b891]: Updating Parsoid to fd49ab4
16:11 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 T174047 T194341
16:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1106 (duration: 01m 21s)
15:29 marostegui: Manually fail disk #6 on db1064 to get it replaced
15:28 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 with full weight (duration: 01m 21s)
15:00 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010
14:39 papaul: shutting down furud for shelves swap
14:35 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 https://phabricator.wikimedia.org/T174047 https://phabricator.wikimedia.org/T194341
14:17 marostegui: Manually fail disk #2 on db1064 to get it replaced
14:03 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1066 IP - T193847 (duration: 01m 21s)
13:58 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1066 IP - T193847 (duration: 01m 17s)
13:50 marostegui: Power off db1066 for a rack change - T193847
13:46 marostegui: Stop MySQL on db1066 for a rack change - T193847
13:45 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 for a rack change - T193847 (duration: 01m 21s)
13:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1105 (duration: 01m 21s)
13:36 jynus: restarted db1105 by mistake, turning it back on
13:15 jynus: stop and reimage db1106
12:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1106 (duration: 01m 20s)
12:50 marostegui: Deploy schema change on s3 codfw primary master (db2043) this will generate lag on codfw - T191519 T188299 T190148
11:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1090:3317 after alter table (duration: 01m 21s)
11:08 marostegui: Stop MySQL and poweroff db1067 - T194852
10:12 mobrovac@tin: Finished deploy [citoid/deploy@8a26508]: Update citoid to 2f35126 T179123 T185217 (duration: 02m 52s)
10:09 mobrovac@tin: Started deploy [citoid/deploy@8a26508]: Update citoid to 2f35126 T179123 T185217
09:30 reedy@tin: Synchronized wmf-config/throttle.php: Throttle for Barcelona Hackathon (duration: 01m 22s)
08:41 jynus: stop and reimage db2049
08:04 jynus: stop and reimage db2056
07:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1093 with low load (duration: 01m 20s)
07:25 elukey: bounced all the prometheus burrow exporters on kafkamon* hosts to refresh their metrics and drop old/expired cgroups
07:22 marostegui: Deploy schema change on db1090:3317 - T191519 T188299 T190148
07:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1090:3317 for alter table (duration: 01m 21s)
07:19 marostegui@tin: Synchronized wmf-config/db-codfw.php: Specify the sanitarium masters in codfw (duration: 01m 21s)
07:12 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1079 after alter table (duration: 01m 20s)
07:05 jynus: stop and reimage db1093
07:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1093 (duration: 01m 22s)
05:52 marostegui: Disable BBU auto-learn on new hosts - T192979
05:31 marostegui: Deploy schema change on db1079 with replication (this will generate lag on labs s7) - T191519 T188299 T190148
05:31 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1079 for alter table (duration: 01m 22s)
05:20 marostegui: Force BBU learn cycle on db1054 - T194867
05:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098:3317 after alter table (duration: 01m 48s)
05:09 marostegui: Deploy schema change on s4 primary master (db1068) - T191519 T188299 T190148
04:10 l10nupdate@tin: ResourceLoader cache refresh completed at Thu May 17 04:10:13 UTC 2018 (duration 7m 42s)
04:02 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 14m 08s)
03:04 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 14m 27s)
00:57 mutante: installing OS on webperf1002, webperf2002
21:58 cmjohnson1: swapping PEM 1 asw-c8-eqiad
20:13 marostegui: Force WriteBack policy on db1067 - T194852
19:38 twentyafterfour: The train for 1.32.0-wmf.4 is blocked by fatals in Echo extension. See T194848
19:28 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/extensions/CongressLookup: sync CongressLookup extension on wmf.4 refs T191050 (duration: 01m 22s)
17:35 milimetric@tin: Finished deploy [analytics/refinery@a205447]: Fix drop partitions script (duration: 15m 12s)
17:33 hoo@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RemexHtml on wikis with < 100 ns0 errors in high priority cats (T193685 ) (duration: 01m 22s)
17:20 milimetric@tin: Started deploy [analytics/refinery@a205447]: Fix drop partitions script
17:16 milimetric@tin: Started deploy [analytics/refinery@15be6ae]: Fix drop partitions script
16:38 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1088 with low weight (duration: 01m 21s)
16:34 marostegui@tin: Synchronized wmf-config/db-codfw.php: Change db1067 IP - T193835 (duration: 01m 21s)
16:29 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Change db1067 IP - T193835 (duration: 01m 17s)
16:22 elukey: upgrade burrow on kafkamon1001 from 1.0 to 1.1
16:18 marostegui: Power off db1067 for rack move - T193835
16:18 godog: move ms-be1036 for asw2-c-eqiad - T187962
16:03 godog: move ms-be1035 for asw2-c-eqiad - T187962
16:03 _joe_: installing cergen 0.2.2 on the puppetmaster frontends
15:50 andrewbogott: upgrading kernel, microcode and rebooting labnet1002
15:50 jynus: stop and reimage db1088
15:49 godog: move ms-be1034 for asw2-c-eqiad - T187962
15:36 godog: move ms-be1025 for asw2-c-eqiad - T187962
15:35 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1088 (duration: 03m 26s)
15:27 _joe_: uploading cergen 0.2.2 to stretch, jessie
15:18 godog: move ms-be1024 for asw2-c-eqiad - T187962
15:17 elukey: upgrade burrow from 1.0.0 to 1.1.0 on kafkamon* hosts
15:17 elukey: upload burrow 1.1.0 to stretch|jessie-wikimedia
15:17 bstorm_: labsdb1004 reboot and upgrade kernel to 4.9.0-0.bpo.6-amd64
15:10 godog: pool ms-fe1008 for asw2 move - T187962
15:02 marostegui: Stop MySQL on labsdb1004
15:01 marostegui: Stop MySQL on db1067 - T193835
14:54 godog: depool ms-fe1008 for asw2 move - T187962
14:50 moritzm: upgrading codfw job runners to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
14:49 godog: pool ms-fe1007 for asw2 move - T187962
14:19 filippo@neodymium: conftool action : set/pooled=no; selector: name=ms-fe1007.eqiad.wmnet
14:17 godog: depool ms-fe1007 for asw2 move - T187962
14:04 jynus: stop and reimage db2048 (s1 master)
13:56 marostegui: Restart mysql on db1116 for testing
13:54 marostegui: Deploy schema change on db1098:3317 - T191519 T188299 T190148
13:54 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1098:3317 for alter table (duration: 01m 20s)
13:48 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1101:3317 after alter table (duration: 01m 21s)
13:43 addshore: SWAT all done
13:42 addshore@tin: Synchronized php-1.32.0-wmf.3/autoload.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
13:41 addshore@tin: Synchronized php-1.32.0-wmf.3/includes/installer/DatabaseUpdater.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
13:40 ottomata: rolling restart of Kafka jumbo brokers to apply jdk.tls.namedGroups=secp256r1 https://phabricator.wikimedia.org/T182993
13:39 addshore@tin: Synchronized php-1.32.0-wmf.3/maintenance/: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 30s)
13:38 addshore@tin: Synchronized php-1.32.0-wmf.4/autoload.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
13:37 jynus: restart dbstore2002 for upgrade
13:36 addshore@tin: Synchronized php-1.32.0-wmf.4/includes/installer/DatabaseUpdater.php: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 21s)
13:34 addshore@tin: Synchronized php-1.32.0-wmf.4/maintenance/: SWAT: Deduplicate archive.ar_rev_id (duration: 01m 31s)
13:26 addshore@tin: Synchronized wmf-config/Wikibase.php: SWAT: Configure WikibaseLexeme after Repo & Client T191458 T194250 (duration: 01m 21s)
13:20 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/ContentTranslation/modules/ve-cx/init/ve.init.mw.CXTarget.js: SWAT: Fix mistake in 84caceee that causes exceptions with MT card T194811 (duration: 01m 21s)
13:17 jynus: restarting mariadb processes on dbstore2001 T194516
13:07 moritzm: rebooting deployment-ms-be03 for tests related to IBPB passthrough
12:56 jynus: stop and reimage db2039 (s6 master)
11:28 addshore: WikibaseLexeme slot done
11:10 moritzm: rebooting multatuli for some tests
11:01 addshore: addshore@terbium mwscript extensions/WikibaseLexeme/maintenance/createBlacklistedLexemes.php --wiki testwikidatawiki
10:57 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Enable WikibaseLexeme on test.wikidata.org T191458 (duration: 01m 20s)
10:47 addshore@tin: Synchronized wmf-config/Wikibase.php: Prepare Lexeme config for test.wikidata.org T194250 PT 2/2 (duration: 01m 21s)
10:45 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Prepare Lexeme config for test.wikidata.org T194250 PT 1/2 (duration: 01m 22s)
10:41 moritzm: uploaded linux 4.9.88-1+deb9u1~bpo8+1 to apt.wikimedia.org/jessie-wikimedia
10:34 jynus: stop and reimage db2046
10:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1067 as it will be moved to a different rack - T193835 (duration: 01m 21s)
10:14 marostegui: Drop unused tables msg_resource msg_resource_links from s2 - T194663
09:57 jynus: stop and reimage db2053
09:25 moritzm: upgrading video scalers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:22 marostegui: Deploy schema change on db1101:3317 - T191519 T188299 T190148
09:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1101:3317 for alter table (duration: 01m 21s)
09:16 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1094 after alter table (duration: 01m 21s)
08:54 jynus: restart dbproxy1002 for upgrade
08:48 moritzm: upgrading mw1221-mw1235 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
08:45 jynus: restart dbproxy1003 for upgrade
08:20 marostegui: Stop MySQL on db1120 on s2, s4, s6 and s7 to copy its content to db2075 - T190704
07:57 _joe_: depooled ULSFO from live traffic in dns
07:41 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2075 to convert it to temporary sanitarium (duration: 01m 20s)
07:36 moritzm: installing systemd updates from stretch SUA
07:33 marostegui: Deploy schema change on db1094 - T191519 T188299 T190148
07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1094 for alter table (duration: 01m 20s)
07:27 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1086 after alter table (duration: 01m 34s)
07:25 marostegui: Drop unused tables msg_resource msg_resource_links from s1 - T194663
07:18 moritzm: upgrading mw1240-mw1258 to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
06:56 moritzm: upgrading mwdebug servers to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
06:45 marostegui: Drop unused tables msg_resource msg_resource_links from s3 codfw - T194663
06:25 marostegui: Drop unused tables msg_resource msg_resource_links from s7 - T194663
06:19 marostegui: Drop unused tables msg_resource msg_resource_links from s4 - T194663
06:19 elukey: update analytics-in4 on cr1/cr2 eqiad to allow conf100[4-6] (new zookeeper hosts)
06:17 marostegui: Drop unused tables msg_resource msg_resource_links from s8 - T194663
06:15 marostegui: Drop unused tables msg_resource msg_resource_links from s6 - T194663
06:11 marostegui: Drop unused tables msg_resource msg_resource_links from s5 - T194663
05:54 elukey: removed acpi_power_meter manually from conf1004 (blacklisted module in puppet), Acpi errors in dmesg
05:22 marostegui: Deploy schema change on db1086 - T191519 T188299 T190148
05:21 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1086 for alter table (duration: 01m 23s)
05:14 marostegui: Deploy schema change on s2 primary master db1054 - T191519 T188299 T190148
04:43 kartik@tin: Finished deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 T191285 ) (duration: 03m 52s)
04:39 kartik@tin: Started deploy [cxserver/deploy@7e898c7]: Update cxserver to 112a1a1 T191285 )
03:46 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 16 03:46:12 UTC 2018 (duration 7m 41s)
03:38 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.4) (duration: 16m 26s)
02:40 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 08m 25s)
23:33 mutante: creating ganeti VM webperf2002.eqiad.wmnet on ganeti2004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390 )
23:11 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2139.codfw.wmnet
23:10 mutante: mw2139 - reimaged, scap pull, apache-fast-test baseurls from naos, repooled with confctl (T194426 )
23:06 mutante: creating ganeti VM webperf1002.eqiad.wmnet on ganeti1004 (link: private, row: A, cpus: 4, ram: 8, disk: 50) (T194390 )
22:39 samwilson@tin: Synchronized wmf-config/InitialiseSettings.php: Deploying GlobalPreferences T190425 (duration: 01m 21s)
22:21 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 to 1.32.0-wmf.4 refs T191050
22:18 twentyafterfour@tin: Synchronized php-1.32.0-wmf.4/includes/api/ApiLogin.php: fix syntax error (duration: 01m 39s)
21:34 twentyafterfour@tin: Finished scap: testwikis to 1.32.0-wmf.4 refs T191050 (duration: 66m 19s)
20:27 twentyafterfour@tin: Started scap: testwikis to 1.32.0-wmf.4 refs T191050
20:22 mutante: [radium:~] $ sudo apt-get autoremove
19:53 cwd: re-enabled process-control
19:42 twentyafterfour@tin: scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_770814178" --threads=10 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 47s)
19:40 cwd: disabled process-control for sql trigger refresh
19:39 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.4
19:05 mutante: mw2139 - wmf-auto-reimage --conftoool --new (because it got "Failed to icinga_downtime" and has a new mainboard (T194426 )
19:03 mutante: mw2139 - wmf-auto-reimage --conftoool --no-verify (T194426 )
17:27 twentyafterfour: branching 1.32.0-wmf.4 refs T191050
17:15 ottomata: rolling restart kafka-jumbo100[456]
16:36 elukey: rolling restart of aqs on aqs* nodes to pick up the new druid config
16:27 milimetric@tin: Finished deploy [analytics/refinery@679cf09]: Update partition drop script after schema change (duration: 07m 13s)
16:25 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: (no justification provided)
16:21 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: (no justification provided)
16:20 milimetric@tin: Started deploy [analytics/refinery@679cf09]: Update partition drop script after schema change
16:12 elukey: roll restart kafka on kafka-jumbo to pick up new zookeeper settings
16:11 joal@tin: Finished deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch (duration: 05m 47s)
16:05 joal@tin: Started deploy [analytics/aqs/deploy@a736558]: Deploying druid-configuration patch
15:52 elukey: rolling restart of hadoop master daemons to pick up new zookeeper settings
15:20 elukey: roll restart of Kafka Analytics to pick up new zookeeper settings
14:59 elukey: roll restart of kafka daemons on kafka100[1-3] to pick up new zookeeper settings and group.initial.rebalance.delay.ms = 10s
14:28 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change
14:28 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change
14:19 ottomata: temporarily disabling puppet on analytics1003 to run refine-eventbus after jumbo based camus eventbus import finishes
14:14 elukey: swap conf1001 with conf1004 in the zookeeper main eqiad's config + roll restart of the service
14:10 mobrovac@tin: Started restart [cpjobqueue/deploy@58935d5]: Restart after Kafka settings change
14:09 mobrovac@tin: Started restart [changeprop/deploy@e468d8e]: Restart after Kafka settings change
14:00 andrewbogott: rebooting labnet1001
13:50 elukey: roll restart of kafka main codfw (kafka200[1-3]) to pick up group.initial.rebalance.delay.ms = 10s
13:31 jynus: stop db2055 for reimage
13:09 chasemp: disable puppet for all openstack things in eqiad
13:07 andrewbogott: stopping nodepool and puppet on labnodepool1001 for T193579
12:59 andrewbogott: stopping puppet on labnet1001 and 1002, silencing icinga for T193579
12:42 jynus: stop db2060 for reimage
12:14 moritzm: uploaded intel-microcode 20180425
10:57 jynus: stop db2067 for reimage
10:49 joal@tin: Finished deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy (duration: 06m 45s)
10:46 jynus: stop db2066 for reimage
10:43 joal@tin: Started deploy [analytics/refinery@25abeec]: Fix for regular weekly deploy
10:16 jynus: stop db2065 for reimage
10:15 moritzm: installing uwsgi security update on graphite servers in eqiad
10:07 moritzm: installing php5 security updates on trusty
09:51 moritzm: upgrading API server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:47 jynus: stop and restart db2091 for upgrade
09:36 joal@tin: Finished deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy (duration: 05m 38s)
09:30 joal@tin: Started deploy [analytics/refinery@b2f4c3c]: Regular weekly deploy
09:26 jynus: stop and restart db2088 for upgrade
09:21 moritzm: upgrading app server canaries to HHVM 3,18.5+dfsg-1+wmf8+deb9u1
09:03 jynus: stop db2061 for reimage
08:42 jynus: stop db2068 for reimage
03:11 l10nupdate@tin: ResourceLoader cache refresh completed at Tue May 15 03:10:59 UTC 2018 (duration 7m 11s)
03:03 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 06m 32s)
02:33 bawolff@tin: Finished scap: Backport https://gerrit.wikimedia.org/r/#/c/433096/ - log js loads of unregistered user js subpages (duration: 56m 27s)
01:37 bawolff@tin: Started scap: Backport https://gerrit.wikimedia.org/r/#/c/433096/ - log js loads of unregistered user js subpages
01:17 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/433095/ log security channel (duration: 01m 02s)
23:39 bawolff@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/433089/ - re-enable sending csp logs to logstash (duration: 01m 01s)
23:36 bawolff@tin: Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/432334/ Use english messages in badpass log (duration: 01m 16s)
22:24 eileen: update process-control to 97fd42b3ab
21:54 mutante: mwdebug1001 - temp modifying apache 08-wikimedia.conf to test gerrit:429863
21:03 arlolra: Updated Parsoid to 945ed23 T194082 , T194083 , T194084 )
20:54 arlolra@tin: Finished deploy [parsoid/deploy@28fcc4e]: Updating Parsoid to 945ed23
20:42 arlolra@tin: Started deploy [parsoid/deploy@28fcc4e]: Updating Parsoid to 945ed23
20:09 bsitzmann@tin: Finished deploy [mobileapps/deploy@ccffa6b]: Update mobileapps to 39c16e4 T193440 T193439 T194065 ) (duration: 07m 49s)
20:02 bsitzmann@tin: Started deploy [mobileapps/deploy@ccffa6b]: Update mobileapps to 39c16e4 T193440 T193439 T194065 )
19:11 urandom: rolling cassandra restart, restbase dev environment
18:20 niharika29@tin: Synchronized wmf-config/InitialiseSettings.php: Create the eventcoordinator user group on enwiki - T193075 (duration: 01m 02s)
17:12 gehel@tin: Finished deploy [wdqs/wdqs@ef37bf7]: new WDQS GUI version (duration: 08m 43s)
17:03 gehel@tin: Started deploy [wdqs/wdqs@ef37bf7]: new WDQS GUI version
16:19 elukey: umount/remount /mnt/hdfs on stat1005 to pick up new openjdk upgrades
15:11 jynus: stop db2064 for reimage
14:56 marostegui: Drop unused table long_run_profiling from enwiki - T194661
14:45 moritzm: rebooting labtestnet2002 for some microcode/kernel tests
14:44 mobrovac@tin: Finished deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to}, take #2 (duration: 03m 58s)
14:40 mobrovac@tin: Started deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to}, take #2
14:39 mobrovac@tin: Finished deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to} - T163203 (duration: 23m 35s)
14:30 milimetric@tin: Finished deploy [analytics/refinery@541823e]: deploying refinery to update python logic for cron jobs (duration: 19m 46s)
14:30 otto@tin: Finished deploy [analytics/turnilo/deploy@9b2c8f0]: initial deploy (duration: 00m 28s)
14:29 otto@tin: Started deploy [analytics/turnilo/deploy@9b2c8f0]: initial deploy
14:19 moritzm: rebooting silver for some microcode/kernel tests
14:15 mobrovac@tin: Started deploy [restbase/deploy@75dc661]: API: Add /transform/list/tool/{tool}{/from}{/to} - T163203
14:10 milimetric@tin: Started deploy [analytics/refinery@541823e]: deploying refinery to update python logic for cron jobs
13:58 herron: added temporary iptables drop rules on fermium for IPs with many hits logged against the list subscribe rate limit
13:53 reedy@tin: Synchronized php-1.32.0-wmf.3/extensions/CirrusSearch: Partially revert deprecation of global namespace handling in prefix (duration: 01m 20s)
13:50 reedy@tin: Synchronized wmf-config/throttle.php: T194630 (duration: 01m 02s)
13:41 jynus: stop db2063 for reimage
13:18 marostegui: Deploy schema change on dbstore1002:s7 - T191519 T188299 T190148
13:08 akosiaris: remove ganeti2003, ganeti2007 from the ganeti cluster. stretch reimaging in progress
13:06 akosiaris: reboot ganeti2008 for kernel ugprade
12:38 kartik@tin: Finished deploy [cxserver/deploy@a7ef01b]: Update cxserver to 176b507
12:35 kartik@tin: Started deploy [cxserver/deploy@a7ef01b]: Update cxserver to 176b507
11:23 akosiaris: upload apertium-streamparser to apt.wikimedia.org/jessie-wikimedia/main T192978
10:17 moritzm: installing systemd updates from stretch SUA update
09:41 moritzm: installing gunicorn security updates
09:28 moritzm: installing ghostscript security updates on trusty
09:23 moritzm: installing libmad security updates
09:11 moritzm: installing wavpack security updates for stretch (jessie/trusty not affected)
08:41 moritzm: uploaded HHVM 3.18.5+dfsg-1+wmf8+deb9u1 to apt.wikimedia.org
07:57 marostegui: Deploy schema change on s7 codfw master (db2040) with replication, this will generate lag on codfw - T191519 T188299 T190148
07:55 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1091 after alter table (duration: 01m 02s)
06:38 elukey: rolling restart of cassandra on aqs* for openjdk-8 upgrades
06:35 moritzm: installing wget security updates on trusty (Debian already fixed)
06:16 marostegui: Drop unused flaggedrevs from s3 testwiki - T174801
05:22 marostegui: Deploy schema change on db1091 - T191519 T188299 T190148
05:22 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1091 for alter table (duration: 01m 03s)
03:18 l10nupdate@tin: ResourceLoader cache refresh completed at Mon May 14 03:18:25 UTC 2018 (duration 7m 25s)
03:11 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 16m 20s)
21:10 bawolff: Reset some botpasswords associated with T194204
20:09 bawolff: Deployed patch for T194605
06:50 foks: rm 2FA from User:Fatemi (at 6:38 PM UTC, Friday May 11)
21:28 herron: updated lists rate limit to 1 subscribe per 60 minutes as a temporary measure until problem requests slow down T194032
20:59 herron: deployed rate limiting for POST requests to mailman list subscription URIs https://gerrit.wikimedia.org/r/432168 T194032
20:27 chasemp: temp changes on fermium for T194032
19:26 chasemp: on fermium
19:26 chasemp: disable puppet and temp block a few IPs I believe are bad actors hammering mailman
18:21 chasemp: change 'Advertise this list when people ask what lists are on this machine' to no for cloud-admin-l and cloud-admin-feed
17:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1076 with full weight (duration: 01m 02s)
16:26 jynus: restarting mariadb@s8 at dbstore2001
16:20 jynus: removing x1 from dbstore2001
14:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1076 with low load, increase db1066 load (duration: 01m 02s)
14:30 jynus: reset labsdb proxies to its config defaults after rolling restart for upgrade
14:28 elukey: restart kafka brokers on kafka10[20,22,23] to pick up openjdk-7 security upgrades
14:14 andrewbogott: rebooting labvirt1001 for T194258
14:13 elukey: restart Hadoop daemons on analytics100[12] for openjdk security upgrades
13:39 jynus: stopping and restarting labsdb1009 for upgrade
13:32 jynus: reloading haproxy configuration for dbproxy1011 to point to labsdb1011
12:09 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1066 with low load (duration: 01m 02s)
11:48 Amir1: making change_tag_def table on all wikis (T194302 )
10:07 elukey: reimage analytics1052 to Debian Stretch (Hadoop Journal node)
09:30 jynus: stopping db1076 for maintenance
09:25 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1076 (duration: 01m 02s)
07:59 elukey: reimage analytics1035 to Debian Stretch
07:00 jynus: depool and upgrade/restart of dbproxy1011
02:09 ejegg: updated payments-wiki from 55d0808853 c81e25f8d3
23:18 ejegg: updated CiviCRM from 4c6a9c9c1c 5b8c868a00
23:09 thcipriani@tin: Synchronized dblists/categories-rdf.dblist: SWAT: Add wikis with more that 1000 categories to categories dump T194139 (duration: 01m 02s)
20:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2140.codfw.wmnet
20:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2140.co3dfw.wmnet
20:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2144.codfw.wmnet
20:24 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2202.codfw.wmnet
20:17 sbisson@tin: Finished deploy [kartotherian/deploy@06572bb]: Kartotherian: add fonts for Syriac and Inuktituk (duration: 03m 40s)
20:13 sbisson@tin: Started deploy [kartotherian/deploy@06572bb]: Kartotherian: add fonts for Syriac and Inuktituk
20:05 XenoRyet: updated payments-wiki from 4a8aada491 55d0808853
20:02 twentyafterfour: New branch 1.32.0-wmf.3 appears to be stable on all wikis. This completes the train for the week. Tune in again next week, same bat time, same bat channel. refs T191049
19:57 twentyafterfour@tin: rebuilt and synchronized wikiversions files: all wikis to 1.32.0-wmf.3
19:53 twentyafterfour: all deployment blockers resolved, proceeding to deploy mediawiki 1.32.0-wmf.3 to all wikis
19:14 twentyafterfour@tin: Synchronized php-1.32.0-wmf.3/includes/libs/rdbms/: deploy https://gerrit.wikimedia.org/r/#/c/432415/ refs T194308 (duration: 01m 23s)
18:56 maxsem@tin: Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432416/ labs only (duration: 01m 20s)
18:30 sbisson@tin: Finished deploy [tilerator/deploy@a5ec109]: Tilerator: load source and style from yaml (duration: 08m 45s)
18:21 sbisson@tin: Started deploy [tilerator/deploy@a5ec109]: Tilerator: load source and style from yaml
18:16 sbisson@tin: Finished deploy [kartotherian/deploy@3aa87ff]: Kartotherian: load style from yaml (duration: 06m 16s)
18:09 sbisson@tin: Started deploy [kartotherian/deploy@3aa87ff]: Kartotherian: load style from yaml
18:02 ppchelko@tin: Finished deploy [restbase/deploy@fb306e3]: Logging improvements (duration: 15m 33s)
17:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1123, Remove db1072 (duration: 01m 15s)
17:57 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1072 (duration: 01m 20s)
17:46 ppchelko@tin: Started deploy [restbase/deploy@fb306e3]: Logging improvements
15:54 elukey: drain and reimage analytics1028 to Debian Jessie (Hadoop Journal node)
15:00 elukey: rolling restart of Hadoop HDFS datanodes on analytics workers to pick up the new openjdk-8 security upgrades
14:53 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1123 with low weight (duration: 01m 20s)
14:38 elukey: rolling restart of Hadoop Yarn nodemanagers on analytics worker nodes for openjdk-8 security upgrades
14:32 jynus: restarting db1123
14:28 otto@tin: Started restart [eventlogging/eventbus@aa9eb2c]: apply log level changes
14:27 ottomata: rolling restart of eventbus service to apply new log level settings
14:07 elukey: restart hive/oozie Hadoop daemons on analytics1003 for openjdk-8 upgrades
13:56 mutante: mw2139,mw2140,mw2144,mw2202 - reinstall with --no-verify - last few special cases that didnt have puppet certs or failed before - after that all appservers done
13:46 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2138.codfw.wmnet
13:43 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2137.codfw.wmnet
13:42 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2136.codfw.wmnet
13:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2135.codfw.wmnet
13:31 elukey: rolling restart of kafka on kafka-jumbo1* for openjdk-8 security upgrades
13:25 zeljkof: EU SWAT finished
13:19 elukey: reimage analytics1029 to Debian Stretch
13:15 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: cawiki: remove gendered namespace aliases, already on MW core (T113616) (duration: 01m 20s)
13:06 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable TemplateStyles for nowiki (T193786) (duration: 01m 20s)
12:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add db1123 (duration: 01m 19s)
12:07 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1123 (duration: 01m 19s)
12:04 Reedy: that was for "Add throttle rule for Netherlands Hackathon 2018 - Women Tech Storm"
12:04 reedy@tin: Synchronized wmf-config/throttle.php: (no justification provided) (duration: 01m 48s)
11:46 elukey: reimage analytics1030/31 to Debian Stretch
11:07 volans: updated puppet compiler facts
09:07 jynus: stop db1072 for maintenance
08:10 jynus: shutdown and restart labsdb1010 for upgrade
07:12 jynus: depool labsdb1010 from wikireplicas for maintenace
03:04 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 15m 35s)
02:51 mutante: mw2137,mw2138,mw22139 - reinstall with strech (last message was wrong, already running)
02:48 mutante: mw2136,mw2137,mw22138 - reinstall with stret h
02:44 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2141.codfw.wmnet
02:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2142.codfw.wmnet
02:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2143.codfw.wmnet
01:16 twentyafterfour@tin: Synchronized php-1.32.0-wmf.3/includes/libs/rdbms: sync https://gerrit.wikimedia.org/r/#/c/432297/ refs T194308 T191049 (duration: 01m 24s)
00:33 mutante: mw2135, mw2136, mw2137 - reinstall with stretch, depooled, downtimed
00:28 hoo@tin: Synchronized php-1.32.0-wmf.3/extensions/Wikibase/lib/WikibaseLib.php: Remove wgHooks entry for GalleryGetModes (T194316 ) (duration: 01m 16s)
00:26 hoo@tin: Synchronized php-1.32.0-wmf.2/extensions/Wikibase/lib/WikibaseLib.php: Remove wgHooks entry for GalleryGetModes (T194316 ) (duration: 01m 20s)
00:16 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Fix notices about missing index "max" (duration: 01m 20s)
00:06 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on arwiki (T192498 ) (duration: 01m 20s)
00:01 twentyafterfour: no phabricator deployment this evening.
23:53 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable ORES on cawiki, lvwiki, huwiki (T192501 , T192499 , T192496 ) (duration: 01m 29s)
23:24 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1001 (duration: 00m 03s)
23:24 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1001
23:13 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on some wikibooks wikis (T192821 ) (duration: 01m 21s)
23:06 awight@tin: Finished deploy [ores/deploy@bf1e2b1]: ORES: drafttopic (duration: 25m 51s)
23:02 mutante: mw2141,mw2143,mw2142 - reinstalling with stretch - mw2144: puppet cert not found
22:59 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2147.codfw.wmnet
22:58 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2146.codfw.wmnet
22:56 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2145.codfw.wmnet
22:40 awight@tin: Started deploy [ores/deploy@bf1e2b1]: ORES: drafttopic
22:35 awight@tin: Finished deploy [ores/deploy@1b13ef1]: ORES: drafttopic (duration: 03m 15s)
22:33 addshore@tin: Synchronized wmf-config/extension-list: extension-list Add WikibaseLexeme to extension-list (duration: 01m 19s)
22:32 awight@tin: Started deploy [ores/deploy@1b13ef1]: ORES: drafttopic
22:28 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Add default edit rate limit of 90 edits/minute for all users (except wikidata) (duration: 01m 19s)
22:20 awight@tin: Finished deploy [ores/deploy@2a09939]: ORES: force git-lfs install (take 3) (duration: 03m 05s)
22:18 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T184745 T191459 BETA ONLY Enable WikibaseLexeme on BETA wikidatawiki (duration: 01m 19s)
22:17 awight@tin: Started deploy [ores/deploy@2a09939]: ORES: force git-lfs install (take 3)
22:09 awight@tin: Finished deploy [ores/deploy@c0db102]: ORES: force git-lfs install (take 2) (duration: 03m 15s)
22:08 addshore@tin: Synchronized wmf-config/Wikibase-labs.php: T184745 BETA ONLY WikibaseLexeme config (duration: 01m 20s)
22:06 awight@tin: Started deploy [ores/deploy@c0db102]: ORES: force git-lfs install (take 2)
22:06 addshore@tin: Synchronized wmf-config/InitialiseSettings-labs.php: T184745 BETA ONLY WikibaseLexeme config (duration: 01m 19s)
22:05 awight@tin: Finished deploy [ores/deploy@c0db102]: ORES: force git-lfs install (duration: 02m 50s)
22:03 awight@tin: Started deploy [ores/deploy@c0db102]: ORES: force git-lfs install
21:27 ejegg: re-started refund queue consumer
21:25 ejegg: updated CiviCRM from ca8acdccf6 4c6a9c9c1c
21:20 reedy@tin: Synchronized php-1.32.0-wmf.3/includes/DefaultSettings.php: Add default edit rate limit of 90 edits/minute for all users (duration: 01m 20s)
21:18 reedy@tin: Synchronized php-1.32.0-wmf.2/includes/DefaultSettings.php: Add default edit rate limit of 90 edits/minute for all users (duration: 01m 20s)
20:59 ejegg: disabled refund queue consumer
20:30 maxsem@tin: Synchronized php-1.32.0-wmf.2/extensions/CongressLookup/: https://gerrit.wikimedia.org/r/#/c/432146/ (duration: 01m 20s)
20:28 maxsem@tin: Synchronized php-1.32.0-wmf.3/extensions/CongressLookup/: https://gerrit.wikimedia.org/r/#/c/432146/ (duration: 01m 19s)
20:21 arlolra: Updated Parsoid to 5ce2608 T194081 , T188118 )
20:17 arlolra@tin: Finished deploy [parsoid/deploy@181e3b1]: Updating Parsoid to 5ce2608
20:16 ejegg: restarted refund queue consumer
20:08 arlolra@tin: Started deploy [parsoid/deploy@181e3b1]: Updating Parsoid to 5ce2608
20:01 twentyafterfour@tin: Synchronized php: group1 wikis to 1.32.0-wmf.3 (duration: 01m 20s)
20:00 twentyafterfour: group1 to 1.32.0-wmf.3 refs T191049
20:00 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group1 wikis to 1.32.0-wmf.3
20:00 ejegg: updated CiviCRM from 4ff35ad4 ca8acdccf6
19:53 ottomata: rolling restart eventbus service to deploy logstash config
19:51 ejegg: updated SmashPig standalone from 99585c8084 2b4186715e
19:51 otto@tin: Finished deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230 (duration: 01m 17s)
19:49 otto@tin: Started deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230
19:49 mutante: graphite1001/2001 - rm check_uwsgi-coal NRPE check config, reloading nagios-nrpe-server (T194283 )
19:48 otto@tin: Finished deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230 (duration: 00m 15s)
19:48 otto@tin: Started deploy [eventlogging/eventbus@aa9eb2c]: logstash T193230
19:38 otto@tin: Finished deploy [eventlogging/eventbus@c70e8c5]: logstash - T193230 (duration: 03m 33s)
19:36 mutante: graphite1001, graphite2001 - deleting uwsgi-coal and coal sytemd unit files; systemctl daemon-reload (T194283 )
19:35 otto@tin: Started deploy [eventlogging/eventbus@c70e8c5]: logstash - T193230
19:26 mutante: mw2145, mw2146, mw2147 - reinstall with stretch, depooled, downtimed
19:23 Krinkle: Stop and disable coal-web (uwsgi-coal) service on graphite1001/graphite2001 (T194283 )
19:23 Krinkle: Disable coal service on graphite1001/graphite2001 (T194283 )
18:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2212.codfw.wmnet
18:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2213.codfw.wmnet
18:37 sbisson@tin: Finished deploy [tilerator/deploy@a86f8f8]: Make tilerator store up to zoom 15 (duration: 06m 36s)
18:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2214.codfw.wmnet
18:31 sbisson@tin: Started deploy [tilerator/deploy@a86f8f8]: Make tilerator store up to zoom 15
18:29 sbisson@tin: Finished deploy [kartotherian/deploy@ef61ad7]: Make kartotherian serve up to z15 (duration: 02m 20s)
18:27 sbisson@tin: Started deploy [kartotherian/deploy@ef61ad7]: Make kartotherian serve up to z15
18:22 imarlier@tin: Finished deploy [performance/coal@8e57e4a]: Deploy only to webperf (duration: 00m 06s)
18:22 imarlier@tin: Started deploy [performance/coal@8e57e4a]: Deploy only to webperf
18:19 moritzm: installing jenkins security updates on releases*
18:04 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/432030/ (duration: 01m 21s)
18:00 maxsem@tin: Finished scap: Deploy CongressLookup on testwiki T194230 (duration: 105m 19s)
16:56 ottomata: disabled 0.9 MirrorMaker on kafka102[023], enabled 1.x MirrorMaker on kafka-jumbo*
16:39 gehel: restarting blazegraph and updater on wdqs1003
16:15 maxsem@tin: Started scap: Deploy CongressLookup on testwiki T194230
15:56 urandom: starting revision cleanup job, wikipedia_T_mobile__ng_lead keyspace - T192689
15:31 thcipriani: upgrading jenkins on contint2001/contint1001
15:26 vgutierrez: Replacing lvs1003 with lvs1016 - T184293
15:22 mutante: mw2212,mw2213,mw2214 - reinstall with stretch
14:45 ppchelko@tin: Finished deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. T167039 (duration: 00m 34s)
14:45 ppchelko@tin: Started deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. T167039
14:41 ppchelko@tin: Finished deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. T167039 (duration: 00m 53s)
14:40 ppchelko@tin: Started deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. T167039
14:27 ejegg: disabled refund queue consumer
13:59 ottomata: beginning upgrade of Kafka main-eqiad cluster from 0.9.0.1 to 1.1.0 - T167039
13:55 milimetric@tin: Finished deploy [analytics/refinery@a5a8cbc]: Renaming geoeditors druid datasource (duration: 05m 27s)
13:49 milimetric@tin: Started deploy [analytics/refinery@a5a8cbc]: Renaming geoeditors druid datasource
13:28 zeljkof: EU SWAT finished
13:27 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mapframe on all but a few wikis (T191585) (duration: 01m 20s)
13:18 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove unused wgKartographerDfltStyle setting (T191655) (duration: 01m 20s)
13:10 dcausse@tin: Synchronized wmf-config/InitialiseSettings.php: T192064 [cirrus] Increase the number of shards for wikidatawiki_content, enwiki_general (duration: 01m 20s)
13:08 elukey: reimage analytics103[2,3] to Debian Stretch
13:05 milimetric@tin: Finished deploy [analytics/refinery@640bc35]: Renaming geoeditors druid datasource (duration: 06m 11s)
12:59 milimetric@tin: Started deploy [analytics/refinery@640bc35]: Renaming geoeditors druid datasource
11:26 moritzm: installing wget security updates on Debian systems
11:14 moritzm: reimage mw2206 (earlier reimage failed since the host lacked a puppet cert)
11:03 moritzm: updated jenkins packages
10:55 moritzm: reimaging mw2246 to stretch (video scaler with a deprecated one-off partman recipe)
10:50 demon@tin: Synchronized wmf-config/missing.php: remove vendor dep (duration: 01m 20s)
10:45 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1072 (duration: 01m 19s)
10:40 demon@tin: Synchronized multiversion/getMWVersion: remove vendor dep (duration: 03m 27s)
10:30 godog: cycle-load edac kernel modules for scb1002 to reset counters
10:28 godog: cycle-load edac kernel modules for cp1068 to reset counters
10:02 demon@tin: Synchronized docroot/search.wikimedia.org/index.php: improve 5xx/4xx error handling (duration: 01m 27s)
09:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1064 with full weight (duration: 01m 27s)
08:52 moritzm: reimaging mw1336, mw1337 (job runners) to stretch
08:19 no_justification: gerrit: restarting for version bump 2.14.7 -> 2.14.8
08:18 jynus: stop and upgrade db1053
08:17 demon@tin: Finished deploy [gerrit/gerrit@c421c91]: 2.14.7 -> 2.14.8 (duration: 00m 11s)
08:17 demon@tin: Started deploy [gerrit/gerrit@c421c91]: 2.14.7 -> 2.14.8
07:12 moritzm: reimaging mw2152 to stretch (video scaler with a deprecated one-off partman recipe)
07:10 moritzm: reimaging mw2162 to stretch (last jessie job runner in codfw)
07:05 moritzm: reimaging mw1334, mw1335 (job runners) to stretch
06:04 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2211.codfw.wmnet
06:01 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2210.codfw.wmnet
06:00 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2209.codfw.wmnet
05:26 marostegui: Stop MySQL on db2092 to do some clean ups
05:19 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1103:3314 after alter table (duration: 01m 36s)
05:19 marostegui: Stop slave on db1116:s3 to do some gtid cleanups and tests
04:02 l10nupdate@tin: ResourceLoader cache refresh completed at Wed May 9 04:02:56 UTC 2018 (duration 7m 26s)
03:55 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.3) (duration: 16m 34s)
02:57 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 09m 06s)
02:09 maxsem@tin: Synchronized wmf-config: Preparation for T194230 (duration: 01m 15s)
02:06 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: Preparation for T194230 (duration: 01m 16s)
02:02 maxsem@tin: Synchronized php-1.32.0-wmf.2/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 17s)
02:00 maxsem@tin: Synchronized php-1.32.0-wmf.3/extensions/CongressLookup/: Preparation for T194230 (duration: 01m 22s)
01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2211.codfw.wmnet
01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2210.codfw.wmnet
01:36 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2209.codfw.wmnet
01:36 XioNoX: progressively push updated BGP_sanitize_in prefix-length-range to routers - T190317
01:33 mutante: mw2209,mw2210,mw2211 - reinstall wtih stretch
01:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2204.codfw.wmnet
01:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2203.codfw.wmnet
01:22 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2208.codfw.wmnet
01:20 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2207.codfw.wmnet
01:19 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2205.codfw.wmnet
00:54 ejegg: running refund queue consumer overtime
00:35 maxsem@tin: Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/432022/ - noop in prod (duration: 01m 20s)
23:58 thcipriani@tin: Synchronized php-1.32.0-wmf.2/extensions/WikibaseQualityConstraints/src/ConstraintCheck/Helper/LoggingHelper.php: SWAT: Do not try to access null message message key T194140 (duration: 01m 32s)
23:42 XioNoX: progressively push BGP_sanitize_in as-path too-many-hops to routers - T190317
23:25 ejegg: updated CiviCRM from d9d412e496 4ff35ad4bb
23:22 thcipriani@tin: Synchronized wmf-config: SWAT: Add string and external-id types to Wikibase indexing T163642 T99899 (duration: 01m 26s)
23:12 XioNoX: lowering ospf metric of ulsfo-codfw to 390
22:53 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1002 to master (duration: 00m 19s)
22:52 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1002 to master
22:49 awight@tin: Finished deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002 (duration: 01m 59s)
22:47 awight@tin: Started deploy [ores/deploy@5b27205]: Deploy LFS files to ores1002
22:35 XioNoX: remove PREFERRED-TRANSIT Tele2-DTAG from esams/knams routers
22:05 XioNoX: progressively push updated BGP_sanitize_in bogon ASN filters to routers - T190317
21:59 twentyafterfour: MediaWiki train for 1.32.0-wmf.3 group0 is complete. Will resume with group1 tomorrow, same bat time, same bat channel (refs T191049 )
21:45 twentyafterfour@tin: rebuilt and synchronized wikiversions files: group0 wikis to 1.32.0-wmf.3
21:36 eileen: civicrm revision changed from 81e54c850d d9d412e496 96fbded693
21:09 twentyafterfour@tin: Finished scap: testwikis wikis to 1.32.0-wmf.3 (duration: 114m 21s)
20:59 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2204.codfw.wmnet
20:58 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2203.codfw.wmnet
20:24 milimetric@tin: Finished deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors (duration: 07m 07s)
20:17 milimetric@tin: Started deploy [analytics/refinery@2a4633c]: Deploying renamed geowiki jobs as geoeditors
20:02 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2201.codfw.wmnet
20:01 mutante: mw2205,mw2206,mw2207 - reinstalling with stretch - mw2202 - wmf-auto-reimage failed: Timeout of 60 minutes reached waiting for reboot
19:15 twentyafterfour@tin: Started scap: testwikis wikis to 1.32.0-wmf.3
19:14 twentyafterfour: testwikis to 1.32.0-wmf.3 - https://gerrit.wikimedia.org/r/#/c/431821/ refs T191049
19:11 twentyafterfour: updated mediawiki changelog https://www.mediawiki.org/wiki/MediaWiki_1.32/wmf.3/Changelog refs T191049
18:55 mutante: mw2202, mw2203, mw2204 - reinstall with stretch
18:52 marostegui: Manually fail disk #7 on db1073 to get it replaced
18:22 mutante: mwmaint1001 - rebooting
18:18 twentyafterfour: Branching MediaWiki master to wmf/1.32.0-wmf.3 refs T191049
18:08 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2252.codfw.wmnet
18:07 andrew@tin: Finished deploy [horizon/deploy@9245ca9]: rolling out member dashboard (duration: 03m 18s)
18:06 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2251.codfw.wmnet
18:03 andrew@tin: Started deploy [horizon/deploy@9245ca9]: rolling out member dashboard
17:00 bawolff: Clear botpassword throttle for User:TaxonBot (T194160 )
16:45 thcipriani@tin: Synchronized README: Testing Scap 3.8.1-1 (duration: 01m 02s)
16:43 godog: upload scap 3.8.1-1 - T127762
16:40 XioNoX: re-pooling lvs2001 - T193677
16:36 mutante: mwmaint1001 - reinstalling one more time after proxysql issues are resolved, PXE booting (T192092 )
16:27 mutante: mw2251,mw2252,mw2201 - reinstall with stretch
16:22 herron: cleared low count edac counters on hosts mw2205 dbstore1002 db1051 elastic1029 T183177
16:19 urandom: force (split) compaction of wikipedia_T_mobile__ng_lead.data, restbase1016 - T192689
16:15 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2223.codfw.wmnet
16:14 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2222.codfw.wmnet
16:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2215.codfw.wmnet
16:09 XioNoX: failing traffic over lvs2004 - T193677
16:04 ppchelko@tin: Finished deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039 (duration: 01m 03s)
16:03 ppchelko@tin: Started deploy [changeprop/deploy@e468d8e]: Allow protocol version negotiation. Codfw only. T167039
16:01 ppchelko@tin: Finished deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039 (duration: 00m 42s)
16:01 ppchelko@tin: Started deploy [cpjobqueue/deploy@58935d5]: Allow protocol version negotiation. Codfw only. T167039
15:53 mutante: switching performance.wikimedia.org from graphite to webperf backends - running puppet on cache::misc servers (T158837 )
15:46 demon@tin: Pruned MediaWiki: 1.32.0-wmf.1 [keeping static files] (duration: 01m 47s)
15:30 XioNoX: starting pybal on lvs2001 - T193677
15:26 godog: (un)load edac kernel modules on thumbor1004 to test resetting counters - T183177
15:09 XioNoX: stopping pybal on lvs2001 - T193677
15:06 ottomata: beginnng Kafka upgrade of main-codfw: T167039
14:53 XioNoX: re-enable pybal on lvs2004 - T193677
14:48 XioNoX: disabling pybal on lvs2004 - T193677
14:37 mutante: LDAP: added 'sbailey' to group 'wmf' (T194091 )
14:19 ppchelko@tin: Started restart [changeprop/deploy@7e86531]: Restart changeprop to try forcing it rebalancing topics
14:15 mutante: mw2215,mw2222,mw2223 - reinstalling with stretch
13:43 zeljkof: EU SWAT finished
13:42 zfilipin@tin: Synchronized php-1.32.0-wmf.2/extensions/Translate: SWAT: Refactor TranslationUpdateJob to use only primitive types for parameters (T192111) (duration: 01m 11s)
13:25 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable maps i18n everywhere (T191655) (duration: 01m 00s)
13:14 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable AdvancedSearch BetaFeature on all wikis (T193182) (duration: 01m 00s)
13:02 marostegui: Manually fail disk #9 on db1073 to get it replaced
12:20 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1055 (duration: 00m 59s)
12:19 moritzm: reimaging mw2159, mw2160, mw2161 (job runners) to stretch
12:18 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove db1055 (duration: 00m 59s)
12:17 moritzm: upgrading app servers in beta to wikidiff 1.6.0 (T190717 )
12:16 moritzm: upgrading app servers in beta to
12:02 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1064 with low load (duration: 00m 59s)
11:36 marostegui: Deploy schema change on db1103:3314 - T191519 T188299 T190148
11:36 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1103:3314 for alter table (duration: 00m 59s)
11:18 marostegui@tin: Synchronized wmf-config/db-codfw.php: Really depool db2092 (duration: 00m 53s)
10:29 moritzm: reimaging mw1347, mw1348 (API servers) to stretch (last two remaining API servers in eqiad)
10:22 jynus: stop mariadb on db1055 to clone it to db1064
10:15 moritzm: reimaging mw1310, mw1311 (job runners) to stretch
09:58 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1055 (duration: 00m 54s)
09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1121 after alter table (duration: 01m 00s)
09:20 elukey: forced a BBU re-learn cycle on analytics1032
09:17 gehel: reducing replication factor on cassandra v3 (unused) keyspace for maps
08:56 moritzm: reimaging mw1345, mw1346 (API servers) to stretch
08:30 moritzm: reimaging mw2156, mw2157, mw2158 (job runners) to stretch
08:27 moritzm: reimaging mw1308, mw1309 (job runners) to stretch
08:03 marostegui: Stop MySQL on db1116 to transfer its content to db2092 - T190704
07:59 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2092 T190704 (duration: 00m 57s)
07:53 elukey: second attempt to remove the cassandra-metrics-collector (+ cleanup) from aqs*
07:30 jynus: cleaning up maintenance hosts (terbium, etc.) from tendril maintenance files
06:51 marostegui: Stop MySQL on db1060 as it will be decommissioned - T193732
06:50 moritzm: reimaging mw1313, mw1343, mw1344 to stretch
06:26 marostegui@tin: Synchronized wmf-config/db-codfw.php: Remove db1060 from config - T193732 (duration: 01m 01s)
06:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Remove db1060 from config - T193732 (duration: 00m 59s)
06:05 marostegui: Read_only=off on db1069 to finish with the x1 failover
06:05 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Promote db1069 as new x1 master (duration: 01m 00s)
06:00 marostegui: Set db1055 ready only
06:00 marostegui: Start x1 failover
05:41 marostegui: Move db2034 under db1069 for x1 failover - T186320
05:36 marostegui: Move dbstore1002:x1 under db1069 for x1 failover - T186320
05:29 marostegui: Disable puppet on db1055 and db1069 before x1 failover - T186320
05:28 marostegui: Disable gtid on db1069 an db2034 before x1 failover - T186320
05:26 marostegui: Deploy schema change on db1121 with replication (this will generate lag on labs on s4) - T191519 T188299 T190148
05:26 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1121 for alter table (duration: 01m 00s)
05:19 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1011 - https://phabricator.wikimedia.org/T174047
05:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1097:3314 after alter table (duration: 01m 00s)
04:27 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2221.codfw.wmnet
04:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2220.codfw.wmnet
04:24 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2219.codfw.wmnet
02:33 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 05m 45s)
00:12 ejegg: updated CiviCRM from 9752607052 81e54c850d
23:41 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2218.codfw.wmnet
23:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2217.codfw.wmnet
23:39 mutante: mw2219,mw2220,mw2221 - reinstall with stetch
23:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2216.codfw.wmnet
22:31 bstorm_: labsdb1009,labsdb1010,labsdb1011 are now on up-to-date views per T174047
22:23 ppchelko@tin: Started restart [changeprop/deploy@7e86531]: Restart changeprop to try forcing it rebalancing topics
21:28 mutante: mw2216,mw2217,mw2218 - wmf-auto-reimage --conftool , reinstall with stretch
21:25 XioNoX: re-pool eqsin - T193897
20:48 imarlier@tin: Started restart [performance/coal@50fe0dd]: Restart coal-web service everywhere, hopefully
20:48 arlolra: Updated Parsoid to 6e38948 T192909 )
20:41 arlolra@tin: Finished deploy [parsoid/deploy@cd5e875]: Updating Parsoid to 6e38948
20:30 otto@tin: Finished deploy [statsv/statsv@c186340]: Configure api.version via CLI opt -- prep for Kafka main upgrade T167039 (duration: 00m 05s)
20:30 otto@tin: Started deploy [statsv/statsv@c186340]: Configure api.version via CLI opt -- prep for Kafka main upgrade T167039
20:29 arlolra@tin: Started deploy [parsoid/deploy@cd5e875]: Updating Parsoid to 6e38948
20:25 bsitzmann@tin: Finished deploy [mobileapps/deploy@e20f23d]: Update mobileapps to c1f4de6 T191538 ) (duration: 06m 09s)
20:19 bsitzmann@tin: Started deploy [mobileapps/deploy@e20f23d]: Update mobileapps to c1f4de6 T191538 )
19:58 XioNoX: removing onboard ports license from cr1-eqsin config - T193897
17:34 bawolff@tin: Synchronized php-1.32.0-wmf.2/extensions/LoginNotify/includes/Hooks.php: https://gerrit.wikimedia.org/r/#/c/431611/ Do not send loginnotify emails for throttled logins (duration: 01m 08s)
17:29 cmjohnson1: updating f/w lvs1016
17:08 gehel@tin: Finished deploy [wdqs/wdqs@bd4b3ed]: new wdqs GUI, updater and blazegraph (duration: 05m 18s)
17:03 gehel@tin: Started deploy [wdqs/wdqs@bd4b3ed]: new wdqs GUI, updater and blazegraph
16:57 elukey: executed sudo megacli -AdpBbuCmd -BbuLearn -aALL -NoLog on analytics1032 - BBU alerts flapping
16:37 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1011 - https://phabricator.wikimedia.org/T174047
16:26 sbisson@tin: Finished deploy [kartotherian/deploy@9935fdb]: Kartotherian: remove temporary and unused osm-intl-i18n source (duration: 03m 28s)
16:23 sbisson@tin: Started deploy [kartotherian/deploy@9935fdb]: Kartotherian: remove temporary and unused osm-intl-i18n source
15:34 marostegui: Deploy schema change on db1097:3314 - T191519 T188299 T190148
15:34 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1097:3314 for alter table (duration: 01m 00s)
15:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1084 after alter table (duration: 00m 57s)
14:59 marostegui: Reload haproxy on dbproxy1010 to repool labsdb1010 - https://phabricator.wikimedia.org/T174047
14:30 imarlier@tin: Finished deploy [performance/coal@50fe0dd]: Coal version that uses the graphite API to fetch data, instead of reading directly from whisper files (duration: 00m 13s)
14:30 imarlier@tin: Started deploy [performance/coal@50fe0dd]: Coal version that uses the graphite API to fetch data, instead of reading directly from whisper files
14:13 herron: upgraded prometheus-jmx-exporter to 0.3.0-1 on puppetdb servers
14:12 herron: puppetdb updates complete — re-enabling puppet agents
14:01 herron: temporarily disabling puppet agents for puppetdb security update
13:36 zeljkof: EU SWAT finished
13:35 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set wgKartographerEnableMapFrame to true for mrwiki (T193371) (duration: 01m 00s)
13:16 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Make test and test2 using maps i18n correctly (T191655) (duration: 01m 00s)
12:55 sbisson@tin: Finished deploy [kartotherian/deploy@425f279]: Kartotherian: stop using tilerator_storage_id; Add babel upstream of osm-intl (duration: 05m 41s)
12:50 sbisson@tin: Started deploy [kartotherian/deploy@425f279]: Kartotherian: stop using tilerator_storage_id; Add babel upstream of osm-intl
12:27 moritzm: reimaging mw1333 to stretch (last app server in eqiad)
11:28 moritzm: installing Java security updates on elastic* hosts
10:59 moritzm: reimaging mw1330, mw1331, mw1332 (app servers) to stretch
10:50 moritzm: depooled service nginx for mw1221-mw1231 (API servers)
10:10 jdrewniak@tin: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 00m 59s)
10:09 jdrewniak@tin: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 01m 00s)
09:52 moritzm: reimaging mw1305, mw1306 (job runners) to stretch
09:52 elukey: stop graphite cassandra-metrics-collector on aqs* (touch /etc/cassandra-metrics-collector/disable)
09:46 moritzm: rolling restart of Kibana logstash nodes to pick up Java security updates
09:41 marostegui: Manually enable innodb_strict_mode on db1084 - T150949
09:30 ema: cp-text/upload: start varnish upgrades to 5.1.3-1wm8 T192368
09:30 marostegui: Deploy schema change on db1084 - T191519 T188299 T190148
09:30 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1084 for alter table (duration: 01m 03s)
09:25 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1081 after alter table (duration: 01m 02s)
09:02 moritzm: reimaging mw1275 (T192902 )
08:58 mobrovac@tin: Started restart [recommendation-api/deploy@ac66089]: Use the internal WDQS cluster LVS - T190266
08:56 elukey: drain + reimage analytics103[7,8] to Debian Stretch
08:30 godog: eqiad-prod: more weight to ms-be104[0-3] - T190081
08:19 moritzm: rolling restart of logstash to pick up Java security updates
08:19 marostegui: Reload haproxy on dbproxy1010 to depool labsdb1010 - https://phabricator.wikimedia.org/T174047
08:04 gehel: restart cassandra on maps* for JVM upgrade
07:59 gehel: restart cassandra on maps-test* for JVM upgrade
07:53 moritzm: installing libdatetime-timezone-perl stable update for jessie/stretch
07:38 gehel: restart elasticsearch on relforge for JVM upgrade
07:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: db1074 is now db1102's master - T193732 (duration: 00m 59s)
07:32 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1074 - T193732 (duration: 00m 59s)
07:28 marostegui: Change master from db1102:s2 from db1060 to db1074
07:19 marostegui: Stop replication in sync on db1060 and db1074 - T193732
07:19 moritzm: reimaging mw1303, mw1304 (job runners) to stretch
07:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1074 - T193732 (duration: 00m 59s)
07:06 moritzm: reimaging mw1233, mw1234, mw1235 (API servers) to stretch
07:02 moritzm: reimaging mw1327, mw1328, mw1329 (app servers) to stretch
05:37 marostegui: Unused databases devwikiinternal and rel13testwiki from s3 - T118764
05:27 marostegui: Deploy schema change on db1081 - T191519 T188299 T190148
05:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1081 for alter table (duration: 01m 01s)
05:15 marostegui: Deploy schema change on s6 primary master db1061 - T191519 T188299 T190148
03:06 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.2) (duration: 11m 22s)
03:36 XioNoX: commenting rigel out from smokeping
02:16 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2191.codfw.wmnet
02:03 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2193.codfw.wmnet
01:37 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2187.codfw.wmnet
00:35 Krinkle: Delete navtiming 'mediaWikiLoadComplete' metrics (T160315 )
00:26 Krinkle: Purging values for navtiming 'mediaWikiLoadEnd' sub-properties from 2018-04-21 to 2018-05-04T04:40:00 (T160315 )
22:50 mutante: mwmaint1001 - now using mw-maintenance role, upcoming terbium replacement
22:49 mutante: mw2187 - scap proxy - reinstalling with stretch
22:46 mutante: mw2191, mw2193 - wmf-auto-reimage with --no-verify because puppet certs didnt exist
22:02 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2199.codfw.wmnet
22:00 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2198.codfw.wmnet
21:55 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2197.codfw.wmnet
19:50 bblack: rebooting lvs1016 (downtimed, also new and not in service!)
19:02 sbisson@tin: Finished deploy [kartotherian/deploy@8e6b35b]: Use new keyspace (v4) for both i18n and non-i18n sources (duration: 03m 57s)
18:58 sbisson@tin: Started deploy [kartotherian/deploy@8e6b35b]: Use new keyspace (v4) for both i18n and non-i18n sources
18:38 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2196.codfw.wmnet
18:36 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2194.codfw.wmnet
18:35 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2195.codfw.wmnet
18:16 mutante: mw2197,mw2198,mw2199 - reinstall with stretch
18:12 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2192.codfw.wmnet
18:05 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2190.codfw.wmnet
18:03 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2189.codfw.wmnet
17:16 XioNoX: depolled eqsin
16:57 bawolff: adjusted login throttling code (T193762 )
16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2196.codfw.wmnet
16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2194.codfw.wmnet
16:50 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2195.codfw.wmnet
16:50 dzahn@neodymium: conftool action : set/pooled=no; selector: name=mw2194.codfw.wmnet
16:49 dzahn@neodymium: conftool action : set/pooled=no; selector: name=mw2196.codfw.wmnet
16:25 XioNoX: adding BGP graceful shutdown to routers - T190323
16:19 bawolff: Logging adjustment in mediawiki for T193762
15:57 arturo: enabled puppet in labcontrol1001, labnodepol100[1-2] and labtestvirt10[01-22] after patches deployed for T193657
15:16 mutante: mw2194, mw2195, mw2196 - reinstall with stretch - mw2193 - puppet cert not found
15:06 arturo: disabling puppet in labnodepool100[1-2] for T193657
15:05 arturo: disabling puppet in labcontrol1001 for T193657
14:52 arturo: disabling puppet in labvirt10[01-22] to deploy https://gerrit.wikimedia.org/r/#/c/430581/ and https://gerrit.wikimedia.org/r/#/c/430614/ T193657
14:05 mutante: mw2189, mw2190, mw2192 - reinstall with stretch, mw2191 - puppet cert not found
13:33 marostegui: Manually enable innodb_strict_mode on labsdb1009 - T150949
12:09 hashar: restarted Jenkins on contint1001 (java update)
12:04 jynus@tin: Synchronized wmf-config/db-codfw.php: Repool db2081 (duration: 00m 59s)
11:22 jynus: stopping db1056 and moving it to spare
10:55 marostegui: Manually enable innodb_strict_mode just on dbstore2001:3315 - T150949
10:51 moritzm: reimaging mw2153, mw2154, mw2155 (job runners) to stretch
09:52 jynus@tin: Synchronized wmf-config/db-eqiad.php: Remove db1056 (duration: 01m 00s)
09:39 moritzm: uploaded openjdk-8 8u171-b11 for jessie-wikimedia to apt.wikimedia.org
09:18 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Fully pool db1119 in s1 API (duration: 00m 59s)
09:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase API traffic for db1119 (duration: 00m 59s)
08:57 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Increase traffic for db1119 (duration: 01m 07s)
08:45 moritzm: reimaging mw2249, mw2250, mw2253 (job runners) to stretch
08:40 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool db1119 in s1 API (duration: 00m 59s)
08:30 mobrovac@tin: Finished deploy [cpjobqueue/deploy@193cf6f]: Config: Exclude refreshLinks from the RegEx rule (duration: 00m 47s)
08:29 mobrovac@tin: Started deploy [cpjobqueue/deploy@193cf6f]: Config: Exclude refreshLinks from the RegEx rule
08:24 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Slowly pool recently new cloned db1119 (duration: 01m 00s)
07:08 moritzm: reimaging mw2243, mw2247, mw2248 (job runners) to stretch
07:05 marostegui@tin: Synchronized wmf-config/db-codfw.php: Add db1119 to the config (duration: 01m 20s)
07:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Add db1119 to the config (duration: 01m 06s)
06:42 marostegui: Stop MySQL on db1066 to clone db1119 - T192979
06:37 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1066 - T192979 (duration: 01m 11s)
05:46 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clarify that db1060 will be decommissioned (duration: 00m 53s)
05:21 marostegui: Deploy schema change on dbstore1002:s4 - T191519 T188299 T190148
04:24 demon@tin: rebuilt and synchronized wikiversions files: group2 to wmf.2
04:17 demon@tin: Synchronized multiversion/getMWVersion: clean up getRealmSpecificFilename() (duration: 01m 07s)
04:07 demon@tin: Synchronized wmf-config/InitialiseSettings.php: disabling LQT on a few closed/unloved testwikis (duration: 01m 11s)
03:55 demon@tin: Synchronized scap/plugins: No-op plugin style fixes (duration: 01m 11s)
00:55 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Temporary low-level activation of Eventlogging impression data for testing T183978 (duration: 01m 16s)
00:50 niharika29@tin: Synchronized wmf-config/CommonSettings.php: Temporarily add a higher threshold to trigger login attempt notices T193762 (duration: 01m 17s)
00:01 demon@tin: rebuilt and synchronized wikiversions files: group1 to wmf.2
23:56 demon@tin: Synchronized php: symlink bump (duration: 01m 16s)
23:43 demon@tin: Synchronized scap/plugins/: cleanup, no-op (duration: 01m 17s)
23:16 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2186.codfw.wmnet
23:13 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2185.codfw.wmnet
23:12 krinkle@tin: Synchronized php-1.32.0-wmf.2/extensions/NavigationTiming/: If293a156ca T193570 (duration: 01m 16s)
23:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2188.codfw.wmnet
23:09 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming/: If293a156cac T193570 (duration: 01m 17s)
22:52 aaron@tin: Finished scap: Deploy db9acea T193668 ) (duration: 103m 50s)
22:21 ejegg: updated CiviCRM from 0fdef242a3 9752607052
21:21 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2183.codfw.wmnet
21:18 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2184.codfw.wmnet
21:08 aaron@tin: Started scap: Deploy db9acea T193668 )
20:34 XioNoX: started peering/transit with Deutsche Telekom on cr2-esams
20:30 mutante: mw1297 - reinstalling as mwmaint1001
20:09 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming: I1e7f091cba1
20:00 mutante: mw2185,mw2186,mw2188 - reinstall with stretch
19:53 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2254.codfw.wmnet
19:45 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2240.codfw.wmnet
19:34 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2231.codfw.wmnet
19:32 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2229.codfw.wmnet
19:18 mutante: mw1297 - puppet node clean, puppet node deactivate - renaming to mwmaint1001 (T192185 )
19:02 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: CentralNotice EventLogging banner impression data test T183978 (duration: 01m 04s)
18:59 thcipriani@tin: Synchronized php-1.32.0-wmf.2/extensions/NavigationTiming/modules/ext.navigationTiming.js: SWAT: Emit SaveTiming without relying on getNavTiming() T193693 (duration: 01m 16s)
18:49 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2240.codfw.wmnet
18:48 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2229.codfw.wmnet
18:48 dzahn@neodymium: conftool action : set/pooled=inactive; selector: name=mw2231.codfw.wmnet
18:44 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ULS webfonts by default at Bengali Wikisource T193367 (duration: 01m 18s)
18:09 mutante: mw2229,mw2231,mw2240 - wmf-auto-reimage with --new switch because their puppet cert wasn't found on puppetmaster, treated as new hosts that didnt exist before
17:59 mutante: mw2254,mw2183,mw2184 - wmf-auto-reimage with stretch and raid/lvm
17:36 ejegg: updated CiviCRM from 401344fb30 0fdef242a3
16:34 Jeff_Green: authdns-update to add frbast.wikimedia.org service alias
15:04 imarlier@tin: Finished deploy [performance/coal@762d160]: verify coal is deploying properly after shutdown on graphite hosts (duration: 00m 14s)
15:04 imarlier@tin: Started deploy [performance/coal@762d160]: verify coal is deploying properly after shutdown on graphite hosts
14:47 marostegui: Manually set offline disk #1 on db1063 so it can be replaced
14:44 imarlier@tin: Started deploy [performance/coal@bd7568a]: verify coal is deploying properly after shutdown on graphite hosts
14:13 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1056 (duration: 01m 17s)
14:00 ema: cp3030 (text): upgrade varnish to 5.1.3-1wm8 T192368
13:21 addshore: swat done
13:19 addshore@tin: Synchronized wmf-config/: Switch to extension.json for Wikidata.org (duration: 01m 19s)
13:13 addshore@tin: Synchronized wmf-config/: Switch to extension.json for PropertySuggester (duration: 01m 35s)
13:05 godog: stop and mask coal service on graphite hosts - T186774
13:04 gehel: rolling restart of wdqs for jvm upgrade
13:03 marostegui: Deploy schema change on s4 codfw master db2051 with replication (this will generate lag on codfw) - T191519 T188299 T190148
12:35 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Revert: Clarify that db1060 is running an alter table (duration: 01m 17s)
12:26 moritzm: installing openjdk-8 security updates on stretch-based Hadoop workers
11:57 moritzm: reimaging mw1301, mw1302 (job runners) to stretch
10:51 moritzm: reimaging mw1319, mw1325, mw1326 (app servers) to stretch
09:29 moritzm: reimaging mw1227, mw1231, mw1232 (API servers) to stretch
09:09 gehel: rolling restart of elasticsearch completed - T191543 / T191236
09:06 moritzm: reimaging mw1300 (job runner) to stretch
08:33 moritzm: installing Java security updates on wdqs*
08:31 addshore@tin: Synchronized php-1.32.0-wmf.2/extensions/WikimediaEvents/WikimediaEventsHooks.php: T191500 Update campaign prefix for onBeforeInitializeWMDECampaign hook (duration: 01m 16s)
08:29 addshore@tin: Synchronized php-1.32.0-wmf.1/extensions/WikimediaEvents/WikimediaEventsHooks.php: T191500 Update campaign prefix for onBeforeInitializeWMDECampaign hook (duration: 01m 17s)
08:27 mobrovac@tin: Finished deploy [changeprop/deploy@7e86531]: Bug fix: Resubscribe to the proper list of topics on metadata change (duration: 01m 12s)
08:26 mobrovac@tin: Started deploy [changeprop/deploy@7e86531]: Bug fix: Resubscribe to the proper list of topics on metadata change
08:18 mobrovac@tin: Finished deploy [cpjobqueue/deploy@5c1dcb9]: Bug fix: Resubscribe to the proper list of topics on metadata change (duration: 00m 54s)
08:17 mobrovac@tin: Started deploy [cpjobqueue/deploy@5c1dcb9]: Bug fix: Resubscribe to the proper list of topics on metadata change
08:13 ema: cp-misc: upgrade varnish to 5.1.3-1wm8 T192368
08:08 godog: eqiad-prod: more weight to ms-be104[0-3] - T190081
07:54 moritzm: reimaging mw1284, mw1289, mw1290 (API servers) to stretch
07:39 moritzm: reimaging mw1256, mw1257, mw1258 (app servers) to stretch
07:26 marostegui: Drop table flaggedrevs from eswikibooks - T193676
07:11 moritzm: reimaging mwdebug1002 to stretch
05:59 marostegui: Drop mostly empty flagged* tables from metawiki (s7) - T193390
05:57 elukey: reimage analytics10[39,40] to Debian Stretch
05:31 marostegui: Drop empty flagged* tables from eswiki (s7) - T193678
05:28 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Clarify that db1060 is running an alter table (duration: 01m 15s)
05:22 marostegui: Deploy schema change on db1060 with replication (this will generate lag on labs - s2) - T191519 T188299 T190148
03:32 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2182.codfw.wmnet
03:28 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2181.codfw.wmnet
03:26 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2180.codfw.wmnet
02:43 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 20s)
00:33 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/NavigationTiming/modules/: Ie77e77de3b8
00:20 mutante: mw2180,mw2181,mw2182 - reinstalling with stretch (in case there are alerts that's why)
00:02 twentyafterfour: no phabricator upgrade tonight.
23:39 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2179.codfw.wmnet
23:35 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2178.codfw.wmnet
23:29 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2177.codfw.wmnet
23:17 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/Kartographer/: Add maintenance script to purge pages with map tags (T193525 ) (duration: 01m 18s)
22:24 XioNoX: Re-enabled the link between fasw-eqiad and pfw3b-eqiad (backup) - T192104
22:18 ebernhardson: start reindex of viwiki on eqiad elasticsearch, failed on last run due to unrelated issues
22:17 XioNoX: Disabling the link between fasw-eqiad and pfw3b-eqiad (backup) - T192104
22:17 XioNoX: Re-enabled the link between fasw-codfw and pfw3b-codfw (backup) - T192104
21:42 XioNoX: Disabling the link between fasw-codfw and pfw3b-codfw (backup) - T192104
21:33 XioNoX: failing-over RG1 to node0 on pfw3-codfw - T192104
19:54 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2173.codfw.wmnet
19:38 mutante: mw2173 - scap pull (wasn't pooled but should have, bring up to date)
19:25 thcipriani@tin: rebuilt and synchronized wikiversions files: revert group1 to 1.32.0-wmf.2
19:23 thcipriani@tin: rebuilt and synchronized wikiversions files: group1 to 1.32.0-wmf.2
19:18 mutante: mw2177, mw2178, mw2179 - reinstalling with stretch
19:14 ppchelko@tin: Finished deploy [restbase/deploy@1093d1d]: Sample log action api 4xx with 1% probability (duration: 16m 18s)
19:10 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2176.codfw.wmnet
19:08 dzahn@neodymium: conftool action : set/pooled=yes; selector: name=mw2175.codfw.wmnet
18:58 ppchelko@tin: Started deploy [restbase/deploy@1093d1d]: Sample log action api 4xx with 1% probability
17:59 mutante: mw2174 - repooled
17:58 imarlier@tin: Started deploy [performance/coal@bd7568a]: deploy coal to webperf1001
17:55 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on wikis with <100 high-prio issues (T192299 ) (duration: 01m 17s)
17:48 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable RemexHtml on metawiki (T192386 ) (duration: 01m 17s)
17:40 imarlier@tin: Finished deploy [performance/coal@bd7568a]: deploy coal to webperf1001 (duration: 00m 06s)
17:40 imarlier@tin: Started deploy [performance/coal@bd7568a]: deploy coal to webperf1001
17:31 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/MobileFrontend/: T193564 (duration: 01m 20s)
17:29 catrope@tin: Synchronized php-1.32.0-wmf.2/extensions/Kartographer/: Add missing util dependency (duration: 01m 14s)
17:12 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable $wgCiteResponsiveReferences on kowiki (T193491 ) (duration: 01m 17s)
16:35 bawolff: run recountCategories.php on huwiki T169964
16:29 gehel: restarting blazegraph to increase TasksMax
15:18 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1098 (duration: 01m 17s)
15:07 joal@tin: Finished deploy [analytics/refinery@318d449]: Regular weekly deploy (duration: 08m 46s)
14:58 joal@tin: Started deploy [analytics/refinery@318d449]: Regular weekly deploy
14:37 jynus@tin: Synchronized wmf-config/db-eqiad.php: Add and pool db1121 (duration: 01m 17s)
14:33 jynus@tin: Synchronized wmf-config/db-codfw.php: Add db1121 (duration: 01m 16s)
14:23 addshore@tin: Synchronized wmf-config/InitialiseSettings.php: Update comment next to WMDE wmgMonologChannels entry T191500 (duration: 01m 17s)
13:58 gilles: End of mid-day EU SWAT
13:49 ottomata: beginning upgrade of kafka-jumbo brokers from 1.0.0 -> 1.1.0 : T193495
13:47 vgutierrez: Update puppet compiler facts
13:40 vgutierrez: Repool lvs2001 - T191897
13:37 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T187299 Add performance perception QuickSurvey definition (duration: 01m 17s)
13:29 elukey: upgrade zookeeper to 3.4.9 on druid100[4-6] (wikistats 2 backend) - T164008
13:20 elukey: restart druid broker on druid100[1-3] to enable the 'druid.sql.enable' feature
13:15 gilles: T193225 mwscript namespaceDupes.php --wiki=euwikisource --fix
13:13 gilles@tin: Synchronized wmf-config/InitialiseSettings.php: T193225 Add Author namespace on eu.wikisource (duration: 01m 20s)
13:05 gilles: Starting mid-day EU SWAT
13:01 moritzm: re-attempting to reimage mw1250, mw1254, mw1255 (app servers) to stretch, those ran into a timeout earlier which is now fixed in the reimage script
12:48 moritzm: reimaging mw1340, mw1341, mw1342 (API servers) to stretch
12:17 vgutierrez: Depool and reimage lvs2001 as stretch - T191897
11:41 vgutierrez: Repool lvs2002 - T191897
11:12 jynus: stopping db1064 for cloning to db1121 (will create temporary lag on commons wikireplicas)
11:02 kartik@tin: Finished deploy [cxserver/deploy@0aa3532]: Update cxserver to a20bf75
10:56 kartik@tin: Started deploy [cxserver/deploy@0aa3532]: Update cxserver to a20bf75
10:48 moritzm: reimaging mw2200 to stretch
10:30 moritzm: installing openjdk-8 security updates on stat hosts
10:25 vgutierrez: Depool and reimage lvs2002 as stretch - T191897
10:21 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1064 (duration: 01m 17s)
10:17 vgutierrez: Repool lvs2003 - T191897
09:10 vgutierrez: Depool lvs2003 and reimage as stretch - T191897
08:54 moritzm: reimaging mw1250, mw1254, mw1255 (app servers) to stretch
08:36 moritzm: reimaging mw1228, mw1229, mw1230 to stretch (those were logged to SAL before, but failed with IPMI issues before)
08:22 ema: varnish 5.1.3-1wm8 uploaded to apt.w.o T192368
08:11 elukey: upgrading Druid to 0.10 on druid100[4-6] (wikistats 2 backend) - T164008
07:42 elukey: remove openjdk-7 related packages from druid100[1-3] after zookeeper upgrade
07:36 gehel: elasticsearch eqiad rolling restart for plugin update and NUMA config - T191543 / T191236
07:31 elukey: upgrade zookeeper on druid100[1-3] to 3.4.9 - T164008
07:27 jynus: restart db1098 for upgrade and validation
02:44 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 11s)
00:17 ebernhardson: start reindex for commonswiki, eqiad elasticsearch, commonswiki_general appears to have failed previous reindex
22:29 ejegg: updated CiviCRM from 46883844a3 401344fb30
22:19 andrewbogott: rebooting labnet1002 for T193579
22:04 krinkle@tin: Synchronized php-1.32.0-wmf.1/extensions/EducationProgram/resources/: I7ca59823ffbf2
22:01 awight@tin: Finished deploy [ores/deploy@bf182e2]: Rollback ores1001 to master (duration: 01m 13s)
22:00 awight@tin: Started deploy [ores/deploy@bf182e2]: Rollback ores1001 to master
21:57 krinkle@tin: Synchronized php-1.32.0-wmf.1/includes/libs/rdbms/: Iba663c
21:54 awight@tin: Finished deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627 (duration: 03m 21s)
21:50 awight@tin: Started deploy [ores/deploy@52347e0]: Test LFS deployment for ORES; T180627
21:48 awight@tin: Finished deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627 (duration: 00m 26s)
21:48 awight@tin: Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627
21:48 awight@tin: Started deploy [ores/deploy@4601497]: Test LFS deployment for ORES; T180627
21:44 demon@tin: Finished scap: group0 to wmf.2 (duration: 68m 18s)
21:42 XioNoX: re-enabling eqsin-codfw link
21:23 urandom: restbase: begin culling leaked revisions, enwiki_T_mobile__ng_{lead,remaining} - T192689
20:43 urandom: restbase: begin culling leaked revisions, commons_T_mobile__ng_remaining - T192689
20:35 demon@tin: Started scap: group0 to wmf.2
lunch: update stale apifeatureusage-search-svc-eqiad-wmnet template in eqiad elasticsearch and delete unused apifeatureusage template
20:28 urandom: restbase: begin culling leaked revisions, commons_T_mobile__ng_lead - T192689
20:06 demon@tin: Pruned MediaWiki: 1.31.0-wmf.27 (duration: 03m 08s)
20:00 ottomata: rolling restart of eventbus to apply new logstash formatter version T193230
19:59 demon@tin: Pruned MediaWiki: 1.31.0-wmf.30 [keeping static files] (duration: 01m 43s)
19:49 urandom: restbase: begin culling leaked revisions, others_T_mobile__ng_remaining - T192689
19:27 ottomata: rolling restart of eventbus to apply logstash tag https://phabricator.wikimedia.org/T193230
19:17 mutante: mw2174,mw2175,mw2176 ff - reinstalling with wmf-auto-reimage to stretch
19:00 ottomata: rolling restart of eventlogging-service-eventbus to apply logstash logging configs - T193230
18:53 ariel@tin: Finished deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone (duration: 00m 04s)
18:53 ariel@tin: Started deploy [dumps/dumps@5438d41]: keep running even if file we want to report on is moved/gone
17:40 XioNoX: disabling eqsin<->codfw link for high packet loss on link
17:39 otto@tin: Finished deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230 (duration: 02m 29s)
17:36 otto@tin: Started deploy [eventlogging/eventbus@c70e8c5]: remove occasional logging of request.body in prep for T193230
16:15 ebernhardson: T192972 change eqiad elasticsearch disk watermarks from 85/85 to 80/80 to match disk space alerts
16:15 herron: manually kicked off mirror@sodium:~$ /usr/local/sbin/update-ubuntu-mirror to clear ubuntu mirror out of sync alert
15:50 urandom: restbase: begin culling leaked revisions, others_T_mobile__ng_lead -- T192689
15:37 ejegg: updated SmashPig standalone from a4de12d415 99585c8084
13:35 herron: restarted hhvm on mw1233
02:47 l10nupdate@tin: scap sync-l10n completed (1.32.0-wmf.1) (duration: 07m 04s)
01:20 ejegg: re-enabled fundraising jobs
01:01 ejegg: updated CiviCRM from 47197006d5 46883844a3
00:51 eileen: email sent advising of minor interuption
00:50 ejegg: disabled fundraising jobs for db update
00:21 ebernhardson: increase cluster.routing.allocation.balance.threshold from 1.0 to 1.5 for eqiad elasticsearch cluster to reduce rebalancing agressiveness
