Server Admin Log/Archive 22: Difference between revisions

Browse history interactively

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 21:22, 23 May 2013

May 23

21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable echo email bundling on enwiki'
20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted 'E3 latest GettingStarted'
20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour 'E3 latest GuidedTour'
19:17 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo/modules 'syncing Echo modules for css updates'
18:54 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to Master
18:48 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to Master
18:31 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add new eventlogging schema: EchoPrefUpdate'
12:19 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
02:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 23 02:25:13 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 23 02:06:32 UTC 2013

May 22

23:40 logmsgbot: aaron synchronized php-1.22wmf4/maintenance/runJobs.php '2188c14239b74d202b2109933612259193f6fa41'
22:44 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, attempt to fix VisualEditor cache snafu on dewiki'
22:33 paravoid: removing users sara & mwang from ldap/ops
19:12 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
19:05 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
18:51 apergos: restarted gerrit
18:42 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Swithced remaining wikis to 1.22wmf4
18:26 logmsgbot: kaldari synchronized php-1.22wmf4/includes/Preferences.php 'syncing preferences.php for bug 48693'
18:02 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
18:00 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
17:46 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
17:41 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
07:50 apergos: restarted swift proxy on ms-fe1
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 22 02:35:44 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 22 02:14:48 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 22 02:08:27 UTC 2013
00:21 Ryan_Lane: deployed change 64875 to virt0

May 21

21:11 logmsgbot: awjrichards Finished syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
21:04 logmsgbot: awjrichards Started syncing Wikimedia installation... : Regular weekly MobileFrontend deployment
19:27 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:22 logmsgbot: kaldari Started syncing Wikimedia installation... :
18:31 mwalker: (previous log referring to fundraising jenkins!) now upgrading fundraising jenkins for auth plugins
18:31 mwalker: restarted jenkins for rekeying of jobs dir; then updated non auth plugs
18:17 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Disable AFTv5 talk page links'
18:16 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable AFTv5 talk page links'
17:49 cmjohnson1: powering down db26 for last time
17:45 Jeff_Green: rebooting several fundraising hosts for kernel upgrades
17:14 cmjohnson1: replacing disk10 (L11) on ms-be1009
14:38 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Disable $wgCentralAuthLoginWiki for testwiki'
13:59 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
13:59 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Set $wgCentralAuthLoginWiki for test & test2 (SUL2 testing)'
13:14 logmsgbot: anomie synchronized php-1.22wmf4/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
13:13 logmsgbot: anomie synchronized php-1.22wmf3/extensions/CentralAuth 'Update CentralAuth to master, for SUL2 testing on testwiki and test2wiki'
09:41 hashar: made Jeroen De Dauw a maintainer of https://packagist.org/packages/mediawiki/data-values
08:10 hashar: zuul: running git gc on all Zuul git repositories: find /var/lib/zuul/git -maxdepth 4 -type d -name .git -print -exec git --git-dir={} gc \;
08:02 hashar: Running git gc on Zuul git repositories (gallium: /var/lib/zuul/git/operations/puppet )
07:18 LeslieCarr: allowed labs access to the new labsdb hosts
02:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 21 02:26:30 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 21 02:09:52 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 21 02:06:41 UTC 2013
01:31 cmjohnson1: cycling titanium to check bios settings

May 20

23:59 mutante: dist-upgrading celsus
23:36 logmsgbot: mflaschen synchronized php-1.22wmf4/extensions/GettingStarted/ '-m Sync GettingStarted for E3 lightning deploy'
22:26 mutante: rebooting Tampa parsoid servers, one by one
22:20 mutante: dist-upgrading, rebooting kuo
22:11 mutante: dist-upgrading, rebooting arsenic
22:04 mutante: dist-upgrading, rebooting strontium
21:48 mutante: dist-upgrading, rebooting palladium
21:31 mutante: dist-upgrading, rebooting niobium
20:58 logmsgbot: krinkle synchronized php-1.22wmf4/extensions/VisualEditor/modules/ve/ve.EventEmitter.js 'touch, fixup cache snafu'
20:51 Ryan_Lane: temporarily disabling nova-api ratelimiting
20:46 akosiaris: added cloudera debian packages hadoop, zookeeper, bigtop-jsvc, bitop-utils to apt.wikimedia.org
20:07 Ryan_Lane: rebooting all pmtpa virt nodes
20:07 Ryan_Lane: shutdown all labs instances
19:12 mutante: dist-upgrading cp1(sq51)
19:09 logmsgbot: csteipp synchronized php-1.22wmf4/includes/ 'Security fix'
19:08 logmsgbot: csteipp synchronized php-1.22wmf3/includes 'Security fix'
19:04 logmsgbot: reedy synchronized w/
19:01 logmsgbot: reedy synchronized wmf-config/wgConf.php
19:00 logmsgbot: reedy synchronized wmf-config/wgConfVHosts.php
18:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable translate on commonswiki'
18:46 Reedy: Created translate tables on commonswiki
18:44 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
18:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:30 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache
18:21 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache
18:15 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
18:14 logmsgbot: reedy synchronized php-1.22wmf4/extensions/ProofreadPage/
18:08 logmsgbot: reedy synchronized wmf-config/ 'Enable TemplateData on all wikis'
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf4
17:49 MaxSem: Reindexing solr
17:48 mutante: upgrading/rebooting mobile varnish cache cp1041-cp1044
17:35 mutante: kernel upgrade cp1043
17:00 mutante: resurrecting icinga-wm and nsca
16:59 mutante: ping logmsgbot
16:52 mutante: rebooting neon for upgrade
16:51 cmjohnson1: shutting down lanthanum to add ssd rt5074
15:49 ottomata: rebooting gadolinium (source of webrequest multicast stream) to upgrade kernel
14:55 ottomata: rebooting emery to upgrade to 3.2.0-43-generic kernel
14:53 ottomata: upgraded all analytics* machines to linux 3.2.0-43-generic kernel
02:30 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 20 02:29:25 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 20 02:13:42 UTC 2013
02:05 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Mon May 20 02:05:04 UTC 2013

May 19

02:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 19 02:06:55 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 19 02:01:54 UTC 2013
02:02 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sun May 19 02:01:19 UTC 2013

May 18

02:29 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 18 02:29:09 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 18 02:11:28 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Sat May 18 02:06:30 UTC 2013
01:23 mutante: singer boot issue - affects service contacts.wm (but no others)
00:13 mutante: dist-upgrading yvon
00:09 mutante: dist-upgrading nitrogen

May 17

23:48 mutante: dist-upgrading singer
23:46 mutante: dist-upgrading nickel
23:45 RobH: all tampa based apaches have had kernel upgrades
23:40 mutante: dist-upgrading praseodymium
23:33 RobH: upgrading all srv*.pmtpa.wmnet via dist-upgrade in salt.
23:28 mutante: dist-upgrading gurvin
23:13 RobH: rebooting all pmtpa mw servers
23:13 mutante: dist-upgrading hydrogen, manutius
23:12 RobH: still doing pmtpa mw upgrades, ignore all icinga alarms for now
23:05 mutante: dist-upgrading chromium
23:02 RobH: using salt to dist-upgrade all the tampa apaches... nothing could go wrong....right?
22:58 akosiaris: upgrading and rebooting cp1001-1020 (3 at a time)
22:55 mutante: dist-upgrading capella ( IPv6 tunnel relay)
22:34 mutante: shutting down zinc
22:23 mutante: !!log doesn't log
22:21 Reedy: [23:18:29] <binasher> !!log running hotbackup of db71 to pre-labsdb for s1
22:14 mutante: rebooting spence for upgrades
22:10 logmsgbot: reedy synchronized w
22:08 logmsgbot: reedy synchronized docroot
22:08 mutante: rm -rf php-1.21wmf12 on tin per reedy
22:03 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
21:58 apergos: mw80 memtest failure dimm1, all other image scalers in pmtpa updated
21:53 mutante: dist-upgrading zhen (mobile vumi)
21:49 mutante: DNS update - kill storage1 and 2
21:34 akosiaris: about to reboot fenari
21:13 hashar: gallium Manually blackholed some web engine crawler (via ip route)
19:55 mutante: Wikimedia IRC server working again
19:46 Ryan_Lane: rebooted ssl1/2 ssl3001 ssl1001/1002
19:42 hashar: Jenkins restarted successfully.
19:40 Ryan_Lane: depooling ssl3001 ssl1/2 ssl1001/2
19:40 Ryan_Lane: repooling ssl1003/4 ssl3/4
19:31 Ryan_Lane: repooling ssl3002/3
19:30 hashar: restarted Jenkins
19:29 Ryan_Lane: rebooted ssl3/4 ssl3002/3 ssl1003/1004
19:29 Ryan_Lane: depooled ssl3/4 ssl3002/3 ssl1003/1004
19:19 RobH: blog back up, whew.
19:16 RobH: holmium (blog server) rebooting for kernel upgrade
19:12 cmjohnson1: shutting titanium down to add disks (take 2)
19:04 apergos: eqiad imagescalers done
19:04 mutante: rebooting gallium
19:03 mutante: wikimedia irc is back
19:00 hashar: gallium: stopping jenkins and zuul
18:56 RobH: cerium rebooting for kernel upgrade
18:56 Ryan_Lane: dist-upgrading all ssl boxes
18:55 mutante: dist-upgrading ekrem (IRC)
18:55 cmjohnson1: shutting down titanium for add'l disk
18:50 RobH: i hate you morebots.
18:50 RobH: caesium rebooting
18:47 mutante: dist-upgrading calcium
18:42 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Removing france as a redirect country for fundraising (7649276ddb3f20678de265c37fe75e1d5e642956)'
18:31 hashar: Jenkins: created a Jenkins slave on gallium using jenkins-slave as user and /srv/ssd/jenkins-slave as working directory.
18:17 RobH: carbon rebooting for kernel update, tftp in eqiad will be offline for a few
17:50 apergos: updating and rebooting image scalers one by one
17:46 cmjohnson1: shutting down cerium to add add'l disk
17:39 apergos: rebooted snapshots 1001-4 (new kernel)
17:35 mutante: bugzilla is back
17:31 ^demon: rebooting formey (svn) in 2 minutes...why are you still using SVN?
17:30 mutante: rebooting kaulen
17:24 mutante: rebooting kaulen (Bugzilla) in 5 minutes, please save your bugs and expect 5 minutes downtime
16:49 ^demon: rebooting antimony
15:00 paravoid: setting Varnish send_timeout to 600 on upload, mobile, bits (eqiad), upload, bits (esams) for both frontend & backend
13:31 hashar: Zuul: applying project templates 63674
12:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
12:29 logmsgbot: reedy synchronized php-1.22wmf4/maintenance/checkUsernames.php
05:06 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
02:34 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 17 02:33:21 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 17 02:12:44 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Fri May 17 02:07:14 UTC 2013
00:56 K4-713: Woohoo! Logs!
00:51 logmsgbot: reedy synchronized php-1.22wmf3/maintenance/checkUsernames.php

May 16

23:48 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
23:45 logmsgbot: pgehres synchronized php-1.22wmf4/extensions/AccountAudit/ 'Updating AccountAudit to master. Adding aa_method'
23:43 logmsgbot: reedy synchronized database lists files:
23:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:39 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
22:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:29 logmsgbot: mwalker synchronized php-1.22wmf4/extensions/CentralNotice/ 'Updating CentralNotice to master'
22:28 logmsgbot: reedy synchronized wmf-config/
22:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:02 logmsgbot: reedy synchronized wmf-config/flaggedrevs.php
22:00 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
21:53 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 scap for deployment of GettingStarted and GuidedTour
21:50 superm401: Scap failed due to wrong ssh configuration
21:49 Ryan_Lane: rebooting virt1005 and virt1007
21:49 logmsgbot: mflaschen Finished syncing Wikimedia installation... : E3 deployment
21:48 logmsgbot: mflaschen Started syncing Wikimedia installation... : E3 deployment
21:47 Ryan_Lane: rebooting virt1000
21:40 superm401: Started scap for E3 deployment of GettingStarted and GuidedTour
21:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:29 binasher: running accountaudit_login aa_method schema migrations on all wikis
21:20 Ryan_Lane: rebooting virt0
20:56 Ryan_Lane: running dist-upgrade on virt cluster
20:51 logmsgbot: reedy synchronized wmf-config/
20:40 logmsgbot: reedy synchronized php-1.22wmf4/extensions/Wikibase/
20:35 paravoid: restarting a ceph-osd osd.3 for testing
20:03 logmsgbot: reedy synchronized wmf-config/ 'Updating interwiki cache'
19:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:58 logmsgbot: reedy synchronized database lists files:
19:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:51 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:49 logmsgbot: reedy synchronized database lists files:
19:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:01 logmsgbot: reedy synchronized wmf-config/
18:56 logmsgbot: reedy synchronized wmf-config/
18:55 logmsgbot: reedy synchronized wmf-config/session.php
18:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'EP on dewikiversity'
18:41 mwalker|0_0: updating payments from fdf308c8fa92ca75adbb3154b8b31a9f7237e885 to head
18:34 gwicke: updated Parsoid to b0229051eb08c8
18:29 mutante: magnesium (racktables): upgrading kernel and rebooting
18:21 mutante: zirconium (planet): upgrading kernel and rebooting
18:17 paravoid: restarting a ceph-osd osd.0 for testing
18:02 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
17:51 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding l10n cache for wikidata deploy
17:27 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
17:13 logmsgbot: reedy synchronized php-1.22wmf4/extensions/
16:53 ottomata: rebooting stat1 for kernel upgrade
16:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pooling db1043'
16:18 gwicke: Parsoid update to current master: done
16:16 gwicke: Parsoid update to current master: start
15:57 Krinkle: Graceful reload of Zuul deploying Ic9194c079fa7d46
15:28 Krinkle: Graceful reload of Zuul deploying I7565eff83a7a128e
13:50 Krinkle: Graceful reload of Zuul deploying I0ea73ca1439c8aa6
07:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/ 'I9d5afff9c328'
07:47 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/GeoData/ 'I9d5afff9c328'
07:30 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
06:42 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
02:31 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 16 02:30:28 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 16 02:11:55 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Thu May 16 02:06:34 UTC 2013
00:37 mutante: puppetstoredconfigclean.rb ocg3.pmtpa.wmnet
00:14 mutante: puppetstoredconfigclean.rb ms2.pmtpa.wmnet
00:06 logmsgbot: reedy synchronized database lists files:
00:05 logmsgbot: reedy synchronized database lists files:

May 15

23:44 mutante: pushing wikimedia-task-appserver 2.9-1
23:32 logmsgbot: reedy synchronized database lists files:
23:30 logmsgbot: reedy synchronized database lists files:
22:30 logmsgbot: krinkle synchronized php-1.22wmf4/resources/mediawiki/mediawiki.js 'touched'
22:01 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
21:54 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/includes/PageRenderingHooks.php
20:57 Ryan_Lane: OpenStackManager is reenabled on virt0. login to wikitech is working again
20:56 Ryan_Lane: openstack in pmtpa now at the folsom release
20:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
20:51 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend/
20:15 Ryan_Lane: force running puppet on all virt nodes to bring them back up
20:14 Ryan_Lane: upgrading schema for keystone, then glance, then nova
20:13 Ryan_Lane: backing up keystone, nova, and glance databases
20:12 logmsgbot: aaron cleared profiling data
20:12 logmsgbot: aaron cleared profiling data
20:10 Ryan_Lane: upgrading glance, keystone and nova-scheduler on virt0
20:09 Ryan_Lane: upgrading nova services on compute nodes
20:07 logmsgbot: aaron synchronized php-1.22wmf4/includes/specials/SpecialActiveusers.php 'df79a82'
20:05 Ryan_Lane: disabling OpenStackManager
20:05 Ryan_Lane: stopping all openstack services. it won't be possible to log into wikitech during this process
19:44 paravoid: running refreshFileHeaders.php on terbium
19:25 logmsgbot: aaron cleared profiling data
19:19 Ryan_Lane: adding temporary ubuntu-cloud archive apt changes in a non-puppetized way on pmtpa virt nodes
19:18 Ryan_Lane: disabling puppet on pmtpa virt cluster
19:09 logmsgbot: reedy synchronized wmf-config/ 'touch!'
19:09 logmsgbot: reedy synchronized php-1.22wmf4/resources/ 'touch!'
18:42 Ryan_Lane: cherry-picked change 62100 on virt0 until permanent fix for jquery.chosen minimum width is added
18:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf4
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: commons back to 1.22wmf3 due to broken UploadWizard
18:35 ^demon: restarting gerrit
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aaaand wikisource to 1.22wmf4
18:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf4
18:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikibooks to 1.22wmf4
18:22 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage and wiktionary to 1.22wmf4
18:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, special, wikimedia and private to 1.22wmf4
18:01 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/
17:58 RobH: authdns update, killin dataset1 entries
17:56 logmsgbot: yurik synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
17:50 Ryan_Lane: deploying new project filter on virt0
17:44 logmsgbot: yurik synchronized php-1.22wmf4/extensions/ZeroRatedMobileAccess/
16:57 RobH: authdns update, cleaning out old decom entries
15:35 Jeff_Green: rebooting boron
11:16 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
11:15 logmsgbot: reedy synchronized wmf-config/squid-labs.php
11:14 logmsgbot: reedy synchronized wmf-config/squid.php
11:13 logmsgbot: reedy synchronized wmf-config/squid-labs.php 'wmf-config/squid.php'
11:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:35 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 15 02:34:37 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Wed May 15 02:14:58 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 15 02:08:15 UTC 2013
00:49 mutante: graceful apache on stat1001
00:31 logmsgbot: maxsem synchronized php-1.22wmf4/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'
00:29 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/63806/'

May 14

23:57 mwalker: lies -- went from d895730067e3325fd630d09196ee81eeba24b5ee back to head on civi because it was working -- the tester was just broken
23:42 mwalker: rolling back civi from deploy head to d895730067e3325fd630d09196ee81eeba24b5ee because thank you isn't working
23:38 logmsgbot: aaron synchronized wmf-config/StartProfiler.php
23:23 logmsgbot: maxsem synchronized php-1.22wmf3/resources/startup.js
22:38 RobH: authdns-update for a bunch of decom updates
22:32 mutante: DNS update - removing owa servers
22:23 logmsgbot: tstarling synchronized wmf-config/db-pmtpa.php
22:17 Ryan_Lane: finished upgrading wikitech
22:10 Reedy: Running sync-common on srv193
22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:03 Ryan_Lane: upgrading mediawiki to 1.22wmf4 on wikitech
22:03 mwalker: upgrading fundraising civi (d895730067e3325fd630d09196ee81eeba24b5ee --> head)
21:53 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:30 RoanKattouw: Repooled wtp1004, added wtp1005-wtp1024 to the pool
21:01 sbernardin: changing HD in ms-be5
20:52 binasher: s4 now all mariadb, new master info = MASTER_LOG_FILE='db1059-bin.000012', MASTER_LOG_POS=473448023
20:45 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/63596/'
20:44 sbernardin: changing drives on ms-be10
20:39 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/Gadgets/ 'https://gerrit.wikimedia.org/r/#/c/60954/'
20:33 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
20:32 mutante: apply umask 0002 for wikidev users fix on tin and terbium, class generic::wikidev-umask
20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to writeable on new master'
20:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting s4 to read-only for a sec'
20:22 logmsgbot: maxsem synchronized docroot/bits/mobile/W.png
20:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
19:53 RoanKattouw: Running salt-call state.highstate on wtp1005-24
19:51 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php 'd58dc596a25607deaecf0aa1c3be877f834da533 '
19:47 RoanKattouw: Accepted salt keys for wtp1005-1024 on sockpuppet
19:22 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
19:12 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
18:52 mutante: gracefulling apaches with dsh
18:46 Ryan_Lane: running an apache-graceful-all
18:44 ottomata: deployed puppet change to make nginx send webrequest logs back to main multicast firehose. NOTE: These logs are not sent to the udp2log instance on emery.
18:35 sbernardin: replacing bad drive in ms-be11
18:34 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
18:31 logmsgbot: reedy synchronized wmf-config/
18:30 logmsgbot: aaron synchronized php-1.22wmf3/includes/filerepo/file/LocalFile.php '0cf6b27aa43ad2b65725081ee4d36ccb0f8a34c6'
18:28 logmsgbot: reedy synchronized wmf-config/
18:27 mutante: powercycling frozen mw1173
18:25 logmsgbot: kaldari synchronized php-1.22wmf4/extensions/Echo 'syncing Echo for 1.22wmf4'
18:12 logmsgbot: reedy synchronized wmf-config/
17:56 logmsgbot: reedy synchronized wmf-config/
17:53 logmsgbot: reedy synchronized wmf-config/
17:28 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Enable upload of XML files on Incubator (for MediaWiki dumps)'
17:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Namespace configuration for ur.wikipedia'
17:15 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:06 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:18 logmsgbot: reedy synchronized wmf-config/
15:14 Reedy: Created EducationProgram tables on dewikiversity
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'thwiki collation'
13:53 cmjohnson1: running dns update
13:53 hashar: Zuul: applying a patch to prevent it from fetching a change multiple time.
10:51 apergos: re-enabling puppet on ssl1001 for a bit of real traffic
10:42 apergos: repooled ssl1
09:58 apergos: puppet temp disabled on all ssl terminators except ssl1 in pmtpa
09:37 apergos: depooled ssl1 in pmtpa for testing
08:51 logmsgbot: tstarling synchronized php-1.22wmf4/LocalSettings.php 'remove testwiki special case'
08:51 logmsgbot: tstarling synchronized php-1.22wmf3/LocalSettings.php 'remove testwiki special case'
08:34 logmsgbot: tstarling Finished syncing Wikimedia installation... :
08:26 logmsgbot: tstarling Started syncing Wikimedia installation... :
08:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:44 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:34 logmsgbot: tstarling Started syncing Wikimedia installation... :
07:16 logmsgbot: tstarling synchronized w/MWVersion.php
07:15 logmsgbot: tstarling synchronized refresh-dblist
07:14 logmsgbot: tstarling synchronized multiversion
07:12 logmsgbot: tstarling synchronized docroot/noc/db.php
06:40 logmsgbot: tstarling synchronized README
04:34 Tim: moved scap to tin. Copied source from NFS to tin. Fixed up broken git submodule config using sed.
04:03 logmsgbot: tstarling synchronized cgi-bin
03:10 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 14 03:09:49 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.22wmf4) at Tue May 14 02:31:15 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 14 02:12:10 UTC 2013
02:11 logmsgbot: LocalisationUpdate failed: mwversionsinuse returned empty list
01:13 logmsgbot: tstarling synchronized README
01:00 logmsgbot: tstarling synchronized README
00:20 akosiaris: added myself to root email alias at mchenry
00:17 akosiaris: installing package-upgrades on zirconium

May 13

23:19 logmsgbot: catrope synchronized php-1.22wmf4/extensions/VisualEditor/ 'VE bug fix'
22:46 RobH: authdns-update for wikipedia.is
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:11 hashar: jenkins: updating all phpcs-HEAD jobs to make them ignore i18n.php files bug 48420 / 63571
21:02 logmsgbot: reedy synchronized wmf-config/
20:34 logmsgbot: aaron cleared profiling data
20:33 cmjohnson1: morebots back
14:40 gallium: deleted an ip route blacklist
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon May 13 02:35:49 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Mon May 13 02:06:03 UTC 2013

May 12

14:59 Krinkle: Graceful reload of Zuul to deploy Ic22d7c6cb811fd7e3
14:31 jeremyb: imported missing !log entries from channel log
14:23 bblack: restarted adminbot on wikitech-static
14:22 bblack: fixed typo in rt.wikimedia.org vhost hostname, was breaking RT http stuff
14:19 bblack: morebots is broken, someone should fix that when they read this
09:32 logmsgbot: olivneh synchronized wmf-config/throttle.php '(Bug 48301) Add throttle exception for Haifa University workshop'
03:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 12 03:03:03 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 12 02:13:17 UTC 2013
01:28 logmsgbot: aaron synchronized php-1.22wmf3/maintenance/copyFileBackend.php '3e103e93f01c24d7e454aaa5db07418405d273c3'
01:22 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendMultiWrite.php '308ebddc237e76a9602b20585386fc26876f0bee'

May 11

13:24 hashar: restarted Jenkins. Got hit by a memory bug again.
03:48 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 11 03:48:00 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 11 02:16:23 UTC 2013

May 10

23:20 mutante: DNS update - remove various old Tampa hosts
22:49 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Small little update to maintenance script'
21:46 mutante: shutdown -h blondel .. bye
21:39 RobH: switch port disabled for blondel & bellin
21:27 RobH: decom of bellen/blondel : setting network ports to disabled
21:27 AaronSchulz: Running listing based copy-sync scripts in terbium for all originals+transcodes on all wikis for swift->ceph to fix any outdated/missing files
21:05 mutante: fixing some more varnishncsa monitoring
20:58 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend/FileBackendStore.php '2717d876ba2507df26ac2c5ab746ad7895af1bbf'
20:31 logmsgbot: aaron synchronized php-1.22wmf3/includes/filebackend
20:06 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS
19:57 logmsgbot: aaron synchronized php-1.22wmf3/maintenance
19:56 logmsgbot: aaron synchronized php-1.22wmf3/includes '6c01414775bb9c5269b391bee7ffd1611f9ff74f & d1940caa1f4a11e7c7d8d2ff5b178da7de8e2a92'
18:50 mutante: killing and restarting nagios-nrpe-server on a bunch of cp10xx boxes to fix monitoring of varnishncsa
18:23 mutante: powercycling srv291
18:17 RobH: authdns-update to remove blog.wikimediafoundation as it wasnt setup and pointed to decom server
17:42 RobH: authdnsupdate - removing racktables2 entry
17:19 hashar: jenkins: cleared all occurrences of an old plugin in build histories. bug 48025. Jenkins should be fine now. Will monitor tonight.
13:20 mark: Migrated toolserver routing to cr1-esams and cr2-esams via csw2-esams access
12:28 mark: Removed knsq23-knsq30 from the text squid config
12:25 hashar: jenkins: cleaning up all occurrences of an old plugin in build histories. The script run in a tmux on gallium (see https://bugzilla.wikimedia.org/show_bug.cgi?id=48025#c19 )
12:09 mark: Depooled knsq23-knsq30 text squid frontends
11:52 mark: Moved amslvs1 and 2 from csw1-esams to csw2-esams
11:52 mark: Started PyBal on amslvs1
11:51 mark: Stopped PyBal on amslvs1
11:49 mark: Reenabled PyBal on amslvs2
11:44 hashar: restarting jenkins
11:44 mark: Disabled Puppet and stopped PyBal on amslvs2
11:27 mark: Moved ssl3001-3004 from csw1-esams to csw2-esams
10:09 hashar: jenkins: upgrading plugins and restarting.
09:43 hashar: restarted jenkins twice to get it to finally serve something :(
09:27 hashar: killed -9 jenkins :(
09:25 hashar: restarting jenkins
09:18 hashar: gallium: dumping java heap of Jenkins using jmap -dump:format=b,file=/root/jenkins-bug48025.jmap -F 24508
08:34 hashar: Jenkins hit by the evil bug again bug 48025. Taking as much trace as possible before restarting it
04:46 paravoid: swift: pushing new rings, set weight 0 for ms-be5 sdb1, ms-be11 sdh1 (broken disks); balance 999.99, needs further rebalance
03:38 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 10 03:37:41 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 10 02:15:49 UTC 2013
00:15 logmsgbot: pgehres synchronized wmf-config 'Actually enabling wgCentralAuthAutoMigrate'

May 9

23:34 logmsgbot: mwalker synchronized wmf-config/InitialiseSettings.php 'Enabling siteNotice div on all alpha and beta mobile sites'
23:27 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/CentralAuthUser.php 'Moving one more CentralAuth log line'
23:22 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Updating CentralNotice to master for some bugfixes'
23:07 logmsgbot: pgehres synchronized php-1.22wmf3/extensions/CentralAuth/ 'Updates to CentralAuth'
21:55 hashar: Jenkins / Zuul is back up (for now).
21:45 hashar: restarted Jenkins to enable access logs in /var/log/jenkins/access.log 63068
21:42 hashar: forced Zuul to stop. All events have been dropped :(
21:31 hashar: restarting Zuul to make it query Jenkins directly without SSL + apache proxy 63069.
21:29 Jeff_Green: pdns-update to cut back to aluminium, restarted pdns on nescio b/c it failed
21:28 ^demon: gerrit repositories will now recursively merge
21:17 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/WikimediaMessages 'Updating WikimediaMessages to master'
20:48 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/GuidedTour 'Actually updating GuidedTour to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating GuidedTour to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/EventLogging 'Updating EventLogging to master'
20:46 logmsgbot: olivneh synchronized php-1.22wmf3/extensions/ConfirmEdit 'Updating ConfirmEdit to master'
20:21 hashar: restarting Jenkins.
20:20 hashar: Jenkins died :/
19:17 mutante: fixing file permissions in /a/e3/E3Analysis repo on stat1001
19:09 mutante: restarted Apache and then puppetmaster on stafford, that revived it
18:56 mutante: attempting to restart broken puppetmaster on stafford
18:27 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling CentralAuth debug log'
18:00 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling wgCentralAuthAutoMigrate on all wikis that use CentralAuth'
15:56 cmjohnson1: powering down barium /relocate server to c1
15:27 logmsgbot: anomie synchronized wmf-config/CommonSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
15:27 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
15:26 logmsgbot: anomie synchronized wmf-config/InitialiseSettings-labs.php 'Make $wgCodeEditorEnableCore configurable per wiki, prep for bug 39653'
11:25 hashar: upgrading Zuul to latest master (makes report times friendlier)
08:54 hashar: Jenkins apparently working fine. I have upgraded a few plugins, that seems to have fixed the Gerrit jobs.
08:22 hashar: restarting Jenkins again
08:21 hashar: upgrading Jenkins plugins 'promoted build' and 'M2 release'. That should fix the Gerrit jobs disappearing (ping qchris)
07:39 hashar: restarting jenkins. All its http thread are locked :/
07:25 hashar: Jenkins HTTP interface died :/
03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 9 03:39:55 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 9 02:15:54 UTC 2013

May 8

22:27 RobH: authdns update to migrate racktables from hooper to magnesium. both servers are already using db1001 as master, which is why hooper is slow to load racktables data
22:24 logmsgbot: mholmquist synchronized php-1.22wmf3/extensions/UploadWizard/UploadWizardHooks.php 'Fixing Special:Preferences for wikis with UploadWizard'
21:45 logmsgbot: mholmquist Finished syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
21:37 logmsgbot: mholmquist Started syncing Wikimedia installation... : Fix UploadWizard concurrency and minor message updates
21:07 Krinkle: Running small maintenance script foreachwiki to gather data about odd wgRC2UDPPrefix values (bug 28276)
20:21 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'CentralNotice security bugfix 48255'
20:15 Jeff_Green: authdns-update to swap in grosley for dead aluminium
19:41 RobH: magnesium is future racktables host, disregard errors for now
19:41 cmjohnson1: accidently removed power from magnesium
19:00 Reedy: Ensuring securepoll database tables exist on all wikis
18:35 logmsgbot: reedy synchronized php-1.22wmf3/extensions/CentralAuth
18:31 Jeff_Green: updating and rebooting grosley
18:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:17 logmsgbot: reedy synchronized wmf-config/
18:16 logmsgbot: reedy synchronized database lists files:
18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable EducationProgram on mkwiki'
18:07 Reedy: Created EducationProgram tables on mkwiki
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.22wmf3
18:00 RobH: authdnsupdate for racktables2
17:46 apergos: )
17:46 apergos: about to graceful apaches *and api servers* (which somehow are not in the actual apache graceful all script
17:42 Jeff_Green: kernel updates and reboot on pappas
17:29 logmsgbot: yurik Finished syncing Wikimedia installation... : Zero extension
17:18 RobH: restarting all mw jobrunners
17:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Disabled TTMServerMessageUpdateJob jobs'
17:12 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'revert hack'
17:11 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php 'revert hack'
17:08 logmsgbot: yurik Started syncing Wikimedia installation... : Zero extension
15:01 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
14:22 paravoid: ceph osd out 39, primary for stuck unclean pg 3.5c6
14:21 paravoid: ceph osd out 128, primary for stuck unclean pg 3.19e1
13:59 logmsgbot: mark synchronized wmf-config/mc-eqiad.php
13:59 mark: Removing mc1009 from the eqiad memcached pool
06:14 apergos: restarted adminbot (morebot) on wikitech-static

May 7

23:42 logmsgbot: catrope synchronized php-1.22wmf3/resources/startup.js 'touch'
23:42 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/modules/ve/ve.js 'touch'
23:39 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:35 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/ 'Appling update to fix CentralNotice message editing bug'
23:30 mutante: running addWiki.php for iegcomwiki
23:26 logmsgbot: dzahn synchronized wmf-config/InitialiseSettings.php
23:23 mutante: depool mw1041 in pybal
23:22 mutante: removed mw1041 from dsh groups mediawiki-installation and apaches
23:20 mutante: mw1041 - RT-4517, RT-5065, RT-5105 ...and broken again :p
23:17 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GettingStarted/ 'E3 Lightning deploy for GettingStarted'
23:17 LeslieCarr: upgrading srv* machines
22:48 Tim: on marmontel: disabled logrotate
22:36 logmsgbot: aaron synchronized wmf-config/mc-pmtpa.php 'removing mc15 for hw maint'
22:34 mutante: upgrading PHP_CodeSniffer on gallium
22:31 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62744/'
22:10 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/62650/'
22:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/
21:47 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:09 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueGroup.php
20:53 logmsgbot: aaron synchronized php-1.22wmf3/includes/job/JobQueueGroup.php 'Live hack to avoid translationNotificationJob jobs.'
20:44 logmsgbot: reedy synchronized database lists files:
20:30 andrewbogott: copied passwords from stafford to sockpuppet, then reset --hard on stafford to get things back in sync.
20:17 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:11 logmsgbot: reedy synchronized docroot/login/
20:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:02 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:57 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: loginwiki
19:56 logmsgbot: reedy synchronized wmf-config/
19:55 logmsgbot: reedy synchronized database lists files: loginwiki
19:37 mutante: DNS update - switch old secure.wm
19:30 mutante: apache-graceful-all, move old secure.wm redirects
18:54 cmjohnson1: authdns update
18:42 logmsgbot: dzahn synchronized docroot
18:42 mutante: sync-docroot for login.wm
18:38 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1042,1059 to s4'
18:03 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
17:59 logmsgbot: awjrichards synchronized php-1.22wmf3/extensions/MobileFrontend/stylesheets/common/ui.css 'Cherry-pick e37e9d9ae6b8a581569c39ede38fc21827983128'
17:43 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/FundraiserLandingPage/ 'Updating FundraiserLandingPage for WMF fundraising in FR'
17:38 kaldari: killed copyExistingEmailPreference.php on enwiki
17:29 Reedy: Killed php-1.21wmf1 through 1.21wmf5 on fenari and mediawiki-installation hosts
17:26 logmsgbot: mwalker Finished syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
17:19 mutante: killed hanging pdns related processes on ns1 and restarted pdns
17:15 kaldari: begin running extensions/Echo/maintenance/copyExistingEmailPreference.php on enwiki db
17:02 mutante: DNS update - add login.wikimedia and login.m.wikimedia
16:57 logmsgbot: mwalker Started syncing Wikimedia installation... : Update CentralNotice and core for HTMLForms, iframe preview, and devices!
16:37 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:35 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:31 RobH: professor being worked on, network is offline =(
16:29 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
16:19 gwicke: updated Parsoid to current master
16:03 logmsgbot: reedy synchronized multiversion/MWMultiVersion.php
15:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized docroot
14:47 cmjohnson1: authdns update
14:33 cmjohnson1: authdns update
13:09 mark: Power cycled line card 2 in csw1-esams (which has the SSL servers) to fix esams SSL reachability issues
09:15 logmsgbot: nikerabbit synchronized wmf-config/CommonSettings.php 'New ULS config part 2'
09:14 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'New ULS config part 1'
09:09 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
08:21 paravoid: ceph: all radosgw & mon + osd.24 upgraded to 0.56.6
07:59 paravoid: partially upgrading ceph to 0.56.6
07:27 Tim: on holmium: reverted firewall, too hard
07:22 Tim: on holmium: added some ad hoc iptables rules intended to keep the rest of the site mostly safe from it overnight
07:00 Tim: on holmium: chown'd to root all script files writable by www-data and all directories which are configured to allow PHP execution
06:28 Tim: on marmontel: disabled apache logrotate, stopped apache, disabled apache init script
04:20 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue May 7 04:19:39 UTC 2013
02:25 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue May 7 02:25:11 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue May 7 02:16:06 UTC 2013
01:29 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Two fixes to User::getNewMessageRevisionId() from kaldari'
01:29 logmsgbot: reedy synchronized php-1.22wmf3/extensions/WikimediaMaintenance
00:38 logmsgbot: reedy synchronized php-1.22wmf3/includes/User.php 'Live hack to stop fatals'
00:07 logmsgbot: bsitu synchronized php-1.22wmf3/extensions/Echo 'Update Echo to master'
00:07 logmsgbot: bsitu synchronized php-1.22wmf3/docs/hooks.txt 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:06 logmsgbot: bsitu synchronized php-1.22wmf3/RELEASE-NOTES-1.22 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:06 logmsgbot: bsitu synchronized php-1.22wmf3/includes/WikiPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/User.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:03 logmsgbot: bsitu synchronized php-1.22wmf3/includes/Skin.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'
00:02 logmsgbot: bsitu synchronized php-1.22wmf3/includes/OutputPage.php 'Adding GetNewMessagesAlert hook and wgUserNewMsgRevisionId JS global'

May 6

23:25 mutante: restarting prelabsdbdb on db1054
22:48 mutante: apache-graceful-all, add iegcom virtual host
21:00 mutante: blog is back up
21:00 mutante: holmium: bring up interface lo, comment "search" line in resolv.conf, restart varnish, ...
20:46 paravoid: swift replication scripts running on ms-fe1002 under screen -- kill if they have any effect on production
20:15 mutante: DNS update - sync with ns2
20:04 mutante: restarting parsoid on wtp1001
20:03 mutante: restarting pdns on ns0
20:01 mutante: killing parsoid server.js on wtp1001
19:59 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php 'f257ddac39194f8bfcfd145104999fdba83d33f5'
19:59 mutante: DNS update - switch all blog cnames over to holmium
19:30 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/
19:25 LeslieCarr: moved gateway from csw1-esams to cr1-esams
19:18 Krinkle: Reloading Zuul to deploy I259585e78ee
19:18 LeslieCarr: rebooting ns2
19:16 logmsgbot: reedy synchronized wmf-config/
19:13 binasher: rebuilt db1016 from hotbackup of db1001; pmtpa side of m1 needs to be rebuilt as well
18:42 RobH: ns2/nescio appears offline, investigating
18:37 RobH: ns2 appears offline
18:37 RobH: authdns-update crashed on ns2, rerunning
18:33 RobH: holmium
18:33 RobH: authdns-update for new server homium
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf3
17:18 LeslieCarr: marmontel's interface is now in the sandbox vlan
17:16 LeslieCarr: turned off marmontel's network port
16:12 logmsgbot: reedy synchronized docroot
13:55 hashar: gallium: installed manually libcidr0-dev libanon0-dev . Puppet change is 62434
09:51 paravoid: upgrading Ganglia Web to 3.5.7
08:25 hashar: Zuul upgrading to upstream version 7191ee8 + 1 wmf hack.
06:45 Nemo_bis: Translate search on vanadium keeps being broken https://toolserver.org/~nemobis/tmp/SearchTranslations.log (now RT-5085)
06:15 paravoid: pushing new swift rings: ms-be1 weight 50->0, ms-be2 33->75
02:19 paravoid: restarting gerrit
02:18 logmsgbot: LocalisationUpdate failed: git pull of extensions failed

May 5

10:56 Nemo_bis: "Search service is temporarily unavailable" since unknown time https://meta.wikimedia.org/wiki/Special:SearchTranslations?query=amen&uselang=en
03:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun May 5 03:36:17 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun May 5 02:25:44 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sun May 5 02:17:22 UTC 2013

May 4

22:42 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
21:44 Reedy: Manually force ran TorBlock/loadExitNodes.php
21:43 logmsgbot: reedy synchronized php-1.22wmf3/extensions/TorBlock/
16:03 logmsgbot: reedy synchronized wmf-config/
16:02 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
04:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat May 4 04:36:09 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat May 4 02:20:30 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Sat May 4 02:12:01 UTC 2013
01:17 Krinkle: binasher restarted gerrit

May 3

23:49 Krinkle: Gerrit is completely unresponsive (all http and ssh time out, serving HTTP 500). Filed details at https://bugzilla.wikimedia.org/show_bug.cgi?id=48061
22:19 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adding new pmtpa slave'
21:10 cmjohnson1: db10 is decommissioned...powering down for the final time
20:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
19:54 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/includes/MobileFrontend.hooks.php
07:52 MaxSem: OMG logmsgbot is a terminator!
06:16 logmsgbot: tstarling synchronized wmf-config/missing.php
04:03 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri May 3 04:03:12 UTC 2013
02:26 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri May 3 02:26:02 UTC 2013
02:16 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Fri May 3 02:15:45 UTC 2013
01:25 hashar: Zuul processed all its events, seems to be fine now. *sleep*
00:33 logmsgbot: dzahn synchronized docroot
00:32 mutante: sync-docroot for iegcom wiki
00:25 logmsgbot: aaron synchronized multiversion/MWMultiVersion.php '6512451451ac6617bedbfce7be17045bed794d70'
00:16 mutante: DNS update - add iegcom

May 2

23:20 mutante: restarting lucene on search1021, search1022, search1017, search1018 (with some waiting in between)
23:18 hashar: yeah after hours and hours of fighting, Jenkins is finally working again.
23:11 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
22:59 hashar: restarting jenkins… it got a few threads blocked and the main process is at 100% usage for now reason
22:59 mutante: repooling search1016
22:40 mutante: gallium, run puppet, graceful Apache to deploy split log files
22:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1020 for upgrade'
22:32 mutante: depooling search1016, restarting lucene, etc.. (Search#Adding_new_wikis)
22:29 mutante: repooling search1015
22:23 hashar: Zuul restarted. The bug about slowness is bug 48025
22:15 hashar: restarting Zuul
22:13 hashar: Deploying a workaround on Zuul to make it stop querying the Jenkins API when it just want to check whether a job exist. 62095
22:04 mutante: depooling search1015 in pybal
21:43 logmsgbot: mflaschen synchronized php-1.22wmf3/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf3 for E3 deployment'
21:42 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GuidedTour/ 'Sync GuidedTour to 1.22wmf2 for E3 deployment'
21:38 mutante: importing wikimania2014wiki into search indexers
21:35 mutante: restarting search indexers on searchidx2, searchidx1001 to make sure the indexer knows about new wiki
21:28 logmsgbot: mflaschen synchronized php-1.22wmf3/skins/common/shared.css 'Sync font-size change for edit section links'
21:15 logmsgbot: spage synchronized php-1.22wmf2/extensions/ConfirmEdit 'update 1.22wmf2 to wmf3 version of ConfirmEdit'
21:14 LeslieCarr: adding GEANT via fiberring to avoid-paths
20:21 logmsgbot: reedy synchronized php-1.22wmf3/includes/
20:15 logmsgbot: reedy synchronized docroot/bits/WikipediaMobileFirefoxOS/
19:51 Jeff_Green: authdns-update to move db1025 to frack.eqiad.wmnet
19:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
19:40 ottomata: rebooting oxygen
19:33 ottomata: re-enabling puppet on cp1031, it was administratively disabled. running puppet there.
19:24 RobH: professor was already powered down (why?) starting it back up now
19:23 ottomata: all webrequest udp2log loggers (squid and varnish) now send to gadolinium for socat unicast -> multicast relay
19:22 RobH: rebooting professor
19:16 Jeff_Green: moving db1025 into frack-fundraising1-c-eqiad
19:14 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo 'sync Echo ext for en.wiki'
19:14 ottomata: deployed squid frontend.conf.php changes to remove locke and send logs directly to gadolinium for multicast relay
19:11 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
19:09 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo 'sync Echo ext'
18:52 ottomata: removed varnishcsa-locke instance from varnish hosts: (dsh -c -g varnishncsa-all 'test -f /etc/init.d/varnishncsa-locke && service varnishncsa-locke stop && update-rc.d -f varnishncsa-locke remove && rm -v /etc/init.d/varnishncsa-locke')
18:47 logmsgbot: reedy synchronized wmf-config/interwiki.cdb 'Updating interwiki cache'
18:41 ottomata: varnishncsa now sends traffic to gadolinium instead of oxygen for multicast relay
18:30 Krinkle: Installed Monitoring plugin from Jenkins control panel <https://wiki.jenkins-ci.org/display/JENKINS/Monitoring>
17:59 Krinkle: Jenkins restart complete. No visible improvement. Jenkins is still idling most of the time while Zuul is still halted by an unknown factor on spawning jobs.
17:56 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:48 logmsgbot: reedy synchronized wmf-config/
17:37 Krinkle: Jenkins keeps clogging up. Starting an emergency restart.
17:00 Krinkle: Jenkins is nearing 100% CPU on gallium, what is Jenkins doing?
16:59 Krinkle: Zuul is somehow having trouble kicking off Jenkins jobs (less than 1 event processed per minute). Jenkins shows that 10/10 executors are idle. Investigating...
16:52 logmsgbot: nikerabbit synchronized php-1.22wmf3/extensions/UniversalLanguageSelector/ 'ULS to master'
15:13 hashar: Jenkins made Jenkins to instantly restart (was bug 47120) ) by deleting the downstream-buildview plugin.
15:11 hashar: restarted Zuul
14:12 hashar: Upgraded Jenkins from 1.480.3 to LTS 1.509.1. Restarted it.
14:10 hashar: shutting down Zuul
14:09 hashar: upgrading Jenkins (unscheduled maintenance).
13:01 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/61941/'
12:59 mark: Migrated amslvs3 and amslvs4 PyBal BGP peerings from csw2-esams to cr2-knams
09:55 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:53 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:50 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/ZeroRatedMobileAccess/ZeroRatedMobileAccess.body.php 'https://gerrit.wikimedia.org/r/#/c/61926/'
09:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/GeoData/GeoData.body.php 'https://gerrit.wikimedia.org/r/#/c/61962/'
06:04 apergos: powercycle snapshot1001, upgrade to precise
05:46 apergos: powercycle snapshot1004, upgrade to precise
05:28 apergos: snapshot1003 powercucle, upgrading to precise
05:28 Tim: on professor: manually started carbon-cache.py
05:15 Tim: professor is down, no response on serial console, rebooting
05:11 logmsgbot: tstarling synchronized wmf-config/jobqueue-pmtpa.php
05:08 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
04:47 Tim: on rdb1001-1002, set requirepass and masterauth in soft state
04:44 Tim: on mc1-16 and mc1001-1016, setting requirepass and masterauth to the new password in soft state
04:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu May 2 04:39:48 UTC 2013
04:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:59 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:57 logmsgbot: tstarling synchronized wmf-config/jobqueue-eqiad.php
03:47 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
03:40 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
02:46 logmsgbot: kaldari Finished syncing Wikimedia installation... :
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu May 2 02:32:10 UTC 2013
02:22 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Thu May 2 02:21:30 UTC 2013
02:21 Tim: on wikibugs-l: disabled bounce processing and re-enabled mail delivery to wikibugs-irc (was disabled due to excessive bounces)
02:21 logmsgbot: kaldari Started syncing Wikimedia installation... :
01:20 logmsgbot: mwalker synchronized php-1.22wmf3/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
01:17 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/modules/ext.centralNotice.bannerController/bannerController.js 'Poking bits to try and get the new banner controller deployed for CentralNotice'
00:16 mutante: @stafford:~# puppetstoredconfigclean.rb db10.pmtpa.wmnet

May 1

23:28 logmsgbot: tstarling synchronized php-1.22wmf3/extensions/GettingStarted
23:23 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
23:18 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/GettingStarted
22:42 mutante: powercycling frozen mw1041
22:14 logmsgbot: mwalker Finished syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
22:06 logmsgbot: dzahn synchronized ./multiversion/MWMultiVersion.php
22:02 logmsgbot: dzahn synchronized docroot
21:44 logmsgbot: mwalker Started syncing Wikimedia installation... : Updating CentralNotice from a8797938bc for historical allocations and some refactoring
21:32 logmsgbot: dzahn gracefulled all apaches
21:25 ottomata: syncing new redirects.conf to apachces to redirect wikipediazero.org
21:14 Jeff_Green: stopping mysql on db78 for cloning
20:50 ottomata: adding new wikipediazero.org zone alias, running authdns-update on dobson
19:45 logmsgbot: reedy synchronized php-1.22wmf3/cache/l10n/
19:15 mutante: DNS update - remove project2, anthony and yongle :p
19:05 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata
18:36 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 18:36:02 UTC 2013
18:32 mutante: DNS update - add wikimania2014 entries
18:31 RobH: disabled owa1-3 switchports, starting remote wipe
18:21 RobH: owa1/2/3 offlinging and wiping
18:20 logmsgbot: reedy synchronized php-1.22wmf3/extensions/ 'Update extensions for wikibase deploy'
18:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wiktionary to 1.22wmf3
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource, wikivoyage and wikiversity to 1.22wmf3
18:11 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikiquote to 1.22wmf3
18:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia to 1.22wmf3
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl to 1.22wmf3
17:58 logmsgbot: kaldari synchronized php-1.22wmf3/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
17:33 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/Hooks.php 'syncing Hooks in Echo'
17:20 LeslieCarr: puppet working on bits caches again after gerrit change 61793
17:01 RobH: torrus compiling, manutilus apache offline
16:53 RobH: torrus deadlocked, fixin
15:54 mark: Converted link between cr2-knams:xe-1/0/0 and csw2-esams:xe-0/1/1 to LACP aggregated link cr2-knams:ae1 / csw2-esams:ae2
14:13 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
14:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
10:30 logmsgbot: reedy synchronized php-1.22wmf2/extensions/ParserFunctions/ 'Update to master to fix common fatals'
10:08 mark: Removed old Toolserver IP range 91.198.174.192/27 on primary and backup router interfaces
03:50 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed May 1 03:49:55 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Wed May 1 02:27:30 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed May 1 02:16:40 UTC 2013
02:09 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Turned autoResync back on again for multiwrite backend'
00:46 logmsgbot: tstarling synchronized wmf-config/PrivateSettings.php

April 30

23:52 Krinkle: Gracefully restarted Zuul on gallium
23:52 Krinkle: Ryan_Lane restarted the gerrit service
23:44 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo'
23:34 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo'
23:28 logmsgbot: kaldari synchronized /php-1.22wmf3/extensions/Echo 'Updating Echo for some minor bugfixes'
23:18 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Revert MobileFrontend'
23:15 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Revert MobileFrontend'
23:12 logmsgbot: tstarling synchronized wmf-config/missing.php 'rewrite (I17989fc4)'
23:01 Krinkle: Graceful restart of Zuul on gallium
22:56 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Enabling Extension:AccountAudit on all wikis and wmgEchoHelpPage'
22:09 Jeff_Green: rolling back db1025 change b/c "mysqld: unknown option '--federated'"
22:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'grrr'
22:00 logmsgbot: maxsem synchronized wmf-config
21:58 Jeff_Green: restarting mysqld on db1025 to enable federated engine for testing
21:56 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
21:40 logmsgbot: maxsem synchronized php-1.22wmf3/extensions/MobileFrontend/ 'Weekly mobile deployment'
20:04 logmsgbot: kaldari synchronized echowikis.dblist 'syncing echowikis.dblist for cron jobs'
19:59 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'turning on Echo for English Wikipedia'
19:49 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:19 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:03 cmjohnson1: removing db10 certs from puppet
17:48 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing CommonSettings for Echo deployment'
17:45 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing InitialiseSettings for Echo deployment'
13:57 cmjohnson1: jenkins seems to not be working -- restarting
11:27 Nikerabbit: refresh-translatable-pages.php finished for mediawikiwiki and metawiki
04:06 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 30 04:06:11 UTC 2013
02:34 logmsgbot: LocalisationUpdate completed (1.22wmf3) at Tue Apr 30 02:33:50 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 30 02:15:03 UTC 2013
02:04 paravoid: cleaning up neon's /var/log (100% full), restarting icinga...
00:59 logmsgbot: catrope synchronized php-1.22wmf3/extensions/VisualEditor/ 'Update VE'
00:44 logmsgbot: pgehres synchronized wmf-config/InitialiseSettings.php 'Touching InitialiseSettings'
00:32 logmsgbot: pgehres Finished syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing
00:00 logmsgbot: pgehres Started syncing Wikimedia installation... : Deploying Extension:AccountAudit and an Echo thing

April 29

23:20 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'updated s5 pmtpa topology'
23:09 binasher: s5 pmtpa master changed to db73 - MASTER_LOG_FILE='db73-bin.000090', MASTER_LOG_POS=978211000
23:08 AaronSchulz: finished creating AccountAudit tables
23:06 logmsgbot: anomie synchronized wmf-config/InitialiseSettings.php 'Re-enable WikiLove, bug fix is deployed'
23:04 logmsgbot: catrope Finished syncing Wikimedia installation... :
22:40 logmsgbot: catrope Started syncing Wikimedia installation... :
22:28 anomie: (Including backport of gerrit change 61494 in the scap)
22:27 RoanKattouw: Running scap
22:23 RoanKattouw: Running scap again
22:19 logmsgbot: catrope Started syncing Wikimedia installation... :
22:13 hashar: gallium: restarted apache2 to kill off http connections established by mod_proxy to the jenkins backend.
22:03 hashar: jenkins web server is no more responding on port 8080. Might have to restart it.
22:00 hashar: gallium / jenkins : blackholed a web drawer using ip route add blackhole <IP>/32
21:54 hashar: Jenkins is in trouble
21:52 logmsgbot: reedy synchronized php-1.22wmf3/includes/DefaultSettings.php 'Fix wgVersion'
21:31 notpeter: upgrading db45 to precise
21:24 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enabling VisualEditor on viwiki and test2wiki'
20:31 AaronSchulz: Doing another original/math/timeline sync/copy run
20:28 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Enabled multiwrites for ceph/swift'
19:19 logmsgbot: kaldari synchronized php-1.22wmf2/extensions/Echo/maintenance/setEmailOptionTemp.php
19:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf3
19:06 logmsgbot: reedy Finished syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
18:46 logmsgbot: reedy Started syncing Wikimedia installation... : Scap again to see if that fixes ROOTPAGENAME and ROOTPAGENAMEE. Otherwise I'm reverting it
18:37 logmsgbot: reedy synchronized php-1.22wmf2/.git/modules/extensions/WikiLove
18:21 logmsgbot: reedy synchronized php-1.22wmf3/extensions/
18:06 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
18:03 notpeter: deploying new frontend squid conf
17:51 LeslieCarr: restarting icinga-wm
17:35 cmjohnson1: removing asw-c1 for replacement
17:30 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf3 and rebuild l10n cache
17:09 mutante: fix NRPE checks on ms-be2, running as "weird user id" bug
17:05 mutante: removing reinstalled ms-be2 from ssh_known_hosts with ssh-keygen and then making it world-readable again manually so deployers don't get errors :p
17:05 logmsgbot: csteipp synchronized php-1.22wmf2/includes/
17:05 logmsgbot: reedy synchronized live-1.5
17:02 mutante: attempt to fix NRPE checks on ms-be9, restart as correct user
16:49 logmsgbot: reedy synchronized docroot
16:44 logmsgbot: reedy synchronized php-1.22wmf3/ 'initial sync of 1.22wmf3 files'
03:26 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 29 03:25:35 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 29 02:12:33 UTC 2013

April 28

13:14 hashar: restarting Zuul, it has a finished build stuck in the `postmerge` pipeline.
02:49 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 28 02:48:51 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 28 02:12:10 UTC 2013

April 27

03:07 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 27 03:06:47 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 27 02:13:53 UTC 2013
00:08 mutante: make SSL on Bugzilla also use SSLCACertificatePath and c_rehash certs on kaulen

April 26

23:58 mutante: upgrading mysql packages on marmontel (blog)
23:53 mutante: make SSL on the Blog use SSLCACertificatePath and c_rehash certs on marmontel
20:11 hashar: killed a node process on gallium. Probably a daemon that has gone standalone when jenkins died
19:19 hashar: jenkins died again sorry :( But this time I most probably found out the reason why it is slowed down: a web spider attempt to crawl all our builds!
18:04 mwalker: Upgraded fundraising civicrm install from a3db312836b6ac004694f831071b1c7d00560fbf to a65fbcec0dbb5bb79a97aca220a998e56d08b067
15:56 hashar: jenkins: enabled ExtraSettings.php loading 61022. That enables a bunch of MediaWiki settings for all mediawiki related jobs, namely displaying errors, moving the temp dir to tmpfs, showing exceptions details and enabling development warnings.
13:49 mark: Rebooting hooft
13:48 mark: dist upgrade of hooft to precise
13:48 mark: Moved udpmcast from hooft to nescio
13:16 ^demon: gerrit replication freaked out, restarting
06:38 hashar: Restarted Jenkins. It was stuck in some while loop doing tons of stat() on unexisting file :( Would be up in roughly 40 minutes :(
03:46 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 26 03:45:44 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 26 02:12:38 UTC 2013
00:57 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'temporarily disabling WikiLove per bug 47457'
00:53 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
00:33 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Clean up VisualEditor config'
00:33 Tim: on fenari: fixed permissions on /etc/ssh/ssh_known_hosts
00:33 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Remove wmgVisualEditorHide'

April 25

23:09 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing new favicons for incubator, wiktionary, wiktionary.en, wikibooks, wikinews, and wikispecies'
22:32 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:22 ^demon: new ssl cert (gerrit.wikimedia.org) live on gerrit.
20:55 notpeter: reimaging db55
20:45 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:17 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:07 awight: payments cluster updated from 7f70ccd580d to fdf308c8f
19:12 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating for new schema number'
18:52 logmsgbot: reedy synchronized docroot
18:19 LeslieCarr: cleaning up old home directories on fenari
17:00 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Removing oldskin notice'
16:50 logmsgbot: catrope synchronized wmf-config/InitialiseSettings.php 'Enable VisualEditor on 14 more Wikipedias'
16:38 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor/ 'Update VisualEditor to master'
16:20 hashar: Jenkins is busy reparsing all the build history for some reason. That slow down the continuous integration box by an order of magnitude :/ Should take roughly 40 minutes (no idea when it started)
16:09 hashar: Zuul git operations are slow again :/
14:57 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'rm debug hack'
14:56 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log back off, its not phps fault'
14:53 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch debug log on'
14:52 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php 'Debugging'
11:56 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I2f0990fa087a'
11:56 logmsgbot: krinkle synchronized docroot
03:40 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 25 03:39:44 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 25 02:14:20 UTC 2013
00:45 mutante: upgrading packages on manutius
00:30 mutante: DNS update - removing internproxy, removing tesla (minus mgmt)

April 24

23:04 logmsgbot: dzahn gracefulled all apaches
22:39 logmsgbot: dzahn gracefulled all apaches
22:39 LeslieCarr: upgrading mw's
22:38 LeslieCarr: replaced smokeping targets for sdtpa c3 and sdtpa d3
22:26 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'New location for icons - Ib5554c54729'
22:26 mutante: sync-apache, add coffee
22:24 logmsgbot: krinkle synchronized docroot/bits/apple-touch/ 'New location for icons - Ib5554c54729'
21:47 mutante: DNS update - remove survey, remove wikitech-old, remove argon
21:47 logmsgbot: demon synchronized php-1.22wmf2/extensions/Wikibase 'b4b5fb4046175ca29834a4669c253c97ac5ccd07'
21:38 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php
21:34 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Turn mwsearch log back on'
21:33 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
21:26 mutante: DNS update - readding voip
21:17 cmjohnson1: removing sq33 from dsh node groups and pybal cfg
20:32 RobH: racktables on hooper upgraded to recent stable
20:26 RobH: racktables flap again, working on it
20:16 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch
20:04 logmsgbot: demon synchronized php-1.22wmf2/extensions/MWSearch 'Deploying I48480cb6'
20:03 logmsgbot: demon synchronized php-1.22wmf2/includes/api/ApiQuerySearch.php 'Deploying Ib7abe8d5'
19:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/filerepo/file/LocalFile.php 'deployed 0e42bdadd9349fe5b6119ff65322e6c52d813a6e'
19:17 mutante: deploying Bugzilla wikimedia extension changes for improved wikilinking
19:16 RobH: racktables going offline a moment for upgrade
18:45 logmsgbot: reedy synchronized wmf-config/ 'wikidata stuffs'
18:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything non enwiki to 1.22wmf2
18:09 mutante: DNS update - removing unused remnants
18:09 sbernardin: replacing bad disk on db59
17:31 sbernardin: taking down ms-be2 for hw replacement
15:09 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
15:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/skins/MinervaTemplate.php 'bug 47597'
09:05 hashar: restarted Zuul. Upgraded to e403579 (latest master)
08:58 hashar: upgrading Zuul
06:04 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
03:42 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 24 03:42:12 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 24 02:31:43 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 24 02:18:59 UTC 2013
00:03 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Disable mobile sitenotice everywhere but testwiki'
00:03 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Disable mobile sitenotice everywhere but testwiki'

April 23

23:38 mutante: cleaning up all unused Apache sites on singer, gzip, move to /root for backup
23:33 mutante: disabling old techblog apache site config on singer
23:32 logmsgbot: maxsem synchronized wmf-config/mobile.php
23:15 logmsgbot: mwalker synchronized docroot/bits/favicon/ 'Pushing wikidata, wikimania, wikiquote, and wikiversity favicons from Isarra'
23:12 logmsgbot: dzahn gracefulled all apaches
23:07 logmsgbot: maxsem synchronized wmf-config 'Mobile banners again, hopefully fixed'
23:05 mutante: sync-apache, add wikepdia.org and .com typo domains
22:28 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
22:21 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score
22:12 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
22:05 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
22:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'rv debugging'
22:00 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Score'
21:59 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Score/Score.body.php 'debugging'
21:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable score debug log'
21:40 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/60512/'
21:23 logmsgbot: maxsem synchronized wmf-config
21:17 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Score
21:15 logmsgbot: maxsem synchronized wmf-config 'Mobile banners everywhere - https://gerrit.wikimedia.org/r/#/c/60362/'
21:14 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'temp logging'
21:13 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/ 'Weekly mobile deployment'
21:12 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
21:06 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/ZeroRatedMobileAccess/ 'Weekly mobile deployment'
21:05 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/ 'Weekly mobile deployment'
20:56 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
19:36 logmsgbot: anomie synchronized php-1.22wmf2/extensions/Cite 'Update to master for bug 47291'
18:44 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on enwiki'
17:49 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:49 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
16:34 logmsgbot: reedy synchronized php-1.22wmf2/extensions/TorBlock
16:34 logmsgbot: reedy synchronized php-1.22wmf1/extensions/TorBlock
12:34 paravoid: reprepro python-statsd 2.0.1-1~wmf1 & voluptuous 0.6.1-1~wmf1 for precise-wikimedia
09:57 pp-pdf1: restarted all services
09:56 pp-pdf1: - alow numprocs to be configured via initenv.sh
09:16 pp-pdf1: restarted all services
09:12 pp-pdf1: restarted all services
09:10 pp-pdf1: restarted all services
08:46 logmsgbot: maxsem synchronized docroot/ 'https://gerrit.wikimedia.org/r/#/c/60375/'
08:41 logmsgbot: maxsem synchronized wmf-config 'Beta changes, no-op for production'
02:25 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Tue Apr 23 02:25:25 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 23 02:10:32 UTC 2013
02:07 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 23 02:06:54 UTC 2013
00:02 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves (this time for real)'
00:01 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'bookkeeping for pmtpa slaves'

April 22

23:51 notpeter: reimaging db44
22:57 binasher: mysql query logs / ishmael migrated from db9 to db1001
21:41 LeslieCarr: upgrading even more mw servers in tampa
20:12 Reedy: Running namespaceDupes on kowikiversity
19:57 Reedy: Running namespaceDupes on zhwikibooks
19:55 Reedy: Running namespaceDupes on uzwiki
19:55 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy synchronized images/
19:50 logmsgbot: reedy synchronized docroot
19:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Score extension on the cluster'
18:49 logmsgbot: catrope synchronized echowikis.dblist
18:44 LeslieCarr: sq33 is going to be decomissioned… long live…. nothing else in tampa
18:39 mutante: changing #mediawiki to #wikimedia-dev in wikibugs startup script (just in case, should not even be needed by puppetized ircecho)
18:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:26 LeslieCarr: still doing package upgrades on mw* machines
18:20 LeslieCarr: rebooting nonresponsive sq33
18:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf2
17:37 LeslieCarr: dist-upgrading some mw machines, should be low/no impact
16:51 mutante: restarting wikibugs on mchenry
16:43 logmsgbot: mark synchronized wmf-config/InitialiseSettings.php
16:37 mutante: added abaso to wmf-deployment group on gerrit (new mediawiki deployer)
16:11 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'New caching everywhere'
16:02 logmsgbot: reedy synchronized php-1.22wmf2/extensions/Wikibase
15:57 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php '$wgMFVaryResources on enwiki, https://gerrit.wikimedia.org/r/#/c/60286/'
14:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:15 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Mon Apr 22 02:14:58 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 22 02:08:56 UTC 2013
02:06 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Mon Apr 22 02:05:30 UTC 2013
00:13 Reedy: Running namespaceDupes.php --fix via foreachwiki in screen session on terbium

April 21

02:37 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sun Apr 21 02:37:21 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 21 02:16:56 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sun Apr 21 02:10:46 UTC 2013

April 20

16:54 ottomata: resetarting apache2 on stafford in attempt to fix puppetmaster
02:36 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Sat Apr 20 02:35:55 UTC 2013
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 20 02:14:31 UTC 2013
02:09 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Sat Apr 20 02:09:07 UTC 2013
00:09 logmsgbot: aaron synchronized php-1.22wmf1/includes/job/JobQueueRedis.php 'deployed c9e702e7f73f41f7bcd2deb045085689d5cb4229'
00:03 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/JobQueueRedis.php 'deployed d2e4fdfed03148fdd1a564925ce351bec6f9b935'

April 19

23:07 Coren: brief reboot of labstore3 -> test to see that everything starts up as expected.
20:39 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Setup redis job queue debug log'
18:36 Jeff_Green: increased php memory limit for fundraising civicrm instance
17:55 LeslieCarr: restarting etherpad on hooper
17:15 binasher: running enwiki.abuse_filter_log migration - drop afl_user index, added user_timestamp
16:10 LeslieCarr: flushed celsus varnish cache
06:31 Tim: deploying apache conf change for www.wikidata.org redirect (I7bb872fd)
03:33 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Fri Apr 19 03:32:40 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 19 02:27:17 UTC 2013
02:26 logmsgbot: aaron cleared profiling data
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Fri Apr 19 02:16:42 UTC 2013
01:54 logmsgbot: aaron cleared profiling data
01:54 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Removed cli profiling for now.'
00:29 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
00:29 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'

April 18

23:40 logmsgbot: catrope synchronized php-1.22wmf2/extensions/VisualEditor 'Update VisualEditor to master'
23:40 logmsgbot: catrope synchronized php-1.22wmf1/extensions/VisualEditor 'Update VisualEditor to master'
23:38 logmsgbot: catrope synchronized wmf-config/CommonSettings.php 'Deploy 5c5455c81bc8dee0ae2a0410655da639e2f84e87 ($wgVisualEditorParsoidProblemReportURL)'
23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/processEchoEmailBatch.php 'More echo db fix'
23:25 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/DbEchoBackend.php 'More echo db fix'
22:51 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Enable mobile sitenotice on testwiki'
22:51 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Enable mobile sitenotice on testwiki'
22:48 notpeter: reimaging db35
22:46 logmsgbot: mwalker synchronized docroot/bits/favicon/wikisource.ico 'New and shiney ws.o favicon by Isarra'
22:45 logmsgbot: mwalker synchronized docroot/bits/favicon/mediawiki.ico 'New and shiney mw.o favicon by Isarra'
22:35 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
22:35 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'adjusting weights on pmtpa slaves'
22:26 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
22:15 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
22:09 notpeter: correction: that was db33 out, not db31
22:03 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'db31 and db35 out, db72 and db73 in'
21:56 logmsgbot: bsitu synchronized php-1.22wmf2/extensions/Echo/includes/EchoDbFactory.php 'Db cache fix'
21:41 logmsgbot: reedy synchronized php-1.22wmf2/extensions/EducationProgram
21:39 logmsgbot: reedy synchronized php-1.22wmf1/extensions/EducationProgram
21:38 mutante: merging changes on bugzilla to now support RT in "See also" field
20:45 logmsgbot: mflaschen synchronized php-1.22wmf2/extensions/GettingStarted/ 'Sync GettingStarted 1.22wmf2 for E3 deploy'
20:43 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Sync GettingStarted for E3 deploy'
20:36 logmsgbot: reedy synchronized wmf-config/
20:15 logmsgbot: kaldari Finished syncing Wikimedia installation... :
19:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'wgMaxImageArea and wgMaxAnimatedGifArea to 50MP'
19:52 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:14 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Echo email config + db cluster update to testwiki'
19:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Echo email config + db cluster update to testwiki'
18:57 logmsgbot: aaron synchronized php-1.22wmf2/includes/job/jobs 'deployed 1e3eaed6fbe7fd91c15399cc17eac8602b34c9cd'
18:28 logmsgbot: reedy Finished syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
17:57 logmsgbot: reedy Started syncing Wikimedia installation... : rebuild l10n cache for wikidata deploy
17:50 RobH: authdns0update for git.w.o
17:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.22wmf2
17:06 logmsgbot: reedy synchronized php-1.22wmf2/extensions/
16:58 logmsgbot: reedy synchronized php-1.22wmf1/extensions/Wikibase '5d496b4ea2c8a68b9e1280a282530f7ce5232b68'
16:47 logmsgbot: ariel synchronized wmf-config/throttle.php
16:10 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
16:10 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
15:39 cmjohnson1: authdns update
15:10 mark: Reversed replication of volume fr_archive, now nas1001-a:fr_archive -> nas1-a:fr_archive
14:26 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:24 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:14 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
14:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
14:04 logmsgbot: maxsem synchronized php-1.22wmf2/extensions/MobileFrontend/includes/MobileContext.php 'https://gerrit.wikimedia.org/r/#/c/59818/'
13:24 logmsgbot: maxsem synchronized wmf-config/mobile.php
13:03 mark: Rebooting cp1041
08:48 logmsgbot: maxsem synchronized wmf-config/mobile.php
07:57 logmsgbot: olivneh synchronized php-1.22wmf2/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
07:57 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ConfirmEdit 'Updating extension/ConfirmEdit (Bug 46132 / Change I8ee3fd136)'
07:43 hashar: jenkins raised max # of executors from 6 to 8
07:07 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/59767/'
05:59 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:21 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
05:15 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch/MWSearch_body.php
05:12 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php
05:09 logmsgbot: tstarling synchronized php-1.22wmf1/includes/specials/SpecialSearch.php
03:57 logmsgbot: tstarling synchronized php-1.22wmf2/extensions/MWSearch
03:55 logmsgbot: tstarling synchronized php-1.22wmf2/includes/HttpFunctions.php
03:48 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/MWSearch
03:46 logmsgbot: tstarling synchronized php-1.22wmf1/includes/HttpFunctions.php
03:27 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Thu Apr 18 03:26:48 UTC 2013
02:24 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 18 02:23:54 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Thu Apr 18 02:14:26 UTC 2013
01:03 mutante: re-enabling Tomasz' RT account
00:42 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'setting db1004 to full weight'
00:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1004 at a warmup weight'
00:09 logmsgbot: demon synchronized docroot

April 17

23:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1004'
23:21 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting 'Finished update of CN and CR to latest'
23:20 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice/
23:14 hashar: If jenkins/zuul goes wild overnight: connect to gallium, killoff puppet in crontab then /etc/init.d/zuul stop and drop me an email. Will take of it whenever I wake up.
23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
23:12 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Beginning update of CentralNotice and ContributionTracking to latest'
22:59 hashar: restarting Zuul
22:20 hashar: Jenkins repointing all jobs to use the new Zuul repo ( /srv/ssd/zuul/git ) 59744
22:06 hashar: jenkins restarted. Restarting Zuul.
21:20 hashar: stopped Zuul. Will wait a bit till jenkins start up
21:19 hashar: restarted Jenkins
21:19 hashar: restarted Zuul
21:16 hashar: finished migrating zuul git repositories ( rsync -av /var/lib/zuul/git /srv/ssd/zuul )
21:09 hashar: Zuul ETA: it is copying MobileFrontend extension right now so half way done
21:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionaries to 1.22wmf2
21:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.22wmf2
21:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikiversity to 1.22wmf2
21:00 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki back to 1.22wmf1
20:59 hashar: stopping Jenkins that somehow restarted
20:59 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'moar'
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:55 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews.dblist and special wikis to 1.22wmf2
20:51 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
20:49 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikimania and wikimedia to 1.22wmf2
20:44 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Some wikis to create some noise
20:42 logmsgbot: reedy synchronized php-1.22wmf2/includes/db/DatabaseMysql.php 'debugging'
20:30 hashar: Switching zuul dir to /srv/ssd/zuul ( 58898 and 58899 )
20:18 hashar: stopping Zuul gracefully
20:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Revert that due to large number of warnings...
20:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: fishbowl, private and special to 1.22wmf2
19:57 hashar: gallium slowness is partly due to the ssd install. The git repositories are on the slow raid where as the workspaces are on the ssd. That means `git clone` can't do hardlink anymore :(
19:43 binasher: enwiki, dewiki, wikidata are fully on mariadb 5.5.30
19:33 hashar: jenkins: reducing number of executors from 10 to 6.
19:32 hashar: jenkins gallium has a huge queue of jobs. investigating.
19:31 hashar: jenkins / gallium has a huge queue of jobs. Investigating.
19:19 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 's1/s5 writeable from pmtpa'
19:17 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 writeable'
19:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1/5 read-only'
19:12 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'preparing for master swaps'
19:03 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1437b57bd475e5ecb06800f9f64b14ccffd0dedf'
19:00 logmsgbot: aaron synchronized php-1.22wmf2/maintenance/doMaintenance.php 'deployed 1306792ad24be5b8d17355c58da655eb88c2fcc6'
18:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1050'
18:18 LeslieCarr: installing new linecard in cr2-eqiad , .1% chance of issues
18:16 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050'
17:50 logmsgbot: aaron synchronized php-1.22wmf2/includes/User.php 'deployed 6a66776fd7afdf883ad6f82270453fc76833508b'
16:20 ^demon: running puppet on manganese (will restart gerrit)
13:05 apergos: restarted icinga on neon (the usual: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe)
13:03 hashar: jenkins: refreshing job mediawiki-core-phpcs-HEAD (points to a wrong git repo path)
09:50 mark: Inserted libvpx 1.1.0-1+wmf1 packages into the precise-wikimedia APT repository
06:57 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 06:57:02 UTC 2013
06:51 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 06:51:28 UTC 2013
06:48 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 06:47:37 UTC 2013
03:14 logmsgbot: LocalisationUpdate ResourceLoader cache refresh completed at Wed Apr 17 03:13:56 UTC 2013
02:20 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Wed Apr 17 02:19:57 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 17 02:10:22 UTC 2013
00:38 mutante: blog: enabled new eventlogging by ori, removed old logging class, converted plugins/WMBlog from svn to git and sync with gerrit
00:31 Tim: on hume: changed UID for l10nupdate to be 10002, for consistency with puppet admins.pp

April 16

23:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1058 weight'
23:57 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1005, adding db1058 to s5'
23:55 logmsgbot: aaron cleared profiling data
23:49 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:48 logmsgbot: reedy synchronized wmf-config/abusefilter.php
23:46 logmsgbot: aaron synchronized wmf-config/StartProfiler.php 'Enabled 1:1 profiling for cli scripts and put "cli" in the profile ID'
23:22 logmsgbot: awjrichards synchronized php-1.22wmf2/extensions/MobileFrontend/
23:22 logmsgbot: awjrichards synchronized php-1.22wmf1/extensions/MobileFrontend/
23:00 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/ContributionReporting/ 'Contribution reporting now pulls 2013 data'
22:59 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/ContributionReporting/
22:51 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 22:51:08 UTC 2013
22:37 Tim: running l10nupdate manually on fenari to test change 58911
22:03 logmsgbot: anomie synchronized php-1.22wmf2/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
22:02 logmsgbot: anomie synchronized php-1.22wmf1/extensions/WikimediaMaintenance 'Update WikimediaMaintenance to master for l10nupdate fix'
21:49 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/59449'
21:46 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/ZeroRatedMobileAccess/ 'https://gerrit.wikimedia.org/r/#/c/59525/'
21:43 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
19:54 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:53 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:38 logmsgbot: pgehres synchronized php-1.22wmf2/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:33 logmsgbot: pgehres synchronized php-1.22wmf1/extensions/CentralAuth/maintenance/migrateAccount.php 'Updating CentralAuth/migrateAccount.php'
19:18 binasher: upgrading db1005 to precise + mariadb
19:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1005 from s5 for upgrade'
19:02 binasher: streaming s5 hotbackup from db45 to db1058
18:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1058 from s1'
18:34 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/HTMLCacheUpdate.php 'deployed 344ddf85b6d7b79c40a1b6b9ec7e0a1e05c0705b'
18:26 logmsgbot: mlitn synchronized php-1.22wmf2/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
18:26 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
18:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turning on footer contact link for en.wiki'
18:13 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
18:13 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Enable email bundling for test2wiki and mediawikiwiki'
16:42 logmsgbot: aaron synchronized php-1.22wmf2/includes/WikiPage.php 'deployed 38d60cdafe036625a6ec8fa3a9e8a94ef450c85c'
16:41 logmsgbot: aaron synchronized php-1.22wmf2/includes/db/Database.php 'deployed fa819f8e3579855f6d95552655de98f92ec1789c'
16:40 logmsgbot: aaron synchronized php-1.22wmf2/includes/cache/BacklinkCache.php 'deployed 1a94124385df4c452c68fadaa74bc2714a204761'
16:37 logmsgbot: aaron synchronized php-1.22wmf2/includes/OutputPage.php 'deployed 997ab92458946d10fed96ff4ba41c74eeaee0d20'
05:04 Tim: moving most cron jobs from hume to terbium
03:23 legoktm: test
02:35 logmsgbot: LocalisationUpdate completed (1.22wmf2) at Tue Apr 16 02:34:59 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 16 02:16:58 UTC 2013

April 15

23:57 logmsgbot: csteipp synchronized php-1.22wmf2/includes/ 'User gerrit versions of security fixes'
23:18 logmsgbot: mwalker synchronized php-1.22wmf2/extensions/CentralNotice
23:13 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice/ 'CentralNotice banner= fix'
22:31 logmsgbot: dzahn gracefulled all apaches
22:16 andrewbogott: Moved labs-morebots to tools project.
21:43 LeslieCarr: doing a few more random host upgrades of misc hosts
21:32 LeslieCarr: upgrading mw1070-mw1079
21:24 LeslieCarr: upgrading mw1060-mw1069
21:22 RobH: linode wikitech is now shut down \o/
21:13 logmsgbot: dzahn gracefulled all apaches
20:32 mutante: restarting pdns on ns1
20:30 mutante: DNS update - add wikpedia.org
19:37 logmsgbot: ariel gracefulled all apaches
18:15 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.22wmf2
15:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
15:19 logmsgbot: reedy Started syncing Wikimedia installation... : Build localisation cache for 1.22wmf2
14:59 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.22wmf2, scap to come
14:56 logmsgbot: reedy synchronized docroot
14:55 logmsgbot: reedy synchronized w
14:54 logmsgbot: reedy synchronized wmf-config/
14:49 logmsgbot: reedy synchronized php-1.22wmf2/ 'Initial sync of 1.22wmf2'
12:24 mark: Depooled amssq47-amssq62 frontend squids
12:04 mark: Depooled knsq16-knsq22 frontend squids
09:20 hashar: jenkins died :(
08:22 hashar: updating Jenkins jobs to fetch ZUUL_COMMIT instead of ZUUL_BRANCH. That would solve a race condition between jobs ( bug 46723 / 58865 ).
05:50 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Adding throttling exception for itwiki GLAM event which starts in four hours (Bug 47221, change Id9ae600ac)'
02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 15 02:03:51 UTC 2013

April 14

02:04 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 14 02:04:17 UTC 2013

April 13

23:54 Coren: rebooting labstore3. autofs is being silly
15:11 Coren: Fixed boot problem on labstore3; back to operating.
04:54 Coren: rebooted labstore3 again; kernel and lvm2 upgrade
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 13 02:12:27 UTC 2013
01:10 Coren: rebooted labstore3 again -> we hates lockd. We hates it!
00:41 Coren: rebooting labstore3 -> silly lockd module

April 12

23:31 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Contact Wikipedia' footer link on test/test2 (Iab1f5f527)'
23:14 Coren: rebooted labstore3 -> hung MegaCLI processes
22:41 notpeter: temp depooling search1015
20:53 mutante: install package upgrades on sanger
19:57 Ryan_Lane: finished opendj upgrade on virt0
19:23 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Disable mwsearch log'
18:54 mutante: restarting parsoid on wtp1004
18:25 logmsgbot: aaron synchronized wmf-config/jobqueue-pmtpa.php 'Set pmtpa queue config to match eqiad'
18:17 logmsgbot: demon synchronized php-1.22wmf1/extensions/MWSearch/MWSearch_body.php 'Moar logging'
18:16 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Moar logging'
18:00 Ryan_Lane: upgrade opendj on virt0
16:00 LeslieCarr: adding another link to asw-c-eqiad's uplink
15:05 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
15:02 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Debug group for mwsearch'
15:01 hashar: Changing Jenkins workspace directories from ${ITEM_ROOTDIR}/workspace to /srv/ssd/jenkins/workspace/${ITEM_FULLNAME} . My change has not been taken in account early on :(
14:54 hashar: jenkins is back up. Starting Zuul
14:44 cmjohnson1: analytics1007 has never worked..but working on it
14:21 hashar: restarting jenkins
14:03 hashar: gallium: created primary partition on /dev/sdb . formatting using: mkfs.xfs /dev/sdb1
13:36 hashar: gallium can't find the SSD, shut downing, will attempt another slot
12:59 hashar: gracefully shut-downing Jenkins and Zuul for scheduled maintenance. Will shutdown server gallium just after.
09:39 hashar: jenkins: updated mediawiki-core-whitespaces job to use ZUUL_COMMIT as a refspec specifier (for bug 46723 )
02:15 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 12 02:15:30 UTC 2013
00:08 mutante: DNS update - point old bugzilla3 entry over to actual bugzilla server

April 11

23:35 LeslieCarr: upgrading mw1052 to mw1060
23:30 logmsgbot: csteipp synchronized php-1.22wmf1/extensions/RSS/
23:30 LeslieCarr: upgrading mw1051
23:24 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
23:22 mutante: shutting down xenon
23:20 LeslieCarr: upgrading mw1050
23:11 LeslieCarr: upgrading 1040 to 1049
23:04 mutante: install package upgrades on neon
23:00 LeslieCarr: upgrading mw1030 - 1039
22:51 K4-713: synchronized payments to 7f70ccd580d64
22:47 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralAuth 'Peter's log line change to CentralAuth'
22:41 logmsgbot: mwalker synchronized wmf-config/CommonSettings.php 'Enable translate on CentralNotice'
22:39 logmsgbot: mwalker synchronized docroot/bits/favicon/wikipedia.ico 'Bi resolution wiki favicon'
22:38 LeslieCarr: upgrading mw1120 to mw1129
22:37 mutante: removing spence from dsh group mediawiki-installation
22:33 logmsgbot: mwalker synchronized php-1.21wmf12/extensions/CentralNotice
22:26 logmsgbot: mwalker synchronized php-1.22wmf1/extensions/CentralNotice 'Updating CentralNotice to the latest and greatest (from 68f0262be838320ecc9c87eb4485c28b0699bc1f)'
22:22 LeslieCarr: upgrading mw1110 to mw1119
21:41 logmsgbot: kaldari synchronized php-1.22wmf1/extensions/Echo 'syncing Echo for bugfix'
21:35 ottomata: emery back up and upgraded to precise (thanks mutante!)
21:32 ^demon: fenari: php-1.21wmf12 and php-1.22wmf1 both rebased to remove silly merge commits.
21:14 mutante: emery - revoking old puppet cert, resiging, initial puppet run
21:04 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/ArticleFeedback/ArticleFeedback.hooks.php 'Remove jquery.clickTracking from AFT dependencies'
21:01 logmsgbot: olivneh synchronized php-1.22wmf1/extensions/GettingStarted/SpecialGettingStarted.php
20:53 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'Update GettingStarted extension for E3 deployment'
20:34 logmsgbot: mflaschen synchronized wmf-config 'Config change for GettingStarted'
20:32 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:11 logmsgbot: kaldari Started syncing Wikimedia installation... :
19:58 mutante: emery - upgrading from lucid to precise
19:46 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'syncing InitialiseSettings to turn off VectorSectionEditLinks on testwiki'
19:41 Jeff_Green: running authdns-update to propagate frdata cname
19:31 AaronSchulz: Refreshed the job queue aggregator hash table
19:31 mutante: running puppet on brewster, testing lucid image on emery, rebooting again
19:17 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo back on for testwiki'
19:16 logmsgbot: bsitu synchronized wmf-config/CommonSettings.php 'Add checkDelay to Echo email bundle jobs'
19:07 AaronSchulz: All jobs finished copying into redis
19:07 sbernardin: taking down ms-be9 for hw replacement
19:02 LeslieCarr: uninstalling nagios from spence
18:51 mutante: PXE booting emery into installer
18:48 ottomata: emery going down for upgrade to precise
18:21 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Turn Echo off on testwiki'
17:35 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved remaining jobs to redis'
17:24 sbernardin: taking down db11 for decommissioning
17:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 17:19:14 UTC 2013
17:15 cmjohnson1: europium power disconnected during routine data center work
13:14 logmsgbot: reedy synchronized wmf-config/ 'Disable ClickTracking :D'
13:08 mark: Inserted python-extras packages into the precise-wikimedia APT repository (backport by hashar)
10:10 hashar: Jenkins is back up
09:23 hashar: restarting Jenkins to get the git verbose mode :(
08:44 hashar: jenkins: stop git cloning mw/core over and over by reusing existing clone 58502. That should help with I/O starvation on gallium.
08:40 Nikerabbit: translation memory database rebuild finished
08:17 Nikerabbit: emptying Translate solr instance and starting rebuilds
08:15 logmsgbot: nikerabbit synchronized php-1.22wmf1/extensions/Translate/ 'to master'
07:11 logmsgbot: hashar synchronized docroot/noc/conf/highlight.php 'Revert highlight.php recent modifications bug 47112 58664'
07:02 hashar: Put a live hack on fenari for bug 47112
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 11 02:16:35 UTC 2013
01:10 Ryan_Lane: disabling puppet on labstore3/4 - making sure gluster won't die without them and we're going to rebuild them

April 10

23:31 LeslieCarr: deactivated XO ipv6 transit
22:25 Ryan_Lane: added live hack to OpenStackManager on wikitech, ensuring that DynamicSidebar support is only enabled for logged-in users
22:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'moved refreshLinks to redis.'
22:20 ^demon|sick: restarting gerrit, was hung
22:15 Ryan_Lane: enable DynamicSidebar support in OpenStackManager
22:14 Ryan_Lane: temporarily disabling DynamicSidebar support in OpenStackManager
21:59 notpeter: gracefulling apache on stafford
21:36 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved htmlCacheUpdate jobs to redis'
21:19 ottomata: added python-statsd to apt repo for hashar
21:18 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php
21:07 ottomata: added python-voluptuous to apt repo (for hashar)
20:46 ottomata: added python-jsonschema to wikimedia apt repo: ori-l :)
20:00 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Moved async upload jobs to redis'
19:13 notpeter: running puppetd --enable on all nodes via salt
19:09 mwalker: eqiad frack is now using clustered memcache and has site local configuration for DB, ActiveMQ, MemCacheD shtuff
18:24 logmsgbot: aaron synchronized wmf-config/jobqueue-eqiad.php 'Enabled use of redis for null test jobs'
18:06 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Switched all remaning wikis to 1.22wmf1
17:45 RobH: authdns-update to rename db1012 to stat1002
17:29 logmsgbot: aaron synchronized php-1.22wmf1/maintenance/copyJobQueue.php 'deployed 54f74111901ddebdc69f26847e3904140b3723d5'
17:24 LeslieCarr: restart unresponsive cp1037
17:14 RobH: authdns-update
15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036
15:40 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Nostalgia ext'
15:40 logmsgbot: demon synchronized wmf-config/extension-list 'Nostalgia ext'
15:38 logmsgbot: demon synchronized php-1.22wmf1/extensions/Nostalgia
14:33 logmsgbot: maxsem synchronized wmf-config/mobile.php
14:19 logmsgbot: maxsem synchronized wmf-config 'Enable $wgMFVaryResources on test'
14:12 mark: Inserted varnish 3.0.3plus-rc1-wm10 packages into the APT repository
12:23 apergos: removed outright puppet.log on neon, was > 7gb and no room left on device
07:53 logmsgbot: krinkle synchronized php-1.22wmf1/resources/startup.js 'Ia54dd738b3ce0995fa'
05:57 Tim: on stafford: swapdeath repeat narrowly averted via killall ruby && /etc/init.d/apache2 stop
05:34 Tim: on brewster: proxy down due to full root partition, again. Will fix it properly this time.
05:28 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Add 'Zero' & 'Zero talk' namespaces to metawiki (If6b3ce5e4)'
04:53 Tim: stafford went into swapdeath. I killed all ruby processes a couple of times, and then eventually stopped apache2 for a while, while it recovered
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 10 02:28:06 UTC 2013
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 10 02:17:19 UTC 2013
00:21 mutante: removing xenon and db1012 from icinga configs, running puppetstoredconfigclean.rb on them, restarting icinga
00:07 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'
00:06 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/includes/MobileFrontend.hooks.php 'https://gerrit.wikimedia.org/r/#/c/58436/'

April 9

23:29 logmsgbot: tstarling synchronized php-1.22wmf1/extensions/Nostalgia
23:21 notpeter: running upgrades on db56 and rebooting
23:19 notpeter: running upgrades on db64 and rebooting
22:55 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend/
22:53 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend/
22:36 logmsgbot: reedy synchronized wmf-config/
22:34 LeslieCarr: clearing a few varnish caches - fyi if db load is seen to increase
22:17 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:45 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
21:45 hashar: Zuul is slowly dequeuing a large number of jobs. There is currently rougly 30 minutes lag between change submission and actually processing of the job. No idea why but investigating.
19:56 Ryan_Lane: installed Nuke on wikitech
18:11 K4-713: updated payments anti-fraud rules
13:07 hashar: jenkins plugin upgrade is tracked in bug 47040. Luckily git plugin has been fixed just in time ;-D Just need to migrate the config history.
12:37 hashar: jenkins plugins seems to have been upgraded overnight :(
10:26 mark: Deployed experimental varnish build (3.0.3plus~rc1-wm10) on dysprosium and cp1022
08:54 hashar: Jenkins tests for mediawiki/core are broken (bug 47031) due to a faulty change that has been merged in master branch. Root cause is the gating job not testing the proper change bug 46723 :/
08:43 Nikerabbit: Got rid of local changes and undeployed commit related to noc
08:41 logmsgbot: nikerabbit synchronized wmf-config/InitialiseSettings.php 'Narayam and WebFonts'
05:42 Tim: on mw1149: enabled apache access log
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 9 02:30:59 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 9 02:18:53 UTC 2013
00:26 ^demon: jenkins is back. java gc took awhile.
00:19 ^demon: jenkins is hung, looking into it.
00:16 awight: payments cluster updated from 22943dc28 to d3954768e4dd

April 8

23:16 logmsgbot: catrope synchronized php-1.22wmf1/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
23:15 logmsgbot: catrope synchronized php-1.21wmf12/extensions/WikiEditor 'Deploying 17a90bc74e87341199a50ebe5348576e05be028a'
22:55 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Changing from emery to vanadium'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/AutoLoader.php 'Backported Poolcounter convenience class'
22:13 logmsgbot: demon synchronized php-1.21wmf12/includes/PoolCounter.php 'Backported Poolcounter convenience class'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-pmtpa.php 'Enable poolcounter for search'
21:59 logmsgbot: demon synchronized wmf-config/PoolCounterSettings-eqiad.php 'Enable poolcounter for search'
21:42 logmsgbot: demon synchronized php-1.21wmf12/extensions/MWSearch/MWSearch_body.php 'Deploying Poolcounter updates'
21:17 notpeter: temp stopping slave on db55
20:53 logmsgbot: aaron synchronized wmf-config/filebackend.php 'Bumped file journal ttl to a year'
20:18 AaronSchulz: Moved ceph sync scripts to terbium and started a second pass
20:15 logmsgbot: aaron synchronized php-1.22wmf1/extensions/FlaggedRevs 'deployed c61688baef682b7d97068fefdcc399256786a387'
19:08 ^demon: gerrit back
19:07 ^demon: gerrit restarting, config change to fix idle timeouts.
19:00 ottomata: removing some unused filters from emery's udp2log instance
18:34 Jeff_Green: deployed 1.22 on payments
18:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.22wmf1
17:09 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at full weight'
17:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1058 to s1 at a low warmup rate'
11:57 ^demon: gerrit back, deployed 2.6-rc0-154-gfcdb34b which contains a temporary fix for the stream-events timeout issues. See bug 46917 for info.
11:53 ^demon: bringing gerrit down for urgent update
10:31 hashar: gerrit: created operations/debs/libvpx for j^ . Initialized pristine-tar and upstream branches using empty commits and force push.
07:48 hashar: restarting Zuul for demo purposes :-)
04:00 Tim: on brewster: root partition was full, removed squid access.log and store.log and started squid
03:55 Tim: on all apaches: upgrading libpoppler
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 8 02:25:07 UTC 2013
02:18 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 8 02:18:18 UTC 2013

April 7

22:46 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
22:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:31 logmsgbot: reedy synchronized wmf-config/ 'Kill readerfeedback config, it aint coming back'
22:26 logmsgbot: reedy synchronized wmf-config/
22:20 logmsgbot: reedy synchronized php-1.22wmf1/extensions/CentralAuth
11:35 ^demon: restarting gerrit, stream-events seized up again.
07:26 hashar: gallium: restarted Zuul just to be sure. Gerrit on manganese is stuck again and need to be restarted bug 46917
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Apr 7 02:17:09 UTC 2013
02:11 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sun Apr 7 02:11:21 UTC 2013

April 6

12:27 ^demon: also restarted zuul
12:19 ^demon: restarting gerrit again, stream-events halted like friday.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Apr 6 02:22:48 UTC 2013
02:12 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Sat Apr 6 02:11:56 UTC 2013
01:55 binasher: iwlinks iwl_prefix_from_title migration completed for all projects
00:52 K4-713: adjusted anti-fraud rules on payments cluster

April 5

23:11 binasher: running iwlinks index migration on s4 (commons)
21:27 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 to full weight, pulling db1023'
21:13 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1021 at a low weight'
20:05 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1021 for upgrade'
19:53 binasher: streaming backup of db1056 to db1050 (converting s1 snapshot slave to file_per_table)
18:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 's1: moving watchlist to db1049, pulling db1043, adding db1056'
18:50 Jeff_Green: increased jvm heap size on silicon (payments activemq)
18:42 notpeter: restarting gerrit
18:22 logmsgbot: reedy synchronized php-1.21wmf12 'Killing active users in 1.21wmf12'
09:39 hashar: Ariel kindly restarted gerrit on manganese which unbroken the stream-events process in Zuul. It again receives events.
09:33 hashar: restarted Zuul
09:23 Nemo_bis: varnish esams purging still broken, e.g. https://upload.wikimedia.org/wikipedia/commons/f/fe/Wikipedia-logo-v2-km.png
08:58 hashar: for Jenkins / Gerrit issue see bug 46917
08:48 hashar: Jenkins no more receiving ANY events from Gerrit. Effect: no jobs are triggered.
02:23 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Apr 5 02:23:11 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Fri Apr 5 02:12:34 UTC 2013
00:49 logmsgbot: csteipp synchronized php-1.21wmf12/includes/
00:46 logmsgbot: csteipp synchronized php-1.22wmf1/includes/
00:30 Tim: on fenari: testing a new version of mw-update-l10n which doesn't use stdout of mergeMessagesFileList.php
00:20 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:19 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo to master
00:07 Tim: all 1.22wmf1 wikis briefly showed error messages due to display_errors being enabled when mergeMessageFileList.php generated ExtensionMessages-1.22wmf1.php. Fixed temporarily.
00:04 logmsgbot: tstarling synchronized wmf-config/ExtensionMessages-1.22wmf1.php
00:00 logmsgbot: mflaschen synchronized wmf-config/extension-list 'Remove E3Experiments and LastModified'

April 4

23:51 logmsgbot: demon synchronized wmf-config/extension-list 'Remove e3 since it was deleted'
23:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php
23:46 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:46 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo to master
23:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
23:25 binasher: shutting down mysql on db1043, preparing to reboot
23:24 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043'
23:17 mutante: wikinews.com now redirects to wikinews.org
23:11 mutante: DNS update - adding wikinews.com
23:08 logmsgbot: dzahn gracefulled all apaches
22:52 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Re-enable Echo on testwiki'
22:49 mutante: running updateSpecialPages on cswikinews per request
22:14 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php 'Disable Echo on testwiki'
22:10 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Bump Echo EventLogging schema for new notification types.'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GuidedTour/ 'GuidedTour 1.22wmf1 deployment for E3'
22:02 logmsgbot: mflaschen synchronized php-1.22wmf1/extensions/GettingStarted/ 'GettingStarted 1.22wmf1 deployment for E3'
22:01 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/ 'GuidedTour 1.21wmf12 deployment for E3'
22:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GettingStarted/ 'GettingStarted deployment for E3'
20:21 LeslieCarr: stopped cp3001 varnish
20:18 LeslieCarr: restarting varnish on cp3001
20:15 LeslieCarr: restarted udpmcast on hooft
20:13 mutante: puppetstoredconfigclean.rb xenon.eqiad.wmnet - Killing xenon ...
19:34 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:29 logmsgbot: maxsem synchronized php-1.22wmf1/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/57525/'
19:20 binasher: enabled 1:100 full query logging on db1042/1043, disabled puppet [planning to revert in 4-8 hours]
19:09 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/57528'
19:01 binasher: resuming iwlinks migrations on s3
17:31 logmsgbot: mlitn synchronized php-1.22wmf1/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:31 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
17:27 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Updating ArticleFeedbackv5 frwiki config'
16:16 paravoid: upgrading image & video scalers for libav
15:55 ottomata: now generating webstatscollector dumps on gadolinium. snapshot1 syncs dumps from there over to dataset2 for dumps.wikimedia.org
15:38 K4-713: adjusted anti-fraud rules on the payments cluster
13:26 ^demon: running jgit gc on all repos
05:43 Ryan_Lane: clearing squid logs and restarting squid on brewster
04:16 Ryan_Lane: opendj upgrade on virt1000 complete
03:56 Ryan_Lane: upgrading opendj on virt1000
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Apr 4 02:21:01 UTC 2013
02:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Thu Apr 4 02:13:31 UTC 2013

April 3

23:19 mwalker: Updating CentralNotice schema on testwiki & metawiki with patch-centralnotice-2_3.sql
22:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
21:45 binasher: started iwlinks index migrations on all wikis [gerrit 43389]
21:32 cmjohnson1: Running DNS update
20:34 ottomata: udpated webstatscollector package in apt repo to 0.1-3
18:54 mutante: DNS update - also adding softwarewikipedia.org
18:49 RoanKattouw: Removed caesium and xenon from /home/wikipedia/common/docroot/noc/pybal/eqiad/parsoid
18:43 mutante: restarting pdns on ns1
18:42 mutante: DNS update - adding softwarewikipedia.com for RT-4355
18:41 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiquote and wikibooks to 1.22wmf1
18:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary to 1.22wmf1
18:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews and wikisource to 1.22wmf1
18:31 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity and wikivoyage to 1.22wmf1
18:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special, wikimania and wikimedia wikis to 1.22wmf1
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: private, closed and fishbowl to 1.22wmf1
17:14 logmsgbot: aaron synchronized php-1.22wmf1/includes/objectcache/SqlBagOStuff.php 'deployed b61053ca4b554f6bd18fb6408967839cdb5ccde2'
17:12 logmsgbot: aaron synchronized php-1.21wmf12/includes/objectcache/SqlBagOStuff.php 'deployed 61587acc64cb62400ff7978271c54e8bd8b1f02d'
16:54 logmsgbot: reedy synchronized php-1.21wmf12/includes/Collation.php
16:34 cmjohnson1: authdns update
14:22 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'setting weight on db1028 to 400'
13:15 mark: Stopped and started MySQL on pc1003 after finding mysql deadlocked
13:10 logmsgbot: reedy synchronized wmf-config/
12:20 mark: Rebooting cp3003 for RAID reconfiguration
10:43 mark: Deactivated AS6908 peering on cr2-knams
06:09 ^demon: gerrit: finished doing jgit gc on mediawiki/core. Repo size 3G -> 323M. Fresh clone time to <5m to localhost, <1.5m to other hosts inside wmf. I rock. Backup's in /home/demon/core.git in case something goes wrong. Bed time.
03:05 LeslieCarr: the final upload varnish is on 2gig instead of 1gig ! win!
02:32 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Wed Apr 3 02:31:44 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Apr 3 02:19:30 UTC 2013
01:05 LeslieCarr: applying a big change to varnish interface groups - risk higher than normal
00:05 binasher: removed sda3/varnish.persist on cp1021, restarted varnish
00:00 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/PostEdit

April 2

23:46 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php '(Ia2665c4fe) Enable PostEdit on bn, br, ca, cs, et, ka and zh wikis'
23:46 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php '(Ibc2633f1c) : 10000 => 5000.'
23:25 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad'
23:11 logmsgbot: anomie synchronized php-1.22wmf1/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:10 logmsgbot: anomie synchronized php-1.21wmf12/includes/api/ApiQueryRecentChanges.php 'Fix for API list=recentchanges rccontinue'
23:01 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/#/c/57212/'
22:49 logmsgbot: maxsem synchronized php-1.21wmf12/resources/Resources.php 'https://gerrit.wikimedia.org/r/57208'
22:39 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56696/'
22:34 awight: payments cluster updated from 12bc36c238 to 0ab980b8b
22:24 logmsgbot: maxsem synchronized php-1.21wmf12/includes/resourceloader/ResourceLoaderStartUpModule.php
22:20 LeslieCarr: authdns-update for cp1039 and cp1040 move
22:14 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:11 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend
22:08 logmsgbot: maxsem Finished syncing Wikimedia installation... :
21:42 logmsgbot: maxsem Started syncing Wikimedia installation... :
21:38 logmsgbot: reedy synchronized wmf-config/
21:25 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1028'
20:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:41 logmsgbot: maxsem synchronized php-1.21wmf12/includes/OutputPage.php 'https://gerrit.wikimedia.org/r/#/c/49071/'
20:34 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/57078/'
20:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'moving s1 watchlist to db1052, db1043 to full weight'
20:09 Ryan_Lane: added marc to the ops ldap group
20:01 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001'
19:49 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
19:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:56 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1028 back'
18:54 notpeter: actually starting innopack from db65 to db1054 for pre-labsdb db (sanitarium)
18:48 notpeter: starting innopack from db31 to db1054 for pre-labsdb db (sanitarium)
18:39 binasher: completed image img_media_mime migration on all projects
18:35 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:42 LeslieCarr: restarted varnishncsa-locke on cp1023
17:41 LeslieCarr: restarted varnishncsa-multicast_relay on cp1028
17:34 cmjohnson1: correction db1028
17:34 cmjohnson1: removing disk 0 from db1048 to replace
17:28 Jeff_Green: disable fundraising banner log rotation on locke
17:21 logmsgbot: cmjohnson synchronized wmf-config/db-eqiad.php 'adding db1001 back to production removing db1028'
17:04 cmjohnson1: setting weight to 100 on db1001
15:42 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Message for ancient skin users'
15:41 logmsgbot: anomie synchronized php-1.21wmf12/cache/l10n/l10n_cache-en.cdb
15:34 logmsgbot: anomie synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php
15:13 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 15:12:45 UTC 2013
15:09 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 15:08:37 UTC 2013
14:54 logmsgbot: demon synchronized php-1.21wmf12/extensions/WikimediaMessages/WikimediaTemporaryMessages.i18n.php '8th time is the charm'
14:51 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'I hate l10nupdate'
14:50 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'Notice for users of disabled skins'
14:24 paravoid: deploying squid config, diverting all of upload to swift
04:09 binasher: increased 'wikiadmin'@'10.64.%' max connections to 200 on s1
02:17 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Tue Apr 2 02:17:33 UTC 2013
02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Apr 2 02:10:24 UTC 2013
01:15 Ryan_Lane: removing labstore1 and labstore2 entries from projectstorage.wmnet rr dns entry in preparation for shrinking volumes
00:44 logmsgbot: reedy synchronized wmf-config/
00:43 binasher: now running the image img_media_mime migration on commons (the big one)
00:41 mutante: restarting lucene on all pool4 servers (one by one)
00:36 mutante: creating search index for transitionteamwiki
00:30 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:14 logmsgbot: reedy synchronized php-1.22wmf1/cache/interwiki.cdb 'Updating 1.22wmf1 interwiki cache'
00:10 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
00:01 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:

April 1

23:55 LeslieCarr: search1024 is our guinea pig of using sfp-t's and is connected to asw2-a5-eqiad ge 0/0/18
23:48 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:40 mutante: creating swift containers for private wiki
23:39 logmsgbot: dzahn synchronized php/cache/interwiki.cdb 'Updating interwiki cache'
23:37 mutante: dumpInterwiki.php to update interwiki cache
23:36 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic to pmtpa temporarily'
23:33 mutante: adding transitionteamwiki to private.dblist
23:25 mutante: running addWiki.php for transitionteamwiki - Access denied for user ...
23:21 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
23:14 cmjohnson1: swapping disk1 (slot1) db1001
23:06 mutante: creating upload dir on upload7 for transitionteamwiki
22:57 binasher: beginning image.img_media_mime scehma migrations, starting with s1
22:52 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed ec8f42daf19f386d9af2eece6dfc5f8fa6fc42f3'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/maintenance 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 logmsgbot: aaron synchronized php-1.22wmf1/includes/job 'deployed c7832c8956d79892be93452e5f4d6b52df1d3bf0'
22:26 mutante: recreating transitionteam wiki docroot from skel-1.5 and syncing docroot
22:26 logmsgbot: root synchronized docroot
21:09 LeslieCarr: rebooting lvs1001
21:04 logmsgbot: demon synchronized php-1.22wmf1/cache/l10n 'Manually syncing 1.22wmf1 l10ncache'
20:56 LeslieCarr: dist-upgrading lvs1001
20:48 LeslieCarr: failing over lvs from lvs1001 to lvs1004
20:44 LeslieCarr: CORRECTION restarting sshd on lvs1001
20:44 LeslieCarr: restarting lvs1001
20:21 logmsgbot: demon synchronized php-1.21wmf12/cache/l10n 'Manually syncing 1.21wmf12 l10ncache'
20:20 logmsgbot: LocalisationUpdate completed (1.22wmf1) at Mon Apr 1 20:20:29 UTC 2013
20:00 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:59:44 UTC 2013
19:54 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 19:54:21 UTC 2013
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki
19:13 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: mediawikiwiki and testwiki to 1.22wmf1
19:11 logmsgbot: reedy synchronized php-1.22wmf1/extensions/ 'Wikidata extensions to same as 1.21wmf12'
19:04 logmsgbot: reedy synchronized php-1.22wmf1/includes/
18:56 logmsgbot: reedy Finished syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:49 LeslieCarr: testing logbot
18:20 Reedy: Ran scap-recompile to build texvc
18:20 jeremyb_: re!logged from earlier today while morebots was gone (all times UTC)
18:20 jeremyb_: 17:56:29 <logmsgbot> !log reedy synchronized w
18:20 jeremyb_: 17:55:45 <logmsgbot> !log reedy synchronized docroot
18:20 jeremyb_: 17:55:04 <logmsgbot> !log reedy synchronized php-1.22wmf1 'Initial sync of php-1.22wmf1'
18:19 jeremyb_: 17:27:13 <logmsgbot> !log reedy synchronized wmf-config/ExtensionMessages-1.22wmf1.php
18:19 logmsgbot: reedy Started syncing Wikimedia installation... : test2wiki to 1.22wmf1 and rebuild l10n cache
18:19 jeremyb_: 17:14:31 <Jeff_Green> !log moved aside williams:/opt/otrs-home/.spamassassin/auto-whitelist and restarted spamd to test whether the bloated AWL is resulting in poor filtering
18:19 jeremyb_: 14:39:19 <andrewbogott> !log restarted glusterfs-server on labstore1 and labstore2, because of read errors in public/datasets/public in bots project
18:19 jeremyb_: 13:55:28 <ottomata> !log added webstatscollector-0.1-2 to apt repo
18:18 jeremyb_: 11:54:15 <paravoid> !log repooling ms-fe1; staggered depool/restart/pool of ms-fe2-4
18:17 Ryan_Lane: restarted morebots. wikitech-static had 100% / partition. Have also fixed that and its cause.
18:16 jeremyb_: 07:45:24 <paravoid> !log depooling ms-fe1 for staging
18:16 jeremyb_: 03:58:05 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'returning db1051'
18:16 jeremyb_: 03:44:43 <logmsgbot> !log asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
18:16 jeremyb_: 03:22:54 <binasher> !log reinstated max_user_connections = 80 for wikiadmin on s1
18:15 cmjohnson1: restarted morebots
03:23 binasher: reinstated max_user_connections = 80 for wikiadmin on s1
02:12 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Mon Apr 1 02:12:24 UTC 2013

March 31

02:10 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 31 02:10:18 UTC 2013

March 30

02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 30 02:13:49 UTC 2013
00:01 LeslieCarr: rebuilding mailman archives …. very slowly

March 29

23:51 LeslieCarr: stopping mailman to remove sensitive message
22:38 LeslieCarr: upgrading packages on aluminium
22:25 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
21:02 hashar: jenkins: raised the number of execution for deployment-bastion from 2 to 4. Will let us execute more jobs in parallel
20:14 mutante: scheduling a 2-year downtime for db11 :p
19:33 logmsgbot: dzahn gracefulled all apaches
19:30 logmsgbot: dzahn synchronized ./docroot/transitionteam
19:29 mutante: creating transitionteam docroot
19:25 logmsgbot: aaron synchronized php-1.21wmf12/includes 'deployed fa5c0a6a82c9a1cb63f3cedd87e6c4aba59c994d'
18:49 sbernardin: hot-swapping disk on db29
18:04 ^demon: gerrit back up, now running 2.6-rc0-76-g52fb5ae
18:00 mutante: DNS update - adding transitionteam wiki
17:47 sbernardin: taking down db29 to replace bad hw
17:45 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Re-enabled async upload for commons'
17:23 ^demon: gerrit offline for an update, brb
09:25 logmsgbot: olivneh synchronized wmf-config/throttle.php 'Updating throttle rules for guwiki workshop (Bug 46686)'
04:16 logmsgbot: aaron synchronized php-1.21wmf12/maintenance/runJobs.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5'
04:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/job/JobQueueGroup.php 'deployed 651884d4bdc76c29dd717dfdbbd698632223f3b5 '
02:17 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 29 02:17:06 UTC 2013
01:24 ori-l: Stopping EventLogging daemons to allow Puppet to change 'eventlogging' user's home dir
00:11 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/MoodBar 'Updating MoodBar to remove ClickTracking integration'
00:03 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'
00:00 logmsgbot: mflaschen synchronized php-1.21wmf12/extensions/GuidedTour/GuidedTour.php 'Small bug fix to GuidedTour; removing unneeded dependency. https://gerrit.wikimedia.org/r/#/c/56546/1/GuidedTour.php'

March 28

23:24 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Remove E3Experiments and LastModified (2/2)'
23:24 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove E3Experiments and LastModified (1/2)'
22:35 logmsgbot: khorn synchronized php-1.21wmf12/includes/cache/MessageCache.php 'Reverting changes in language fallback behavior'
22:34 logmsgbot: khorn synchronized php-1.21wmf12/languages/Language.php 'Reverting changes in language fallback behavior'
22:20 logmsgbot: spage Finished syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
22:12 LeslieCarr: removed some log files from ms1004 - it was full
22:10 RobH: bugzilla admin stuff: noc@w.o is now a disabled account cuz im tired of folks who arent on the alias attempting to sign up with it
22:02 logmsgbot: spage Started syncing Wikimedia installation... : E3: Updated GettingStarted and GuidedTour extensions
21:46 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Fix mobileMode property name (was: 'mobile')'
21:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming
20:11 Jeff_Green: add hostname samarium.wikimedia.org and pdns-update
19:51 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1051'
19:44 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1051'
19:44 LeslieCarr: killed gmetad on neon
19:36 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/ArticleFeedback/populateAFRevisions.php 'Fix fatal error triggered by populateAFRevisions.php cronjob when AFT is not loaded.'
19:33 mutante: starting gmetad on neon again
19:24 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo on wmf12'
19:17 mutante: manually deleting db11 references from icinga configs and restarting it
19:12 mutante: pkill -u snmptt on neon, restarting snmptt deamon
19:10 binasher: replaced collector on professor with locally built version from I2bc117d5620b9e545a8e5a6cf3b654ee835b75d7 (testing fix for issues only under very high load)
19:10 mutante: stopping gmetad and ganglia-parser on neon
18:53 mutante: adding gjg (greg-g) to wmf ldap group
18:23 mutante: restarting lucene on pool4 search nodes to pickup new wikivoyage indices
18:12 ottomata: adding reportcard and analytics .wikimedia.org CNAMEs to stat1001
18:08 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:56 mutante: creating search indices for new wikivoyages
17:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
11:49 hashar: search19 and search20 issue with enwiki.prefix is in RT #4845
11:42 hashar: search19 and search20 have enwiki.prefix marked as FAILED. (see: curl --silent http://search19.pmtpa.wmnet:8123/status |grep FAILED and curl --silent http://search20.pmtpa.wmnet:8123/status |grep FAILED).
10:39 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate/ 'Translate to master'
04:19 paravoid: updating jsduck in apt and upgrading it on gallium
03:04 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Thu Mar 28 02:19:24 UTC 2013
00:47 K4-713: synchronized payments cluster to 12bc36c2385d8e56792c
00:27 mutante: once again disabling notifications for db11
00:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

March 27

23:59 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
23:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:40 Ryan_Lane: finished updating wikitech-static
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:38 Ryan_Lane: updating wikitech-static to wmf/1.21wmf12
23:24 Ryan_Lane: finished upgrading wikitech
23:23 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:23 Ryan_Lane: upgrading wikitech to wmf/1.21wmf12
23:17 logmsgbot: reedy synchronized php-1.21wmf12/cache/interwiki.cdb 'Updating 1.21wmf12 interwiki cache'
23:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:02 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:50 logmsgbot: maxsem synchronized php-1.21wmf11/extensions/MobileFrontend 'touch'
22:47 logmsgbot: maxsem synchronized php-1.21wmf12/extensions/MobileFrontend 'touch'
22:24 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:22 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php
22:05 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:48 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment, 2nd attempt
21:12 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 slave weights for testing'
20:10 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:56 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild localisation cache to fix he related ParserFunctions problems
19:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/ParserFunctions/
19:38 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
19:05 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for WikiData deployment
18:51 cmjohnson1: pushing mw1209-1220 into service
18:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf12
18:23 mutante: installing package upgrades on neon (Icinga), incl. ganglia-monitor,gmeta
18:09 mutante: bunch of mw* hosts: fix puppet runs
18:08 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo 'syncing Echo for wmf12'
18:05 mutante: mw1130 - kill and fix puppet runs
17:41 Jeff_Green: authdns-update, pdns on ns1 croaked, restarted
15:05 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: be_x_oldwiki to 1.21wmf12
14:09 hashar: jenkins has been restarted by mistake :-( Job building will be unavailable for up to half an hour.
13:58 hashar: Jenkins: disabled the old Gerrit Trigger Plugin bug 46415
09:56 Nikerabbit: Testing Translate bug fixes on test.wikipedia.org
05:20 Tim: on fluorine: running mw-log-cleanup once to test Id2196e7b
05:02 Ryan_Lane: disabling Bayes-collector cron on sanger
04:55 Ryan_Lane: purging BayesLearning files older than 70 days on mchenry
02:45 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 27 02:44:56 UTC 2013
02:19 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 27 02:19:11 UTC 2013
00:55 LeslieCarr: ms-be3 was crashed with stack traces for swift-container
00:54 LeslieCarr: rebooting ms-be3
00:38 awight: payments cluster updated from 4862fb8eef to f295d851c
00:34 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1052 to full weight'
00:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1001, adding db1052 at a warmup weight'
00:11 awight: payments cluster updated from fa2e2 to a06a03f05 wheee!

March 26

23:54 awight: payments cluster rolled back from e758102 to fa2e240b6e9
23:53 awight: payments cluster updated from fa2e240b6e9 to e758102
23:53 Tim: on all apaches: replacing /usr/local/apache/common-local/live-1.5 with a symlink to w
23:44 logmsgbot: tstarling synchronized w
23:35 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/56019/'
23:33 awight: payment cluster rollback, from 4862fb8 to fa2e240b
23:31 mutante: restarting nrpe on professor
23:29 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Workaround for buggy NavTiming implementation in IE9 (bug 46474)'
23:20 awight: payments cluster updated from fa2e240b to 4862fb8
23:10 mutante: db11 - disabling all notifications
23:07 mutante: mw13 - killing puppet and re-running
23:05 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:00 mutante: removing mysql packages from db1036 (broken dpkg, status "pi")
22:55 mutante: re-installing ganglia-monitor on mw1135, fix dpkg
22:54 logmsgbot: reedy synchronized docroot
22:52 mutante: starting udpprofile (carbon-cache.py) on professor
22:51 logmsgbot: reedy synchronized wmf-config/
22:49 mutante: search21 - remove sun-j2sdk , replaced by oracle-jdsdk
22:47 mutante: search21 - broken DPKG: iF sun-j2sdk1.6
22:45 mutante: mw2 - broken DPKG: pi librsvg2-2 , pi librsvg2-bin
22:39 LeslieCarr: did a service apache2 start on mw108*
22:28 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:10 notpeter: icinga mostly trustworthy again :)
22:08 andrewbogott: restarted nova-api on virt0 because it appeared to have crashed.
21:31 andrewbogott: restarted memcached on virt0 in a futile attempt to address https://bugzilla.wikimedia.org/show_bug.cgi?id=46583
21:30 mutante: killed stuck snmptt procs on neon to fix the others.. apparently :p
21:28 mutante: stopping gmetad on neon
20:52 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'syncing CommonSettings.php for Echo event tracking'
20:41 mutante: more degraded RAIDs: db1001->RT-4832, db1028->4834
20:35 mutante: broken disks/degraded RAID on db servers: db11->RT-4828, db29->RT-4829, db45->RT-4831
20:34 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/ContributionReporting 'syncing ContributionReporting extension'
20:07 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'raising db1051 to full weight'
19:49 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:46 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice 'Fixing CentralNotice bug in which it was possible to initialize multiple times'
19:40 logmsgbot: kaldari synchronized php-1.21wmf12/resources/startup.js 'syncing php-1.21wmf12/resources/startup.js'
19:33 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase/client
19:27 cmjohnson1: mw1209 re-imaging
19:20 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 're-syncing Echo icons dir'
19:19 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Include AFTv5 on frwiki'
19:12 logmsgbot: mlitn synchronized wmf-config 'Enable AFTv5 on frwiki'
19:05 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Collection/
19:04 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:04 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:02 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:58 mutante: starting gmetad on neon - we failed to resolve data source name ms-fe1001.eqiad.wmnet
18:56 mutante: restarting ircecho on fenari
18:48 mutante: stopping gmetad on neon
18:44 mutante: killing all snmp related processes on neon and restarting
18:43 logmsgbot: reedy synchronized php-1.21wmf12/extensions/MobileFrontend/includes/specials/SpecialMobileWatchlist.php
18:41 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/Echo/modules/icons 'syncing Echo icons dir'
18:39 mutante: restarting snmtrapd on neon
18:25 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Turning Thanks extension on on MediaWiki wiki'
18:13 logmsgbot: krinkle synchronized php-1.21wmf12/resources/mediawiki/mediawiki.js 'I80af730daa815 fixing bug 46575'
18:12 Krinkle: Change I34a488247 (mw.loader debug) has been reverted in wmf/1.21wmf12 and pulled on fenari
17:51 cmjohnson1: removing disk from slot 0 ms-be1012
17:14 Krinkle: Change I34a488247 is now merged in wmf/1.21wmf12 and pulled on fenari for test.wikipedia.org, do not sync.
17:04 ottomata: added (crappy, but better than nothing) .deb for webstatscollector to apt.wikimedia.org
15:24 cmjohnson1: swapping disk 6 ms-be1012
14:34 logmsgbot: hashar synchronized multiversion '47564 wikiversions.cdb now vary by realm'
14:10 cmjohnson1: running sync-common on mw1209
10:23 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector/UniversalLanguageSelector.php 'Testing fix for wikidata'
09:28 pp-pdf1: updated mwlib to 0.15.4
08:47 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/Translate
08:46 logmsgbot: nikerabbit synchronized php-1.21wmf12/extensions/UniversalLanguageSelector
07:27 mark: Restarted swift-proxy on ms-fe*
04:16 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:46 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 26 02:45:52 UTC 2013
02:21 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 26 02:21:08 UTC 2013
01:41 mutante: analytics1026 - delete 16G udp2log.log to free disk
01:38 mutante: analytics1003 - gzipping jmxtrans logs to free some disk
01:36 mutante: analytics1003 - gzipping kafka.log to free some disk
00:39 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'adding db1051 at a low warmup weight'

March 25

23:36 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'done'
23:31 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'changing s1 weights, returning db1050'
23:27 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Quick profiling'
22:10 hashar: reloading Zuul
21:44 logmsgbot: khorn synchronized wmf-config/InitialiseSettings.php 'Re-enabling CentralNotice'
21:39 mutante: killing and restarting nagios-nrpe-server on '*'
21:32 logmsgbot: khorn synchronized php-1.21wmf12/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:28 logmsgbot: khorn synchronized php-1.21wmf11/extensions/CentralNotice/ 'CentralNotice jQuery fix'
21:12 mutante: killing and restarting nagios-nrpe-server on gadolinium
21:07 mutante: --last-lines 20 | RT ..would be nice sometimes:)
20:13 logmsgbot: aaron synchronized wmf-config/CommonSettings.php
20:07 logmsgbot: aaron synchronized wmf-config/CommonSettings.php 'Use ffmpeg for ogg thumbs as is done for webm'
19:55 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikibase
19:33 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikibase
18:57 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:46 logmsgbot: reedy synchronized php-1.21wmf12/extensions/Wikibase
18:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
18:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable CN everywhere'
18:14 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: rb
18:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf12
17:25 logmsgbot: reedy synchronized wmf-config/

March 24

18:13 Ryan_Lane: restarting glusterd processes on labstore*
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 24 02:31:22 UTC 2013
02:08 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sun Mar 24 02:08:00 UTC 2013

March 23

02:37 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 23 02:36:53 UTC 2013
02:14 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Sat Mar 23 02:13:44 UTC 2013
00:17 RoanKattouw: Deployed master of Parsoid onto wtp1004; the code is broken but the box is depooled for Gabriel's benchmarking work
00:15 RoanKattouw: That's me messing with wtp1004

March 22

23:33 ottomata: restarting hadoop setting yarn.nodemanager.resource.memory-mb to 16G
23:06 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 logmsgbot: dzahn rebuilt wikiversions.cdb and synchronized wikiversions files:
23:04 mutante: sync-wikiversions
22:29 logmsgbot: aaron synchronized php-1.21wmf12/extensions/Translate 'deployed 0aab5984415a94bd214088767ff1de33ab57c4e0'
22:12 mutante: killall nrpe via salt, then restart nagios-nrpe-server , runs as wrong user
22:09 mutante: kill and restart nrpe on oxygen
22:07 mutante: installing package upgrades on oxygen
21:40 mutante: killing nagios-nrpe on gallium and testing if puppet restores it properly
21:03 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Set all non-Wikipedias back to 1.21wmf12 again.
20:51 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed 0f4914bb4ff2d13cb124cad94dc5ded663123a18'
20:46 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed e372b635ae6d6a589f6d87f14fe4c9452d7a1b4d'
20:38 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'Reverted follow-up fixes too'
19:32 ottomata: removed unicast udp2logging to gadolinium; gadolinium will consume from multicast stream
18:44 K4-713: synchronized payments cluster to fa2e240b6e92440b
18:28 K4-713: synchronized payments cluster to df3c8dbc937c971
17:33 logmsgbot: aaron synchronized php-1.21wmf12/includes/WikiPage.php 'deployed 1862a57aef08432e8b1194a54dc179649f5cae63'
09:08 MaxSem: Recreating Solr index
02:52 Tim: on searchidx1001: started incremental indexer, apparently it died on March 21 at 02:08 when it ran at the same time as a cron job import
02:50 binasher: power cycling db1050, unresponsive on serial console
02:49 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1050, crashed'
02:47 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043'
02:42 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Fri Mar 22 02:42:32 UTC 2013
02:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, oops'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 22 02:30:31 UTC 2013
01:28 ori-l: Finished correcting E3Experiments extension's entry point from 'Experiments.php' (which does not match the submodule name) to 'E3Experiments.php'. Updated extension-list, CommonSettings.php, and the extension itself in both wmf11 and 12.
01:24 logmsgbot: olivneh synchronized wmf-config/extension-list 'Remove temporary workaround for E3 entry-point migration (2/2)'
01:23 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Remove temporary workaround for E3 entry-point migration (1/2)'
01:18 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:18 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/E3Experiments 'Updating E3Experiment extensions's entry point'
01:11 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Temporarily accepting either E3Experiments.php or Experiments.php as extension entry point to facilitate clean-up'
00:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'NavigationTiming: log mobile mode'
00:16 logmsgbot: catrope synchronized wmf-config/ExtensionMessages-1.21wmf12.php 'Remove PHP errors'
00:15 logmsgbot: aaron synchronized php-1.21wmf12/includes/Title.php 'deployed cef327e945e2593fe0291880a7d1976cc5a2f248'
00:09 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/Thanks 'syncing Thanks ext'

March 21

23:24 logmsgbot: robh Finished syncing Wikimedia installation... :
23:17 logmsgbot: robh Started syncing Wikimedia installation... :
23:04 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
22:44 logmsgbot: bsitu Started syncing Wikimedia installation... : Update Echo, PageTriage to master, Add Thanks
21:50 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'Syncing change to CommonSettings for Thanks ext'
21:49 logmsgbot: kaldari synchronized wmf-config/InitialiseSettings.php 'Syncing change to InitialiseSettings for Thanks ext'
21:46 mutante: running puppet and reloading zuul on gallium. enables hashar's new shiny https://integration.wikimedia.org/zuul/
21:38 RobH: authdns-update for terbium
21:05 RoanKattouw: Added fenari back to /etc/dsh/groups/mediawiki-installation per https://rt.wikimedia.org/Ticket/Display.html?id=4794
21:02 logmsgbot: cmjohnson Finished syncing Wikimedia installation... :
20:58 binasher: pulling mw1070, may have network issues
20:57 logmsgbot: kaldari synchronized php-1.21wmf12/extensions/PageTriage/modules/ext.pageTriage.views.list 'syncing PageTriage js on wmf12'
20:57 hashar: fenari disappeared from dsh group mediawiki-installation :-(
20:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:51 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:48 awight: payments cluster updated from 18fd11a4 to df3c8dbc9
20:43 logmsgbot: cmjohnson Started syncing Wikimedia installation... :
20:37 LeslieCarr: restarting gmond on mw1153
20:19 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf12/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
20:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
20:19 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Enable NavigationTiming on mobile alpha and beta'
19:53 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GuidedTour/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/GettingStarted/ 'E3 deploy'
19:52 logmsgbot: mflaschen synchronized php-1.21wmf11/extensions/E3Experiments/ 'E3 deploy'
19:49 logmsgbot: mflaschen synchronized wmf-config/CommonSettings.php 'E3 deployment'
17:39 logmsgbot: csteipp rebuilt wikiversions.cdb and synchronized wikiversions files: Update test2wiki to 1.21wmf12
17:32 logmsgbot: csteipp synchronized php-1.21wmf12/includes/Title.php 'update for loadPageData()'
17:24 notpeter: running puppet on all lucid nodes via salt
17:23 mutante-away: purging wikispecies.net from squid
17:08 notpeter: install timeout package on all lucid nodes to get puppet back on track
12:49 mark: Rebooting palladium with hyperthreading disabled
11:27 mark: Rebooting niobium with hyperthreading disabled
09:29 hashar: created git repo operations/debs/python-statsd
09:15 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
09:14 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Add 'wikidata.org' to $wgCrossSiteAJAXdomains to allow CORS'
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:34 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 Tim: reverting all wikis to 1.21wmf11 due to move page being totally broken (bug 46397)
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
06:32 logmsgbot: tstarling rebuilt wikiversions.cdb and synchronized wikiversions files:
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf12/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'
01:10 logmsgbot: catrope synchronized php-1.21wmf11/includes/db/LoadBalancer.php '319ace81250b97dff0e6ee6df4b1f570f468bf5c'

March 20

23:57 RobH: yay peter!
23:56 notpeter: freeing up roughly 500G of ram across our cluster by calling puppet via cron :)
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:52 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es100[69]'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:51 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s2 servers'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling es100[69]'
23:36 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:35 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning s1 servers'
23:31 RobH: authdns-update for lanthanum
22:45 RobH: ignore mw1209+ they arent deployed
22:04 LeslieCarr: restarting ircecho on fenari to ensure logmsgbot is on operations and tech irc channels
22:03 ottomata: stopping puppet on gadolinium to test for packet loss weirdness
22:01 pgehres: pgehres synchronized php-1.21wmf12/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
22:01 logmsgbot: pgehres synchronized php-1.21wmf11/includes/GlobalFunctions.php 'Fix wfWaitForSlaves() so the parameter actually works correctly'
21:35 LeslieCarr: restarted ircecho on fenari
21:01 Krinkle: asher synchronized wmf-config/db-eqiad.php 'pulling db1049 from s1'
20:53 pgehres: pgehres synchronized php-1.21wmf11/extensions/CentralAuth 'Updating CentralAuth to master'
20:53 pgehres: pgehres synchronized php-1.21wmf12/extensions/CentralAuth 'Updating CentralAuth to master'
20:42 Reedy: reedy synchronized wmf-config/InitialiseSettings.php
19:59 mutante: DNS update - remove zuul CNAME
19:53 Reedy: wikibooks to 1.21wmf12
19:51 Reedy: sync'd wmf-config/InitialiseSettings.php
19:50 Reedy: wikiversity and wikiquote to 1.21wmf12
19:49 Reedy: wikivoyage and wiktionary to 1.21wmf12
19:47 Reedy: wikinews and wikisource to 1.21wmf12
19:47 Reedy: Synchroni[sz]ed php-1.21wmf12/extensions for Wikidata deploy
19:46 Reedy: special and wikimedia dblists to 1.21wmf12
19:46 Reedy: wikimania, private and fishbowl dblists to 1.21wmf12
19:45 Reedy: closed.dblist to 1.21wmf12
19:38 RobH: sync-common on mw23
19:36 mutante: restarting hung apache on mw1202
19:28 LeslieCarr: sync-common on mw1085
19:24 mutante: .tar.gz'ing /var/mwdocs on formey (SVN) and then manually delete it
19:18 LeslieCarr: upgrading mw118* and then restarting apache2 on these boxes
19:11 LeslieCarr: installing package upgrades on mw1170-1179
19:10 mutante: installing package upgrades on formey
18:50 mutante: apache-graceful-all, remove boards.wm redirect
16:03 ottomata: added python-voluptuous package to apt repo
15:12 mark: Fixed purging for all sites but eqiad (which wasn't broken)
07:56 hashar: gallium: reloaded apache2 to make sure recent changes are applied.
02:44 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Wed Mar 20 02:44:03 UTC 2013
02:32 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Wed Mar 20 02:32:16 UTC 2013

March 19

23:52 K4-713: synchronized payments cluster to fe4fb96bea1378e
23:33 mutante: DNS update - add wikiartpedia.org
23:30 logmsgbot: dzahn gracefulled all apaches
23:08 logmsgbot: mlitn synchronized wmf-config/InitialiseSettings.php 'Re-enable AFTv5 on dewiki/testwiki'
22:38 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:36 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:33 ottomata: adding udp-filter_0.3.22 for precise to apt repo (lucid version already was there)
22:17 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
22:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/data/DataModel.php 'Update ArticleFeedbackv5; filter counts are incorrect'
21:52 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:52 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
21:37 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
21:25 mutante: installing package upgrades on kaulen
21:24 mutante: installing package upgrades on sockpuppet
21:13 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:36 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
20:35 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/SpecialArticleFeedbackv5.php 'Touched AFT Special page file'
19:42 ottomata1: added gadolinium as a udp2log host on frontend caches
19:14 logmsgbot: mlitn synchronized wmf-config/CommonSettings.php 're-include AFTv5'
19:08 logmsgbot: mlitn synchronized php-1.21wmf12/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:07 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5 'Update ArticleFeedbackv5 to master'
19:03 paravoid: adding ruby-multi-json (universe), ruby-parallel, ruby-dimensions, ruby-execjs, ruby-jsduck (main) to apt
18:46 mutante: dist-upgrade on gallium
18:03 logmsgbot: robh synchronized docroot
18:01 logmsgbot: robh synchronized docroot
17:21 mark: Ended bits.esams performance testing
16:29 mark: Starting load testing with esams bits servers
16:20 mark: Rebooted cp3019 with hyperthreading and DAPC disabled
16:16 logmsgbot: krinkle synchronized extract2.php
14:29 mark: Rebooting arsenic with hyperthreading disabled
12:54 Tim: on professor: restarted collector
08:46 hashar: gallium: killed stalled puppet agents. Locked on apt-get and update-java-alt
08:41 hashar: gallium: restarted puppet and zuul
08:38 hashar: gallium: stopped zuul and upgrading
08:37 hashar: gallium: stopped puppet to deploy Zuul whenever the queue is empty
08:36 hashar: zuul : giving results processing priority over enqueued jobs. Cherry picked 263fba9 from upstream. That should resolve Zuul being slow to report back to Gerrit bug 46176
03:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:50 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 Tim: started incremental updater on searchidx1001 and searchidx2, apparently has not been running since March 13
02:50 logmsgbot: LocalisationUpdate completed (1.21wmf12) at Tue Mar 19 02:50:19 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 19 02:29:02 UTC 2013
01:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 Ryan_Lane: adding dns entries for labstore systems in eqiad
00:08 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'returning servers'

March 18

23:34 binasher: i <3 running "dpkg -P mysql-server-5.1 mysqlfb-client-5.1 mysqlfb-common mysqlfb-server-5.1 mysqlfb-server-core-5.1 libmysqlfbclient16 libmysqlclient16"
23:11 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db10[10-11],[26-28] at low weights'
22:45 binasher: converted rogue s3 tables (moodbar_feedback, trackbacks, transcache) to innodb
22:28 ottomata: added python-flask-login-0.1.2-1 to apt
22:23 logmsgbot: robh gracefulled all apaches
22:20 LeslieCarr: intradatacenter link is flapping , switching links, this may cause some higher latency
22:14 logmsgbot: robh gracefulled all apaches
22:11 logmsgbot: robh gracefulled all apaches
21:58 mutante: purging wikimediafoundation.org from squid
21:57 logmsgbot: dzahn gracefulled all apaches
21:56 logmsgbot: robh gracefulled all apaches
21:36 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling a db from s3-7 for upgrade'
21:30 paravoid: authdns-update: removing *.ts.wikimedia.org records
21:29 logmsgbot: dzahn gracefulled all apaches
21:25 binasher: ran "ddsh -g apaches -cM '/etc/init.d/apache2 start'"
21:24 RobH: authdns-update rt4683
21:22 RobH: gracefuling all apaches resulted in rendering cluster overload, they didnt restart apache, odd
21:21 logmsgbot: robh synchronized docroot
21:18 LeslieCarr: service apache2 start on the mw-eqiad group via dsh
21:14 binasher: started apache on all imagescalers
21:00 logmsgbot: robh gracefulled all apaches
21:00 RobH: troubleshooting erros in apache restart script
20:59 logmsgbot: robh gracefulled all apaches
20:36 RobH: authdns-update rt4694
20:20 RobH: authdns-update
20:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
20:00 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:44 ^demon: jetty freaked out again, forcing a gerrit restart
19:32 binasher: upgraded all coredb mariadb replicas to 5.5.30
19:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for 1.21wmf12
19:08 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 db1009'
19:04 logmsgbot: reedy synchronized docroot
19:03 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'db1050 for special s1'
18:56 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1043, db1009 for upgrade'
18:40 logmsgbot: reedy synchronized php-1.21wmf12 'Initial sync of php-1.21wmf12'
18:16 awight: payments cluster updated from c19cc66 to fe4fb96b
18:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
18:03 logmsgbot: aaron synchronized wmf-config/PrivateSettings.php 'Removed old testing cruft'
16:19 cmjohnson1: removing/replacing disk 4 ms-be1004
16:01 hashar: Hard restarted Zuul which was dead locked again :-(
13:23 paravoid: depooled, upgraded, restarted and pooled again all ms-fe[1-4]
13:02 paravoid: depooling ms-fe1 for new swift version (patched 1.7.4)
11:27 apergos: restarted both the varnishncsas on niobium, they were giant again
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Mon Mar 18 02:30:03 UTC 2013

March 17

02:31 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sun Mar 17 02:31:16 UTC 2013

March 16

14:58 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:48 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
14:40 logmsgbot: reedy synchronized php-1.21wmf11/extensions/EducationProgram
13:07 hashar: Mar 16, 2013 1:06:47 PM Jenkins is fully up and running
12:55 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EducationProgram/includes/pages/EducationPage.php 'Fix fatal error in EducationProgram (Bug 46188)'
12:33 hashar: restarting Jenkins on a quiet day to have it using umask 0002 (aka g+w). That needs a full restart since the umask is set by /usr/bin/daemon See 53990.
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Sat Mar 16 02:29:19 UTC 2013

March 15

23:24 logmsgbot: laner synchronized wmf-config/CommonSettings.php 'Disabling aft5'
22:45 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning es1007,10 at lower weights'
22:35 binasher: rebooted es1007,1010 for kernel upgrades
22:33 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'temporarily pulling es1007, es1010'
22:23 Ryan_Lane: hid domain dropdown for account creation and login on wikitech
22:23 Ryan_Lane: changed wikitech configuration to require email address on account creation
22:23 Ryan_Lane: deployed change 54093 to wikitech
22:03 hashar: Zuul : opened a bug about this week slowness, breakage: https://bugzilla.wikimedia.org/show_bug.cgi?id=46176
21:51 logmsgbot: aaron synchronized php-1.21wmf11/includes 'deployed 08295adcdc4a4130954f52fb1eb5377638cd97bc'
21:36 binasher: db59 -> mariadb 5.5.30
21:30 hashar: restarted Zuul with the safe version ff79197
21:28 hashar: downgrading Zuul to safe version ff79197 (aka stop doing `git remote update` on ref-update events)
21:25 binasher: added new build of mariadb 5.5.30 to precise-wikimedia on brewster
21:20 hashar: hard restarted Zuul. It was locked down again :/
21:18 mutante: installing package upgrades on gallium
21:14 hashar: restarting Zuul, it is locked somehow. Definitely need to update it :(
21:09 RobH: professor services seem back up
20:51 RobH: professor shutting down for fan swap, short downtime (hopefully)
20:47 hashar: integration.mediawiki.org is not redirecting to the new integration.wikimedia.org (apache conf is disabled). Patch in 54068
20:33 mutante: puppet runs on neon broken due to mysql puppet issue - mysql-client-5.5 can not create alias mysql-client: object already exists at /var/lib/git/operations/puppet/manifests/mysql.pp:513
20:31 mutante: restarting pdns on ns0
20:30 mutante: DNS update - removing ipv4/ipv6/ipv6and4/results.labs.wm
19:37 mutante: maerlant - re-adding 2620:0:862:1::80:2 to eth0, starting nginx
19:22 sbernardin: taking down raskin for decommissioning
19:13 mutante: maerlant - dpkg: error processing ganglia-monitor | backing up /var/www/ to tridge /data/other then deleting contents | removing sites-enabled/ipv6and4 | nginx can't bind to [2620:0:862:1::80:2]:443
18:50 mutante: installing more package upgrades on maerlant (wikimedia-lvs-realserver, libc6, nginx...)
18:46 mutante: installing package upgrades on formey
18:44 mutante: fixing broken dpkg/APT on maerlant - dpkg --configure -a , re-run puppet and let it install package upgrades
18:34 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:19 notpeter: ran puppetd --enable on all mw hosts
18:15 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/
18:07 logmsgbot: mlitn Finished syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:52 logmsgbot: mlitn Started syncing Wikimedia installation... : Update ArticleFeedbackv5 to master
17:36 hashar: jenkins: updated jenkins configuration to have its URL default to wikimedia.org domain instead of the legacy mediawiki.org
17:22 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
17:21 hashar: gallium : reloading Zuul so it reports the status url with the wikimedia.org domain
17:19 hashar: integration website going down while we deploy a new layout.
17:03 hashar: gallium : restarted puppet
17:03 hashar: gallium : update /srv/ to latest version of integration/docroot.git ( aaa4b0c ) and moved all content from /srv/org/mediawiki/integration to the new /srv/org/wikimedia/integration
17:00 hashar: stopped puppet on gallium to fix some permissions before puppet changes are merged in and applied.
05:09 Tim: restarting squid on cp1010 for squid redirector test
05:06 Tim: testing new squid redirector
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Fri Mar 15 02:30:02 UTC 2013
02:16 logmsgbot: reedy synchronized docroot
02:13 logmsgbot: dzahn gracefulled all apaches
01:53 logmsgbot: reedy synchronized wmf-config/
01:52 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:52 logmsgbot: reedy gracefulled all apaches
01:44 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/SpecialGettingStarted.php 'Fix deployed bug in check for editability'
01:44 logmsgbot: reedy synchronized wmf-config/
01:43 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:35 logmsgbot: reedy synchronized wmf-config/
01:35 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:34 logmsgbot: reedy synchronized docroot
01:28 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:27 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
01:21 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/GettingStarted/CategoryRoulette.php 'Fix to use single-parameter version of SRANDMEMBER'
01:14 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
01:13 logmsgbot: dzahn gracefulled all apaches
01:12 logmsgbot: reedy synchronized docroot
01:10 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Missed a file in earlier sync'
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'chapcomwiki readonly'
00:47 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming/modules/ext.navigationTiming.js 'Additional fields for NavTiming'
00:43 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deployment for Mobile and E3
00:17 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deployment for Mobile and E3

March 14

23:48 mutante: DNS update - add affcom in prep. to replace chapcom (bug 39482)
23:47 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo path for enwiki'
23:47 logmsgbot: awjrichards synchronized php-1.21wmf11/extensions/MobileFrontend/ 'Updating MobileFrontend per https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-03-14'
23:44 mutante: installing package upgrades on sockpuppet
23:30 logmsgbot: awjrichards synchronized wmf-config/InitialiseSettings.php 'Updating MF copyright logo for testwiki'
22:52 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:43 mutante: pa.us.wikimedia.org now redirects to pa-us.wikimedia.org (Wikimedia Pennsylvania) (no sub.sub domains)
22:41 logmsgbot: dzahn gracefulled all apaches
22:34 mutante: disabling notifications for puppet freshness on celsus
22:14 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling NavigationTiming at 1:10,000 sampling factor across cluster.'
22:02 logmsgbot: olivneh synchronized wmf-config 'Enable NavigationTiming on test2'
21:57 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/NavigationTiming 'Update of extension to use EventLogging'
21:51 hashar: Gerrit took roughly 5 minutes to merge the change https://gerrit.wikimedia.org/r/#/c/53412/ ( mediawiki/core ) which means Zuul does nothing during that time waiting for the merge to happen.
21:48 hashar: apparently Gerrit takes ages to submit a change that in turn cause a hard lock on Zuul which is waiting for the merge to happen.
21:39 hashar: Zuul is hanging out not reporting changes :/
21:01 mutante: DNS update - add pa-us wiki to replace pa.us
20:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:50 cmjohnson1: authdns update (labsdb1002-3 vlan)
20:45 Ryan_Lane: deleting the jenkins user from ldap
20:36 mutante: DNS update - add zuul as CNAME for gallium
20:34 Ryan_Lane: installed WikiLove on wikitech
20:30 ori-l: Added article edit event to EventLogging, which upped events/sec from ~3.5 to ~30. Expect increased load on vanadium and db1047; bits caches unaffected.
20:20 logmsgbot: olivneh synchronized php-1.21wmf11/extensions/EventLogging 'Log ['HTTP_HOST'] as webHost'
20:05 logmsgbot: olivneh synchronized php-1.21wmf11/includes/WikiPage.php 'Adds CategoryAfterPageAdded / CategoryAfterPageRemoved hooks'
19:32 logmsgbot: aaron synchronized php-1.21wmf11/includes/job/jobs 'deployed 6f76ede163cb114724bce7a1c1b8938e1e30606f '
18:49 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enable wgEnableAsyncUploads only on testwikis for now'
18:44 logmsgbot: mlitn synchronized php-1.21wmf11/extensions/ArticleFeedbackv5/ 'Update ArticleFeedbackv5 to master'
18:41 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 to full weight'
18:32 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1009 at a low weight for warmup'
18:12 logmsgbot: aaron synchronized wmf-config/InitialiseSettings.php 'Enabled $wgEnableAsyncUploads on all wikis'
18:08 binasher: cleaned up myisam cruft (unused t��ables) on s2
17:49 logmsgbot: aaron synchronized wmf-config 'Enabled redis job aggregator fully'
17:48 cmjohnson1: labsdb1001 powering down
17:42 logmsgbot: aaron synchronized wmf-config 'Set redis job aggregator on for testwiki.'
17:27 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Math/
17:24 binasher: shutting down mysql on db1009, upgrading to mariadb
16:59 cmjohnson1: powering down labsdb1002 and labsdb1003
16:59 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'pulling db1009 for mariadb migration'
16:57 cmjohnson1: addin mw1041 back to dsh groups
16:26 logmsgbot: mlitn synchronized wmf-config 'Prepare AFTv5 config for deployment new features'
13:15 mark: Set OSPF cost to 10 on csw1-esams:ve7 to facilitate ip multipath
13:15 mark: Fixed OSPF on cr2-knams:xe-0/0/0 - csw1-esams:e8/2 (earlier)
12:17 hashar: gallium setting up /srv/ as a copy of integration/docroot.git
09:03 apergos: restarted varnishncsa.vanadium on niobium, it was using 9gb memory or so
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Thu Mar 14 02:25:19 UTC 2013
02:21 hashar: jenkins: made Zuul to block changes on operations/debs/ircecho whenever pep8/pyflakes fails.
01:35 mutante: running puppet on celsus to investigate puppet freshness reports from icinga - it started parsoid, but also celsus is a Wikimedia DECOMMISSIONED server (base::decommissioned).
01:10 logmsgbot: reedy synchronized php-1.21wmf11/cache/l10n/
01:00 logmsgbot: reedy synchronized php-1.21wmf11/extensions/Scribunto
00:53 logmsgbot: reedy synchronized wmf-config/

March 13

23:44 mutante: installing package upgrades on zirconium
23:41 logmsgbot: demon Finished syncing Wikimedia installation... :
23:38 Ryan_Lane: deploying change 53464 to OpenStackManager on wikitech
23:23 RobH: no one sync apaches, we wre doing crazy magic on them
23:17 logmsgbot: demon Started syncing Wikimedia installation... :
23:05 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
23:00 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqiad. upgrades over.'
22:53 logmsgbot: reedy synchronized wmf-config/
22:44 logmsgbot: demon synchronized php-1.21wmf11/extensions/Scribunto/ 'Updating Scribunto to master'
22:30 logmsgbot: robh gracefulled all apaches
21:46 awight: payments cluster updated to 61d3f1f
21:43 pp-pdf1: upgraded mwlib to 0.15.3
21:38 Tim: on professor: restarted collector
21:22 RobH: repooling all ssl servers (except broken ssl3004) in all sites. new unified certificate is now in use.
21:15 RobH: depooling ssl servers 3 and 4 in each dc site
21:14 RobH: repooling the first two ssl servers per site
21:08 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all earch traffic to pmtpa for upgrades in eqiad'
21:05 RobH: setting first 2 ssl servers per dc site to false in pybal
20:21 Ryan_Lane: repooling ssl1001
20:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Re-enable collection'
20:16 Ryan_Lane: depooling ssl1001
20:08 LeslieCarr: cp1022 unresponsive, power cycling
19:44 LeslieCarr: switched preferred path between esams and eqiad
19:41 LeslieCarr: deploying a possibly mobile front end traffic effecting ipv6 apache request firewall filter
19:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable Collection extension'
19:08 LeslieCarr: site was popedotted !
18:47 logmsgbot: py synchronized wmf-config/lucene-production.php 'moving all search traffic back to eqaid'
18:31 logmsgbot: py synchronized wmf-config/lucene-production.php 'temp moving all search traffic to pmtpa for upgrades in eqiad'
18:16 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable scribunto EVERYWHERE'
18:09 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf11
17:23 pp-pdf1: downgrade to 0.14.3
16:39 pp-pdf1: upgraded mwlib to 0.15.1.
16:32 apergos: cleaned up borked puppet agent on ms-be12 and re-enabled/restarted it
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Hooks.php 'Fix for bug 46031'
12:30 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto/common/Common.php 'Fix for bug 46031'
12:07 Danny_B: reenabled mail delivery to wikibugs 30 mins ago (disabled automatically for excessive bounces)
11:42 apergos: restarted morebots
01:10 Tim: running updateCollation.php on all uca-* wikis in 6 screens on hume
00:45 logmsgbot: dzahn gracefulled all apaches
00:39 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:35 Ryan_Lane: pooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:31 Ryan_Lane: depooling ssl3 ssl4 ssl1003 ssl1004 ssl3003
00:30 Ryan_Lane: pooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:16 Ryan_Lane: depooling ssl1 ssl2 ssl1001 ssl1002 ssl3001 ssl3002
00:12 Tim: manually creating missing /etc/apache2/wmf symlinks on servers that lack them
00:07 logmsgbot: dzahn gracefulled all apaches

March 12

23:39 Tim: updating PHP to php5-3.5+wmf1 for new ICU
23:39 notpeter: upgrading pmtpa search nodes to lucene-search-2 2.1.7wm1
23:34 logmsgbot: tstarling synchronized php-1.21wmf11/maintenance/updateCollation.php
23:34 logmsgbot: tstarling synchronized php-1.21wmf10/maintenance/updateCollation.php
22:33 Ryan_Lane: pooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:29 Ryan_Lane: depooling ssl3003 ssl3 ssl4 ssl1003 ssl1004
22:28 Ryan_Lane: pooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
22:23 mutante: pushing new version 2.7 of wikimedia-task-appserver to wikipedia-precise repo (to fix apache-sanity-check)
22:21 Ryan_Lane: depooling ssl3001 ssl3002 ssl1 ssl2 ssl1001 and ssl1002
21:23 Ryan_Lane: seems that the unified certificate is missing mediawiki.org, depooling the ssl servers that are serving it
21:22 Ryan_Lane: installing the new unified ssl certificate on all ssl servers
19:20 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 RobH: authdns-update for europium to gadolinium ip
18:47 RobH: europium shuttin down
18:43 mark: Imported ganglia 3.5.0 packages into the precise-wikimedia APT repository
18:19 RobH: europium reclaimed per rt4689
18:02 RobH: en-wp.com/org now redirect to en.wikipedia per rt 4669
18:00 RobH: ran apache restart as myself, not root, had a bunch of fails. restarted them as root, hopefully nothing noticable to users
17:58 logmsgbot: robh gracefulled all apaches
17:53 Jeff_Green: live-hack trivial OTRS patch to support filtering on X-Spam-Score for RT #4713, will check in once project is copied to git
17:42 logmsgbot: anomie synchronized php-1.21wmf11/extensions/Scribunto 'Update Scribunto to master'
17:29 logmsgbot: reedy synchronized php-1.21wmf10/includes/Collation.php
17:28 logmsgbot: reedy synchronized php-1.21wmf11/includes/Collation.php
15:42 paravoid: apache graceful to fix bz 46018 / RT 4695
15:39 logmsgbot: root gracefulled all apaches
15:32 hashar: Restarted Zuul process. Was stuck somehow waiting for a job that has been canceled.
13:30 hashar: Jenkins restarted!
12:55 hashar: gallium : upgrading Jenkins (unscheduled)
08:32 hashar: jenkins: live hacked the ant build script for MediaWiki jobs. Now using the local replicated git directory instead of a handmade replication.
05:03 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Math
05:02 logmsgbot: tstarling synchronized php-1.21wmf11/extensions/Math
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Mar 12 02:52:43 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf11) at Tue Mar 12 02:29:54 UTC 2013
02:16 Ryan_Lane: deploying change 53248 and 52951 to OpenStackManager on wikitech
02:10 Ryan_Lane: deployed change 53304 to OATHAuth on wikitech
00:38 logmsgbot: kaldari synchronized php-1.21wmf11/extensions/UploadWizard/ 'deploying some bugfixes for UploadWizard'
00:15 awight: payments cluster updated to 207a112b5b36a9e125999acc6f4098d9058ef594

March 11

23:48 AaronSchulz: syncFileBackend.php Running for ceph for all wikis now
23:48 AaronSchulz: copyFileBackend.php finished for ceph for all originals and transcodes (plus math/captcha)
23:38 logmsgbot: aaron synchronized php-1.21wmf11/includes/clientpool/RedisConnectionPool.php 'deployed 5af8702a4b4208ac42fab2fb36da69d13f3e3b92'
23:37 logmsgbot: aaron synchronized php-1.21wmf10/includes/clientpool/RedisConnectionPool.php 'deployed b9fdedc57295a4dd1da684859871aeb734bf70fe'
23:10 Ryan_Lane: adminbot upgraded to 1.6.3
23:09 Ryan_Lane: upgrading adminbot on wikitech-static
22:31 LeslieCarr: upgrading ircecho on neon
22:30 Ryan_Lane: upgrading ircecho on manganese
21:20 mutante: gracefulling eqiad Apaches - wikiquote.net now redirects to wikiquote.org
21:19 logmsgbot: dzahn gracefulled all apaches
21:07 Ryan_Lane: deployed change 53267 to OpenStackManager on wikitech
20:59 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
20:59 logmsgbot: reedy synchronized php-1.21wmf11/cache/interwiki.cdb 'Updating 1.21wmf11 interwiki cache'
20:30 ^demon: Upgraded gerrit to 2.5.2-1636-g353e384, all's well
20:18 Ryan_Lane: deploying change 52950 to OpenStackManager on wikitech
19:22 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf11
19:05 mutante: patching Bugzilla for security - RT-4553
17:45 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:34 mwalker: Shields up! Payments deploy from a6fe052a73f1421028b44305908ec6984dbc50fa to head for Japanese localization updates.
17:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:19 RobH: search23 shutting down for mb sap per rt4323, has been disabled in pybal
17:18 RobH: search23 out of search pool1, other servers in pool online
17:17 RobH: setting search23 to false for mb swap
17:10 sbernardin: taking down search23 for MB replacement: https://rt.wikimedia.org/Ticket/Display.html?id=3423
16:48 logmsgbot_: aaron synchronized php-1.21wmf11/includes/limit.sh 'deployed e77397fc1de324dfc6d7b515e9440bd564311dd9'
15:57 ottomata: deployed new webstatscollector filter on locke. This replaces spaces in urls.
11:39 apergos: ook that was wrong started apache2 on stafford (puppet uses passenger), now things look ok
10:51 apergos: that would be 10.5. can't add.
10:50 apergos: shot old puppet agents hanging around on stafford, also old apaches from dead puppet master, restarted puppet master, it died around 6.5 hours ago
02:52 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Mon Mar 11 02:52:06 UTC 2013
02:43 logmsgbot_: krinkle synchronized wmf-config/CommonSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:41 logmsgbot_: krinkle synchronized wmf-config/InitialiseSettings.php 'I7c8174bcf6ca411014a712a6416d3a74e9fecb3d'
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Mon Mar 11 02:29:27 UTC 2013

March 10

22:19 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache
21:59 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache
21:10 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/CentralAuth/
21:00 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 9

02:54 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Sat Mar 9 02:54:23 UTC 2013
02:37 mutante: deploying blog WMBlog.php changes (gerrit 28835, gerrit 29823)
02:33 Reedy: Created cn_notice_log on enwiki. Not sure it's actually needed, but there's dberrors appearing here and htere
02:31 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Sat Mar 9 02:31:06 UTC 2013
02:01 mutante: deploying WMBlog.php changes for EventLogging (gerrit 52945)
01:26 logmsgbot_: olivneh synchronized wmf-config/InitialiseSettings.php '(Change If7ab3d016 / bug 45830) Separate slow parse log destination for private wikis'
01:13 Ryan_Lane: cherry-picking in change 52950 on wikitech
01:12 Ryan_Lane: scratch that
01:11 Ryan_Lane: deploying OATHAuth master to wikitech
00:32 mutante: deleting squid logs on brewster to prevent out-of-disk, and setting logrotate to compress
00:25 Ryan_Lane: upping OpenStackManager to master on wikitech
00:22 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
00:00 Ryan_Lane: upping OpenStackManager to master on wikitech

March 8

22:28 logmsgbot_: reedy synchronized wmf-config/
22:28 logmsgbot_: py synchronized wmf-config/lucene-production.php 'moving pool4 and pool5 search traffic back to eqiad'
22:21 logmsgbot_: reedy synchronized php-1.21wmf10/includes/Collation.php
21:47 awight: payments cluster update to a6fe052a73
20:29 Ryan_Lane: enabling quorum feature for all gluster volumes
19:55 RobH: authdns-update, why? because i can.
19:34 Ryan_Lane: added Babel extension to wikitech
19:20 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'removing a second node per shard for streaming dumps. stopping slave'
15:35 ottomata: adding metrics.wikimedia.org domain, running authdns-update on dobson
06:21 Tim: updated luasandbox to version 1.4 (with pauseUsageTimer)
04:01 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Translate
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Fri Mar 8 02:52:54 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Fri Mar 8 02:28:34 UTC 2013
01:27 mutante: installing package upgrades on ekrem
01:12 logmsgbot_: catrope synchronized wmf-config/InitialiseSettings.php '20388a06 - Normalize VE config on testwiki'
00:57 mutante: added Set(@ReferrerWhitelist, qw(rt.wikimedia.org:443 rt.wikimedia.org:80));
00:55 Ryan_Lane: deploying change 52592 to wikitech
00:55 mutante: killall lighttpd and restart lighttpd and apache on streber
00:19 logmsgbot_: kaldari Finished syncing Wikimedia installation... :
00:06 mutante: installing openjdk/icedtea jre and other package upgrades on gallium (jenkins)
00:05 LeslieCarr: upgrading cp1001

March 7

23:54 logmsgbot_: kaldari Started syncing Wikimedia installation... :
23:47 LeslieCarr: doing tmh upgrades
23:47 mutante: bravely rebooting streber - expect short RT and Observium outage
23:25 LeslieCarr: upgrading solr boxes
23:14 mutante: installing kernel on streber
23:11 ^demon: gerrit: disabling hooks-its plugin for the time being. It's working (yay) but it's overly spammy until we fix it next week.
23:04 LeslieCarr: dist-upgrading pmtpa memcache
23:04 mutante: dist-upgrading zhen
22:38 mutante: rebooting yvon
22:34 mutante: installing package upgrades on yvon despite (defunct apt-get procs)
22:31 mutante: salt pkg.upgrade on yvon
22:28 Ryan_Lane: rebooting marmontel
22:25 mutante: salt pkg.upgrade on spence
22:21 mutante: salt pkg.upgrade on zirconium
22:18 LeslieCarr: upgrading all analytics machiens
22:06 mutante: reboot formey after dist-upgrade via salt
20:05 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php 'fa0a56a89 - parsoidcache VIP in pmtpa'
19:37 logmsgbot_: bsitu synchronized wmf-config/CommonSettings.php 'Add Eventlogging configuration to Echo'
19:17 logmsgbot_: pgehres synchronized php-1.21wmf11/extensions/CentralNotice/ 'Updating CEntralNotice to master, resolving bug 45846'
18:41 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:39 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
17:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
16:07 cmjohnson1: reseating disk4 on ms-be1004
15:28 mark: Reinstalled cp3003
15:00 mark: Restarted gmetad on nickel
02:53 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Thu Mar 7 02:53:50 UTC 2013
02:29 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Thu Mar 7 02:29:38 UTC 2013
01:53 mutante: installing package upgrades on sodium
00:54 RobH: kicked icinga
00:49 LeslieCarr: restarted pybal on lvs3
00:40 RobH: puppet locked on ms1004, killall puppet, refired
00:39 RobH: puppet halted run on cp1003, left in locked state, killed all puppet instances and refired, cleared.
00:33 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/RSS
00:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php

March 6

23:19 LeslieCarr: reinstalling celsus as an internal host
23:07 logmsgbot_: reedy synchronized php-1.21wmf10/resources/ 'touch'
23:06 logmsgbot_: reedy synchronized php-1.21wmf11/resources/ 'touch'
23:05 logmsgbot_: reedy synchronized wmf-config 'touch'
22:54 logmsgbot_: reedy synchronized wmf-config/
22:53 logmsgbot_: reedy synchronized wmf-config/
22:40 mutante: starting haproxy on brewster
22:39 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:39 notpeter: reslaving db36 and db38
22:29 RobH: authdns-update
22:20 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
22:20 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild message cache for Wikibase deploy
22:11 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php
22:09 logmsgbot_: reedy synchronized wikidataclient.dblist
22:07 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:02 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
21:55 logmsgbot_: reedy cleared profiling data
21:55 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on all wikipedias'
21:41 awight: updated payments cluster to 39a265489
21:16 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/
20:54 logmsgbot_: reedy synchronized wmf-config/
20:38 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues/
20:29 Reedy: Creating and populating site_identifiers table on all Wikipedias
20:28 Reedy: Creating and populating sites table on all Wikipedias
20:27 logmsgbot_: olivneh synchronized wmf-config/CommonSettings.php 'Adding redis host config. var for GettingStarted'
20:12 logmsgbot_: reedy synchronized wmf-config/
20:07 logmsgbot_: reedy synchronized wmf-config/
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Wikibase
20:04 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/DataValues
20:03 logmsgbot_: reedy synchronized php-1.21wmf10/extensions/Diff
19:57 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/Wikibase
19:56 logmsgbot_: reedy synchronized php-1.21wmf11/extensions/DataValues
19:43 notpeter: restarting all jobrunners
19:34 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:29 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
19:28 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource to 1.21wmf11
19:25 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: aawiki back to 1.21wmf10 for Aaron
19:21 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
19:20 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikimedia and wikiquote to 1.21wmf11
19:18 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks and wikinews to 1.21wmf11
19:15 Reedy: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikiversity and wiktionary to 1.21wmf11
19:10 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Special wikis to 1.21wmf11
19:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Closed, private and fishbowl to 1.21wmf11
19:03 mutante: gracefulling apaches to remove 1.17 aliases (gerrit 50609)
19:03 logmsgbot_: dzahn gracefulled all apaches
18:58 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
18:47 mutante: regenerating mailman list archives for wmfreqs
18:44 mutante: renaming mailman list 'allhands' to 'wmfreqs' (RT-4640)
18:36 logmsgbot_: aaron synchronized php-1.21wmf11/maintenance/nextJobDB.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/job/JobQueueGroup.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:35 logmsgbot_: aaron synchronized php-1.21wmf11/includes/DefaultSettings.php 'deployed cbc98b09fb0a13b387d82bafab825608e2bb0b3b'
18:10 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings.php 'clear config cache'
18:01 logmsgbot_: hashar synchronized wmf-config 'deploying 52340 adapt Lucene configuration file to support beta'
17:59 hashar: deploying 52340 adapt Lucene configuration file to support beta
07:08 jeremyb_: 02:52:46 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf11) at Wed Mar 6 02:52:46 UTC 2013
07:08 jeremyb_: 02:36:35 <+logmsgbot> !log reedy synchronized wmf-config/InitialiseSettings.php
07:08 jeremyb_: 02:28:21 <+logmsgbot> !log LocalisationUpdate completed (1.21wmf10) at Wed Mar 6 02:28:21 UTC 2013
07:08 jeremyb_: 01:33:50 <+logmsgbot> !log csteipp synchronized wmf-config/CommonSettings.php 'Revert https patch'
07:07 jeremyb_: 01:08:27 <+logmsgbot> !log reedy synchronized wmf-config/CommonSettings.php 'Revert wgSecureLoginDefaultHTTPS'
07:07 Ryan_Lane: restarted morebots
01:10 logmsgbot_: reedy synchronized wmf-config/CommonSettings.php 'Set wgSecureLoginDefaultHTTPS to false for testing'
00:37 binasher: added new deb of twemproxy 0.2.3 built with --enable-debug=log to precise-wikimedia
00:21 logmsgbot_: csteipp synchronized wmf-config/CommonSettings.php 'Enabling HTTPS login'
00:14 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php
00:09 hashar: snapshot1002: rsync: change_dir#3 "/apache/common-local" failed: No such file or directory (2)
00:09 logmsgbot_: hashar synchronized wmf-config/InitialiseSettings-labs.php

March 5

23:57 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'increasing db1009 to full weight'
23:54 LeslieCarr: reinstalling constable as an internal host
23:49 RoanKattouw: Removed decommissioned srv226-234 from /etc/dsh/group/mediawiki-installation
23:47 logmsgbot_: catrope synchronized wmf-config/CommonSettings.php '6fd144c44f26 - Use new service IP for Parsoid in eqiad'
23:46 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'repooling previously crashed mariadb pmtpa slaves'
23:36 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php 'ombudsmenwiki logo'
23:26 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'commenting out one slave per shard that's mysqldumping'
23:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:15 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
23:13 notpeter: stopping slave and starting mysql dumps on dbs 71, 57, 66, 65, 55, 50, and 68
23:00 logmsgbot_: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:40 logmsgbot_: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:36 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'repooling db1009 at 25% weight'
22:18 logmsgbot_: reedy synchronized wmf-config/InitialiseSettings.php
22:17 logmsgbot_: reedy synchronized wmf-config/
22:14 LeslieCarr: restarting pybal on lvs3
22:14 LeslieCarr: restarting pybal on lvs4
22:13 LeslieCarr: restarting pybal
22:07 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
22:05 logmsgbot_: reedy synchronized wmf-config/
21:57 LeslieCarr: restarting pybal on lvs1003
21:55 LeslieCarr: restarting pybal on lvs1006
21:44 Coren: tools rejiggered the webproxy config to be smarter about paths not leading to specific tools
21:43 LeslieCarr: adding parsoidcache service to lvs1006
21:42 mutante: gracefulling eqiad Apaches - add ombudsmen.wm (bug 45744)
21:41 logmsgbot_: dzahn gracefulled all apaches
21:35 mutante: restarting pdns on ns2
21:34 mutante: DNS update - add ombudsmen (bug 45744)
20:49 logmsgbot_: reedy synchronized docroot
20:35 mutante: installing package upgrades on hooper
20:26 logmsgbot_: reedy synchronized wmf-config/
20:22 mutante: installing package upgrades on hume
20:00 mutante: wikipedia.ee now redirects to et.wikipedia, wikimedia.ee now redirects to et.wikimedia
19:56 mutante: gracefulling eqiad Apaches, push wiki(m|p)edia.ee redirects
19:56 logmsgbot_: dzahn gracefulled all apaches
19:18 K4-713: updated payments cluster to 9b063ee2e399a3b
18:42 logmsgbot_: mlitn synchronized wmf-config/InitialiseSettings.php 'Disable AFTv4 (entirely) and AFTv5 (leaving only opt-in) on enwiki'
18:04 cmjohnson1: restarting dhcpd3 service brewster
16:55 cmjohnson1: mw1085 powering off to troubleshoot DIMM error
05:35 ori-l: EventLogging db on db1047: ALTER DATABASE log CHARACTER SET utf8 COLLATE utf8_unicode_ci; (was: latin1)
02:49 logmsgbot_: LocalisationUpdate completed (1.21wmf11) at Tue Mar 5 02:48:58 UTC 2013
02:28 logmsgbot_: LocalisationUpdate completed (1.21wmf10) at Tue Mar 5 02:28:16 UTC 2013
02:03 K4-713: updated payments cluster to 5361c795f4bf31980f
00:46 mutante: fixing exim/mail permissions on neon to make Icinga paging work. missing +x on /var/log/exim4 dir

March 4

23:12 mutante: starting dhcp3 on brewster
23:11 LeslieCarr: p.s. bonding on foundry sucks
23:10 LeslieCarr: bringing down virt2 network to bond it
22:18 mutante: gracefulling eqiad Apaches to push indiawikipedia redirect
22:17 logmsgbot_: dzahn gracefulled all apaches
22:03 logmsgbot_: py synchronized wmf-config/db-pmtpa.php 'shuffling around pmtpa enwiki slaves'
21:50 LeslieCarr: messing around with the network of virt2 - expect possible pmtpa labs network issues
21:50 ottomata: staring puppet back up on oxygen, packet loss is better after disabling a filter
21:30 cmjohnson1: authdns update
21:17 logmsgbot_: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf11
21:16 logmsgbot_: reedy synchronized php-1.21wmf11
21:15 logmsgbot_: reedy synchronized php-1.21wmf10
21:14 RobH: shutting down professor for a fan swap, should be a short window
21:11 cmjohnson1: authdns update
21:05 ottomata: stopping puppet on oxygen to see if we can disable filters and reduce packet loss
20:55 Ryan_Lane: restarted icinga
20:53 logmsgbot_: reedy Finished syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
20:52 RobH: had to restart morebots, but it wasnt really dead, opps
20:51 RobH: had to restart icinga
20:51 RobH: Why arent you taking all the !log
20:50 RobH: i hate you morebots
20:28 logmsgbot_: reedy Started syncing Wikimedia installation... : Rebuild Message Cache for 1.21wmf11
19:55 logmsgbot_: reedy Started syncing Wikimedia installation... : Build 1.21wmf11 message cache
19:41 logmsgbot_: reedy synchronized php-1.21wmf11/
19:34 logmsgbot_: reedy synchronized wmf-config/
19:34 logmsgbot_: reedy synchronized live-1.5/
19:33 logmsgbot_: reedy synchronized docroot
19:30 hashar: on gallium: stopping puppet, cleaning defunct and restarting puppet
19:23 logmsgbot_: reedy synchronized wmf-config/
19:20 mutante: installing package upgrades on sockpuppet
19:13 sbernardin: taking down amaranth for hw replacement: https://rt.wikimedia.org/Ticket/Display.html?id=4497
18:44 cmjohnson1: authdns update
18:17 K4-713: updated payments cluster to 059969c3d29f685c
17:44 cmjohnson1: authdns update
16:56 logmsgbot_: py synchronized wmf-config/db-eqiad.php 'depooling db1009 while investigating error'
16:24 sbernardin: rebooting db70 to reset idrac: https://rt.wikimedia.org/Ticket/Display.html?id=4628
16:14 sbernardin: db27 being taken down for decom per: https://rt.wikimedia.org/Ticket/Display.html?id=4623
10:12 apergos: cleared some space on brewster by rmeoving old squid access/storage logs (2.2gb worth in 3 files!), its root partition was full, restarted squid
10:12 apergos: reinstalled wikimedia-app-taskserver on fenari (how did it get removed?), needed for various cron jobs among other things
02:04 logmsgbot_: LocalisationUpdate failed (1.21wmf10) at Mon Mar 4 02:04:10 UTC 2013

March 3

02:25 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Mar 3 02:25:44 UTC 2013

March 2

02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Mar 2 02:27:35 UTC 2013
01:17 mutante: powercycling emery
00:35 mutante: rebooting emery for kernel upgrade
00:29 mutante: installing package upgrades on emery

March 1

23:59 cmjohnson1: authdns update
23:56 K4-713: updated payments cluster to c8442435a9e2f5c
23:29 notpeter: dropping limesurvey from db9, backup in db9:/a/backup_dumps
22:53 RoanKattouw: Make that apache2-utils, that's the actual name of the package
22:51 RoanKattouw: Temporarily installing apachebench on wtp1004 per Gabriel's request
22:49 RoanKattouw: Depooled wtp1004 so Gabriel can use it for benchmarking
22:46 mwalker: reverting payments to dced5328fd6ae4b395622abc41d2afa0ccad21c8 from 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 because of bug report that some people are not able to make donations due to 'required field city' being missing and then being unable to provide it
22:34 cmjohnson1: authdns update
22:15 cmjohnson1: auth-dns update
22:12 mutante: DNS update - remove status cname
21:56 cmjohnson1: taking wtp1001 down to replace w/new box
21:32 mwalker: pushing 4a3ca6cc36b9d78dbb015c1bf93bbafb14eb0799 to the payments cluster (again, because the instant reviewer, me, was broken)
21:23 mwalker: attempted deploy of payments cluster to new version, but then had to roll back to dced5328fd6ae4b395622abc41d2afa0ccad21c8
20:02 mutante: blocked en.wikivoyage from bugmenot
20:01 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_8708f.html
19:55 logmsgbot: dzahn synchronized ./docroot/wikivoyage.org/block_ab2c3.html
19:55 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php 'parser cache max age 30d'
19:37 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'returning db1043 to s1 watchlist service'
19:31 mutante: DNS update - add CNAME for status.wp/status.wm
19:28 logmsgbot: maxsem synchronized wmf-config/mobile.php 'https://gerrit.wikimedia.org/r/#/c/51291/'
19:13 mutante: install package upgrades on marmontel
19:06 mutante: install package upgrades on streber
19:04 mutante: create missing backup dir on streber to enable RT_Shredder plugin
19:04 Ryan_Lane: maxsem synchronized wmf-config/InitialiseSettings.php 'make upload endpoint for Commons local https://gerrit.wikimedia.org/r/#/c/51612/'
19:03 Ryan_Lane: LocalisationUpdate completed (1.21wmf10) at Fri Mar 1 02:31:13 UTC 2013
19:03 Ryan_Lane: csteipp synchronized php-1.21wmf10/includes
19:03 mutante: db9 - zipping old sql dumps of drupal and civirm to prevent full disk
19:03 Ryan_Lane: dzahn synchronized ./docroot/wikivoyage.org/mywotf83102384bfcd1e52152.html
19:03 Ryan_Lane: restarted glusterd service on labstore1-4
19:03 Ryan_Lane: pgehres Finished syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: pgehres Started syncing Wikimedia installation... : Updating i18n for DonationInterface-langonly
19:02 Ryan_Lane: binasher: shutting down mysql on db1043, converting to mariadb
19:02 LeslieCarr: powering back on spence
19:02 Ryan_Lane: asher synchronized wmf-config/db-eqiad.php 'pulling db1043 and shifting watchlist q's to db1050'
19:02 Ryan_Lane: pgehres synchronized php-1.21wmf10/extensions/DonationInterface/ 'Updating DonatonInterface-langonly'
19:02 Ryan_Lane: LeslieCarr: powering off spence
19:01 Ryan_Lane: asher synchronized wmf-config/lucene.php 'sending search-pool4 traffic to pmtpa, where the index/host distribution has been rebalanced'
19:01 Ryan_Lane: notpeter: upgrading mariadb boxes to 5.5.29
19:01 Ryan_Lane: tstarling synchronized wmf-config/InitialiseSettings.php
19:01 Ryan_Lane: Jeff_Green: stopped gmetad on neon for testing
19:01 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GettingStarted 'Update to split test (3/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/E3Experiments 'Update to split test (2/3)'
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/GuidedTour 'Update to split test (1/3)'
19:00 Ryan_Lane: mutante: create missing dump directory on streber, enables "RT-shredder" plugin which comes with RT since 3.8
19:00 Ryan_Lane: olivneh synchronized php-1.21wmf10/extensions/EventLogging 'Updates to JavaScript API'
19:00 Ryan_Lane: changing A record for wikitech.wm.o and changing labsconsole.wm.o into a cname to wikitech
18:59 Ryan_Lane: disabled editing on wikitech.wikimedia.org
18:59 Ryan_Lane: binasher: resharding search-pool4 in pmtpa - restarted lsearchd on all local nodes and indexers on searchidx2
18:59 mwalker: ... therefore the payments cluster is now running dced5328fd6ae4b395622abc41d2afa0ccad21c8
18:59 mutante: restart nagios-nrpe,gmond,apache,salt-minion on bastion hosts
18:59 Ryan_Lane: binasher: rebuilding db35 from a hotbackup of db55 (s5); testing a new build of mariadb 5.5.29
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:59 Ryan_Lane: reedy synchronized php-1.21wmf10/cache/interwiki.cdb 'Updating 1.21wmf10 interwiki cache'
18:58 Ryan_Lane: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance
18:58 Ryan_Lane: apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
18:58 mwalker: enabled sofort forms for CH and DE on the payments cluster
18:39 mutante: installing package upgrades on bast1001
18:37 mutante: installing package upgrades on fenari
17:40 ^demon: rescanned all trackingids for gerrit on manganese
17:31 ^demon: upgrading gerrit to 2.5.2-1506-g278aa9a on manganese/formey
17:07 Ryan_Lane: test

February 28

12:58 apergos: restarted icinga (not sure why it stopped, log file simply said 'caught TERM signal', in any case it complained that Error: Could not create external command file '/var/lib/nagios/rw/nagios.cmd' as named pipe )
06:28 apergos: stabbed searchpool4 and restarted lucene search etc gah hate hate hate
03:40 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:30 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
03:12 logmsgbot: reedy synchronized docroot
03:10 logmsgbot: reedy synchronized docroot
03:09 logmsgbot: reedy synchronized docroot
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 28 02:30:14 UTC 2013
01:10 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:58 RobH: dsh restarted eqiad apaches in progress
00:57 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:57 logmsgbot: robh gracefulled all apaches
00:54 logmsgbot: reedy synchronized wmf-config/
00:32 RobH: authdns-update to support wikinews.de
00:27 logmsgbot: reedy synchronized wmf-config/
00:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Lowering throttle to 5/day for new users, and'

February 27

23:26 Reedy: Running sync-common on mw1111. Reporting failed opening wmf-config/mc.php
22:57 mutante: rebooting zirconium (planet)
22:54 mutante: installing package upgrades on zirconium
22:52 logmsgbot: reedy synchronized wmf-config/
22:46 mutante: rebooting bast1001
22:36 mutante: installing package upgrades on bast1001
22:16 logmsgbot: reedy synchronized wmf-config/
21:44 RobH: rdb1001 & rdb1002 allocated, network setup, and ready for handoff, per rt4608
21:36 RobH: authdns-update for rdb1001-1002
21:36 RobH: authdns-update for sdb1001-1002
21:31 Tim: running resetUserTokens.php on all wikis, in screen on hume, to fix user_token field corruption (bug 41586)
21:26 logmsgbot: tstarling synchronized php-1.21wmf10/includes/User.php 'user_token fixes'
20:05 K4-713: updated payments cluster to fcbf9211c9836b
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf10
18:55 LeslieCarr: rebooting arsenic
18:42 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild message cache for EducationProgram
18:25 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild message cache for EducationProgram
17:57 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
17:51 K4-713: updated payments cluster to eab489d0889a
17:13 mutante: rebooting fenari for kernel upgrade
08:33 apergos: restarted lucene search on search1015
08:04 logmsgbot: reedy synchronized live-1.5/static-1.21wmf4/
08:04 logmsgbot: reedy synchronized docroot/bits/static-1.21wmf4/
07:58 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
07:05 Krinkle: done adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
07:04 logmsgbot: krinkle Finished syncing Wikimedia installation... :
06:29 apergos: snapshot1002 down for reinstallation
06:27 logmsgbot: krinkle Started syncing Wikimedia installation... :
06:15 Krinkle: adding 1.21wmf3, 1.21wmf4, 1.21wmf5 (bug 44570)
06:15 Krinkle: added 1.21wmf1, 1.21wmf2 (bug 44570)
05:35 apergos: restarted lucene search on search1016
04:57 logmsgbot: krinkle Finished syncing Wikimedia installation... :
04:37 logmsgbot: krinkle Started syncing Wikimedia installation... :
02:57 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 27 02:57:22 UTC 2013
02:31 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 27 02:31:06 UTC 2013
01:35 Krinkle: Bringing back checkouts of MediaWiki 1.21wmf5 - 1.21wmf1 on fenari for bug 44570
01:35 Jeff_Green: dist-upgrade and reboot grosley
00:25 logmsgbot: tstarling cleared profiling data

February 26

23:29 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:27 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/MobileFrontend/ 'https://gerrit.wikimedia.org/r/51053 https://gerrit.wikimedia.org/r/51057'
23:04 K4-713: synchronized payments cluster to 596b854787132c730
22:35 mutante: installing package upgrades on fenari
22:31 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
22:31 LeslieCarr: nagios is dead! spence.wikimedia.org if you want to access it directly
22:16 mutante: gracefulling eqiad Apaches to push redirect for wikivoyage.eu
22:15 logmsgbot: dzahn gracefulled all apaches
22:15 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
20:58 RobH: stat1 puppet runs fail due to dan account error, i am working on it, but i also want to run out and get something to eat, so leaving in broken state for 10 minutes until i return to work on it
20:21 mutante: DNS update - add wikiovyage.eu
19:34 Jeff_Green: package updates & reboots of payments hosts
19:13 paravoid: freezing all exim4 bounces to restore normal email delivery
18:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
17:54 paravoid: restarting opendj on sanger
17:45 AaronSchulz: Copying math files into ceph
15:14 mark: suspended i-000001d7 and i-0000039e for sending out malicious emails
15:06 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php
15:05 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php
12:26 apergos: temp blacklist live hack on sodium for exim4 to prevent some spam, see /etc/exim4/blacklist for that
11:20 logmsgbot: mlitn synchronized php-1.21wmf10/extensions/ArticleFeedback/api/ApiArticleFeedback.php 'Update ArticleFeedback to master'
06:53 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/
06:43 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:40 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:28 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:26 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
06:21 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:10 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:06 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:04 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
03:03 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 26 02:52:01 UTC 2013
02:46 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:44 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram/includes/pagers/ArticleTable.php
02:41 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:37 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:29 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 26 02:28:06 UTC 2013
02:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/EducationProgram
01:59 Ryan_Lane: test
01:59 Ryan_Lane: test
01:58 Ryan_Lane: test
01:40 K4-713: updated payments cluster to ce28b210374d8
00:20 mutante: mc1001,ms1004,mw1029,mw1059,mw1070,mw1157 - fixed puppet freshness
00:03 mutante: amssq33,amssq44 - fix more puppet runs

February 25

23:54 mutante: knsq26,knsq28 - deleting puppet lock file, fix puppet runs
23:45 logmsgbot: reedy synchronized wmf-config/
23:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable SearchExtraNS on commonswiki'
23:34 mutante: re-enabling disabled puppet on srv253,srv235
22:16 Ryan_Lane: restarting opendj/pdns on virt0
22:02 Reedy: Ran namespaceDupes.php on huwikt
21:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:28 logmsgbot: reedy synchronized docroot
20:27 logmsgbot: reedy synchronized wmf-config/
19:50 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:40 mutante: powercycling frozen niobium
19:32 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuilding message cache to update WikimediaMessages
19:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf10
18:54 mutante: wikivoyager.de and wikivoyager.org transferred and both redirect to wikivoyage.org
18:50 mutante: deleting foundation-news-l (deprecated, replaced by WikimediaAnnounce-l)
18:49 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
18:31 AaronSchulz: Copying all captchas into ceph
18:11 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
06:38 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 25 02:52:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Mon Feb 25 02:28:55 UTC 2013
01:15 mark: Powercycled strontium
01:07 paravoid: powercycling niobium (both unresponsive from the load)
01:06 paravoid: powercycling arsenic

February 24

06:28 apergos: restarted search1016 lucene
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 24 02:54:30 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sun Feb 24 02:29:29 UTC 2013

February 23

21:45 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb
21:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/interwiki.cdb
21:43 Reedy: mw110 is asking for a password
06:13 apergos: restarted lucene search on search1016
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 23 02:52:10 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Sat Feb 23 02:27:17 UTC 2013
00:43 RobH: lvs1003 had a ton of hung puppet processes (no output on them when tracing, so dunno whats up). killed them and kicked manual run

February 22

23:59 RobH: argon is shut down now
23:55 RobH: argon is going offline, killing our limesurvey install
23:29 paravoid: deploying squid config for upload's new transcoded container
23:26 paravoid: deploying squid
22:55 Ryan_Lane: changing dns entry for wikitech-static
22:41 RobH: mw110 throwing odd puppet errors, will come back to it later
22:41 RobH: sq41 reinstalled, throws endless puppet errors during redeployment
22:07 MaxSem: reindexing complete
21:39 RobH: harmon and copper online with OS installed for datadumps misc host support
21:39 MaxSem: Nuking GeoData Solr index and reindexing
21:38 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:35 logmsgbot: maxsem synchronized php-1.21wmf10/extensions/GeoData/solrupdate.php 'https://gerrit.wikimedia.org/r/#/c/50388/'
21:05 RobH: installing harmon and copper into dump generation misc use
20:17 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GuidedTour/modules/tours/gettingstarted.js 'Resume split test (3/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/GettingStarted/resources/ext.gettingstarted.js 'Resume split test (2/3)'
20:16 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments/experiments/openTask.js 'Resume split test (1/3)'
20:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:13 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/Math 'Updating Math extension to fix db caching behavior (Change-Id: I9b690e55001859c97fd40330272791d49ec6de75)'
20:09 RobH: authdns-update
19:58 RobH: sq41 boot order was fubar, fixed, reinstalling (as reboot into installer wiped mbr)
19:56 RobH: wtp1002-1004 os installed
19:34 Jeff_Green: start mysqldump of otrs database on db49
19:18 RobH: kicked ntpd manually on mw1045
19:09 RobH: sq41 reinstalling
18:57 RobH: mw112-125 back into pybal api pool
18:56 RobH: mw86-99 & mw111 back into pybal pool
18:49 logmsgbot: robh Finished syncing Wikimedia installation... :
18:37 logmsgbot: robh Started syncing Wikimedia installation... :
17:53 sbernardin: taking down sq41 to replace controller card.... https://rt.wikimedia.org/Ticket/Display.html?id=4550
16:24 mutante: tartupeedia.ee now redirects to et.wp portal page (bug 44893)
16:12 mutante: gracefulling eqiad Apaches via dsh to push tartupeedia.ee redirect (bug 44893)
16:11 logmsgbot: dzahn gracefulled all apaches
16:07 mutante: mw110 - sits at partioning screen in installer
15:47 hashar: gallium removed live hack in /srv/org/wikimedia/doc/puppetsource/manifests/role/labsmediawiki.pp
14:52 ^demon: restarted gerrit on manganese, jetty flapping again
14:18 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
13:14 mark: Halted ms-be3001-3003 for controller replacement
13:11 mark: Halted ms-be3004 for controller replacement
13:07 mark: Halted cp3009 for controller replacement
12:20 mark: Halted cp3010 for controller replacement
11:23 mutante: DNS update - add indiawikipedia.com
07:33 apergos: restarted lucene search on search1016
06:49 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:33 logmsgbot: tstarling synchronized docroot/bits/favicon/wikipedia.ico
05:20 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
05:18 logmsgbot: tstarling synchronized docroot/bits
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 22 02:53:05 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Fri Feb 22 02:29:09 UTC 2013
01:57 binasher: migrating wikivoyage and wikidata external blob stores to innodb
01:08 logmsgbot: aaron synchronized wmf-config/filebackend.php 'deployed f699c5a9ad2f780dc18863592dcae4a51ba6d841'

February 21

23:21 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/CodeEditor 'Syncing patch that disables background linting'
23:18 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
23:02 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, and GuidedTour
22:35 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/
22:33 logmsgbot: awjrichards synchronized php-1.21wmf10/extensions/MobileFrontend
22:12 logmsgbot: awjrichards synchronized wmf-config/mobile.php 'Add photo upload schema for event logging'
21:56 notpeter: running do-release-upgrade on dbs 52, 39, 51, and 35
21:49 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/EventLogging
21:48 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing 1 slave for s2-5 from db-secondary for upgrades and maria'
21:48 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:47 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/CodeEditor
21:37 notpeter: restarting mysql on db1047 to pick up conf changes
21:31 logmsgbot: olivneh synchronized php-1.21wmf10/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:23 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/SyntaxHighlight_GeSHi 'Disable highlighting of very large files.'
21:05 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php Enabling CodeEditor for Schema namespace on test2 and metawiki.
20:02 binasher: some wikis have myisam page_props tables. converting all to innodb via osc
18:39 Ryan_Lane: enabled Extension:OpenID as a provider on labsconsole
11:04 hashar: Ariel saved Gerrit by restarting it!
10:51 hashar: huge CPU spike on manganese (gerrit host)
10:06 hashar: Jenkins: reseting git branch specifier :-/
09:32 hashar: Jenkins: removing git branch specifier from all mediawiki-core jobs.
05:50 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 'shell cgroup'
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 21 02:54:17 UTC 2013
02:30 logmsgbot: asher synchronized wmf-config/db-pmtpa.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:30 logmsgbot: asher synchronized wmf-config/db-eqiad.php 'new extension1 shard (as an externalload config) - initially for AFTv5'
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Thu Feb 21 02:28:20 UTC 2013
00:53 RobH: issues with new mw servers 86+, took back out of pybal until i can troubleshoot
00:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
00:48 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMaintenance/
00:34 binasher: moved wikidatawiki tables on s3 to wikidatawiki_old to keep around for a time just in case; then "drop database wikidatawiki"
00:20 logmsgbot: asher synchronized wmf-config 'enabling wikidatawiki on shard s5'
00:18 logmsgbot: asher synchronized s5.dblist 'adding wikidatawiki'
00:17 logmsgbot: asher synchronized s3.dblist 'removing wikidatawiki'

February 20

22:48 RobH: authdns-update
22:35 binasher: importing wikidatawiki with schema changes into the s5 master, with sql_log_bin disabled
22:24 binasher: importing wikidatawiki with schema changes into the inactive pmtpa side of the s5 shard, with replication of the import just in pmtpa. ignore repl warnings for db{35,44,45,55}
21:48 RobH: authdns-update for wtp1002-1004
21:45 logmsgbot: maxsem synchronized wmf-config/CommonSettings.php 'Take pmtpa Solr servers out of rotation, increase load on master: https://gerrit.wikimedia.org/r/#/c/50056/'
21:41 RobH: ignore db66 errors, asher is working on it
21:25 binasher: running wikidata wb_items / wb_items_per_site schema migrations in parallel on the db66 slave (creating primary keys, cannot be done via online-schema-change)
21:24 RobH: kicked knsq23 to clear puppet freshness alarm
21:24 RobH: maerlant still offline (200 days?). dropped esams ticket for onsite troubleshooting, as mgmt is offline.
21:20 logmsgbot: asher synchronized wmf-config/InitialiseSettings.php 'setting wgReadOnly for wikidatawiki durring maintenance'
21:18 RobH: forced puppet run on lardner, was tired of it showing error for no runs
21:13 RobH: mw112-125 now in service as api apaches in tampa
20:58 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 RobH: mw86-mw111 tested ok, pushing into apache service in pmtpa
20:32 RobH: mw86-mw111 being added to nodelists, ignore errors on them for next few minutes
19:40 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikibooks to 1.21wmf10
19:37 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikiversity, wikisource and wikiquote to 1.21wmf10
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikivoyage, wikinews and wiktionary to 1.21wmf10
19:25 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, special, wikimedia, private and fishbowl to 1.21wmf10
05:34 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php 're-enable Scribunto profiler'
04:32 Tim: updating luasandbox to version 1.2, with profiler bug fixes
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Wed Feb 20 02:53:32 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 20 02:29:11 UTC 2013
01:28 RobH: mw1161-1208 have wrong power setting, changing all to system.power.hotspare.enable 0
01:00 notpeter: do-release-upgrade-ing on db58
00:57 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db58 from db-secondary for upgrades and maria, replacing db43'

February 19

23:47 notpeter: do-release-upgrade-ing on db43
23:38 logmsgbot: py synchronized wmf-config/db-pmtpa.php 'removing db43 from db-secondary for upgrades and maria'
23:21 logmsgbot: maxsem synchronized wmf-config 'https://gerrit.wikimedia.org/r/#/c/49946/'
23:04 logmsgbot: maxsem Finished syncing Wikimedia installation... : Weekly mobile deployment
23:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/
22:49 logmsgbot: maxsem Started syncing Wikimedia installation... : Weekly mobile deployment
22:41 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix MAX call with GREATEST'
22:29 RobH: sq41 offline, rt 4550
22:20 RobH: sq41 puppet freshness bad, wont accept ssh connections, serial redirection results in no output. will need to reboot it
22:17 RobH: sq41 not allowing ssh, drac resetting
22:15 sbernardin: taking down ms-be12 for hardware replacement per: https://rt.wikimedia.org/Ticket/Display.html?id=4546
22:15 RobH: sq79/80 puppet in locked state, manually restarted puppet, removed lock file, ran manually
22:11 logmsgbot: preilly synchronized php-1.21wmf9/extensions/ArticleFeedback 'fix mysql 5.5 / ansi sql incompatibility'
21:54 RobH: mw48 was set to puppet agent disabled, re-enabled and fired puppet run
20:04 logmsgbot: mflaschen synchronized . 'Syncing GuidedTour submodule omitted when new branch cut was first cloned.'
19:59 Ryan_Lane: disabling project storage volumes for all labs projects with no instances
15:02 hashar: restarting jenkins to enable some startup setting
14:58 hashar: manganese CPU spike ongoing. Started roughly at 14:40 UTC
13:29 mutante: wikimedia.jp.net now redirects to wikimedia.org | wikimediacommons.jp.net now redirects to commons.wm
13:27 mutante: DNS update - adding wikimedia.jp.net
13:25 logmsgbot: dzahn gracefulled all apaches
13:20 mutante: gracefulling eqiad Apaches via dsh to push redirect change for .jp.net domains
12:39 apergos: powercycling ms-be7 (unresponsive to ping or from console)
11:29 mutante: installing package upgrads on neon/icinga
02:49 logmsgbot: LocalisationUpdate completed (1.21wmf10) at Tue Feb 19 02:49:32 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 19 02:28:25 UTC 2013
01:12 logmsgbot: anomie synchronized php-1.21wmf10/includes/api/ApiQuerySiteinfo.php 'Backport fix for ugly API bug'
00:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
00:30 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php

February 18

23:51 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
23:51 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
23:46 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
23:25 Tim: on snapshot1002 -- fixed incorrect owner on some MW files, killed broken apt-get and upgraded wikimedia-task-appserver
23:24 Tim: fixed rsyncd.conf on scap proxies to allow connections from new apaches
23:14 logmsgbot: tstarling Finished syncing Wikimedia installation... :
23:11 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:45 logmsgbot: tstarling Finished syncing Wikimedia installation... :
22:42 logmsgbot: tstarling Started syncing Wikimedia installation... :
22:37 logmsgbot: tstarling synchronized php-1.21wmf9/extensions/Scribunto
22:37 logmsgbot: tstarling synchronized php-1.21wmf10/extensions/Scribunto
22:20 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikidatawiki to 1.21wmf10
20:37 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/ 'Scap is a liar'
20:31 logmsgbot: reedy Finished syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
20:15 logmsgbot: reedy Started syncing Wikimedia installation... : Rebuild l10n cache for wikidata related extensions
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Wikibase
19:59 logmsgbot: reedy synchronized php-1.21wmf10/extensions/Diff
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/DataValues
19:58 logmsgbot: reedy synchronized php-1.21wmf10/extensions/WikimediaMessages
19:44 logmsgbot: reedy synchronized php-1.21wmf10/cache/l10n/
19:19 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki and mediawikiwiki to 1.21wmf10
19:03 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
18:45 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache, and push it this time
17:54 logmsgbot: reedy Finished syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:53 logmsgbot: reedy Started syncing Wikimedia installation... : Build 1.21wmf10 message cache
17:42 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: test2wiki to 1.21wmf10
17:28 logmsgbot: reedy synchronized php-1.21wmf10 'Initial sync out'
17:03 logmsgbot: reedy synchronized wmf-config/
17:02 logmsgbot: reedy synchronized docroot
17:01 logmsgbot: reedy synchronized live-1.5/
12:33 mutante: wikiartpedia.(biz|co|info|me|mobi|net) activated - redirect to wikipedia.org (RT-4240)
12:30 mutante: adding wikiartpedia domains to DNS / authdns-update
12:28 logmsgbot: dzahn gracefulled all apaches
12:24 mutante: gracefulling eqiad Apaches via dsh for redirect.conf change
12:22 mutante: mw1045 - eth0 NO-CARRIER - check cable (RT-4545)
12:13 mutante: mw1041 - down - CPU Machine Chk error on mgmt - removing from dsh groups (RT-4517)
05:45 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:16 logmsgbot: tstarling synchronized wmf-config/db-eqiad.php
05:10 Tim: on db1043: running alter table to convert the relevant field to signed, since I figure this is less disruptive than sql_mode = 'NO_UNSIGNED_SUBTRACTION' and less likely to break again than skipping the query
04:58 Tim: replication broken on db1043, possibly others, due to update of article_feedback_revisions with negative value for afr_total
03:40 logmsgbot: tstarling synchronized wmf-config/CommonSettings.php
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 18 02:29:47 UTC 2013

February 17

02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 17 02:29:31 UTC 2013

February 16

13:38 apergos: restarted lucene-search on search1015 about 20 minutes ago
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 16 02:28:26 UTC 2013
01:47 logmsgbot: reedy synchronized php-1.21wmf9/
01:11 logmsgbot: reedy synchronized wmf-config/filebackend.php

February 15

23:56 RobH: authdns-update to correct mw110 reverse entry
23:28 Ryan_Lane: *modified them in LDAP
23:28 Ryan_Lane: modified project objects to indicate whether or not they are using gluster volumes for project and/or home storage
23:04 RobH: disregard mw1161 flaps, im working on it
20:15 Ryan_Lane: restarting opendj and pdns on virt0 and virt1000
20:15 Ryan_Lane: removed opendj from nfs1/2
19:07 RobH: no one sign mw86-mw125 puppet certs, im still working on them
18:23 RobH: mw87-mw125 installing
17:33 binasher: reimaging db1029-1031
16:55 mutante: removing mw1085 from dsh groups - broken hardware (RT-4453)
16:00 hashar: Enabled a linter for operations/apache-config . Any apache typo in operations/apache-config will publicly write your name on the typo wall of shame. See 49251 for implementation details and https://integration.mediawiki.org/ci/job/operations-apache-config-lint/ for some build examples.
14:31 mutante: wikimediafoundation.info activated to redirect .org
14:28 mutante: gracefulling Apaches in eqiad using dsh
14:25 logmsgbot: dzahn gracefulled all apaches
14:15 ^demon: [gerrit] updated replication plugin to 20c6fc7e (1.1-SNAPSHOT) which is back in line with upstream
08:02 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Echo 'Syncing Iab077a77aa75a6ff0cf03acecca38403595dda93 to fix exception 640c60c4'
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 15 02:30:05 UTC 2013
01:19 paravoid: deploying squid changes for s/knams/esams/ for knsq* squids (see git log for rationale)
01:09 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Enabling Schema NS on test2wiki'

February 14

22:49 RobH: mw1194 online and now in service as api apache
22:48 RobH: mw1200 added to api pool, online and in service
22:44 RobH: mw1189-1199 (sans 1194) now online as eqiad api apaches
22:37 logmsgbot: mflaschen Finished syncing Wikimedia installation... :
22:29 logmsgbot: mflaschen Started syncing Wikimedia installation... :
22:21 sbernardin: taking storage1 offline for decommissioning per: https://rt.wikimedia.org/Ticket/Display.html?id=4529
22:18 RobH: pushing mw1161-1188 into apache service (sans 1161/1164) from false to true in pybal, otherwise all online
22:16 logmsgbot: olivneh synchronized wmf-config/CommonSettings.php 'Updating for test2wiki'
22:14 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/Math
22:03 Ryan_Lane: replacing gluster upstarts on all labstore nodes with a regular init script
22:01 logmsgbot: reedy synchronized docroot
22:00 logmsgbot: olivneh synchronized wmf-config
21:49 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling PostEdit on {be|mediawiki|outreach}wikis'
21:45 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
21:39 sbernardin: taking down srv266 and srv268 for decommissioning ... https://rt.wikimedia.org/Ticket/Display.html?id=4534
21:31 ottomata1: restarted solr jetty on solr2
21:30 logmsgbot: bsitu Finished syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
21:25 ottomata: restarted solr jetty on solr3 and solr1003 for MaxSem
21:03 logmsgbot: bsitu Started syncing Wikimedia installation... : Update ArticleFeedbackv5, Echo, PageTriage to master
20:24 logmsgbot: kaldari synchronized wmf-config/CommonSettings.php 'updating CommonSettings.php for Echo'
20:15 apergos: rebooting mw1010 after some issues with OOM
19:22 RobH: new servers are NOT in pybal yet, and thus not getting traffic
19:22 RobH: just added mw1161-1200 into node lists for eventual service, some may throw sync script errors as I track them down
19:15 Ryan_Lane: rebooting labstore3
19:08 Ryan_Lane: rebooting labstore4
19:01 Ryan_Lane: rebooting labstore2
18:52 Ryan_Lane: rebooting labstore1
15:22 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Updating wikibase sort order'
15:22 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/client/includes/SortUtils.php 'Updating wikibase sort order'
14:52 mutante: re-enabled cp1044 in pybal - mobile varnish cache servers all on precise now
14:39 hashar: Jenkins: adding in several extension to Zuul configuration 49041
14:30 hashar: jenkins : updated all jobs using latest version jjb-config
13:35 mark: Fixed fixing
13:27 mutante: disabling cp1044 in pybal, reinstalling with precise
13:22 mutante: re-enabling cp1043 in pybal
13:11 mark: Upgraded varnish to latest version with updated range patch on cp1021-1028
12:16 mark: Inserted varnish 3.0.3plus~rc1-wm7 into the precise-wikimedia APT repository
12:14 mutante: stopping varnish backend on cp1043 and starting reinstall
12:10 mutante: disabling cp1043 in pybal
11:31 mutante: ms1004 - full disk, deleted large HTCPpurger.log file, fixed puppet runs
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 14 02:29:05 UTC 2013
01:17 andrewbogott: updated OpenStackManager to git head on virt0. Previous state is preserved (temporarily) in the local 'premerge' branch.
01:13 paravoid: removing mysql & apache from emery(!)
00:03 RobH: db29 os installed, handed off to peter

February 13

23:50 RobH: db29 reinstalling
23:48 paravoid: reinstalling sq48
23:22 RobH: db27 and db29 will be up and down for reinstallations
23:17 sbernardin: rebooting db29 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:17 sbernardin: rebooting db27 per https://rt.wikimedia.org/Ticket/Display.html?id=4528
23:11 RobH: db1013 online and puppet updated, needs to be pushed into some db cluster service
22:45 RobH: mw27 locked up, powercycling
22:36 notpeter: powercycling all eqiad jobrunners
22:13 RobH: adding mw1201-1208 to eqiad apache pool
21:44 Reedy: Running sync-common on mw1203
21:43 logmsgbot: reedy synchronized php-1.21wmf9 'resync to fix file permission errors'
21:32 Reedy: Running sync-common on mw1204
21:30 Reedy: Running sync-common on mw1202
21:27 RobH: db27 rebooting for reinstallation
21:27 RobH: db29 offline due to hardware issue, related ticket 4526
21:19 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:11 logmsgbot: csteipp synchronized wmf-config
21:09 logmsgbot: reedy cleared profiling data
20:25 logmsgbot: reedy synchronized wmf-config/CommonSettings.php 'Uncommenting enwiki from localClientDatabases'
19:59 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable Wikibase Client on enwiki'
19:47 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase/
19:30 logmsgbot: aaron rebuilt wikiversions.cdb and synchronized wikiversions files: Moved remaning wikis to 1.21wmf9
19:21 logmsgbot: aaron synchronized php-1.21wmf9/maintenance/runJobs.php 'deployed b03384f0ebd95e7f79638fb14ccd55da9c186d97'
19:12 logmsgbot: py gracefulled all apaches
18:59 cmjohnson1: mw1041 is borked and will be powered down and removed from network for h/w checks
18:27 logmsgbot: aaron synchronized php-1.21wmf9/extensions/TranslationNotifications 'deployed d7bf60b57f472e5f75101f58505f61842848f961'
16:26 Nemo_bis: Job queue weirdness and TranslationNotifications duplicates again noticed on Meta (bug 42715)
13:33 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
09:17 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
09:10 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48804/'
08:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/#/c/48797/'
08:25 hashar: jenkins: changing encryption key and regenerating secrets. See bug 44592
05:50 logmsgbot: tstarling synchronized multiversion/MWScript.php
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 13 02:52:39 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 13 02:28:06 UTC 2013
00:42 notpeter: restarting varnishncsa on cp1043
00:03 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend 'Touchhhhh'

February 12

23:54 logmsgbot: awjrichards synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/specials/donateimage.js 'touch file'
22:58 logmsgbot: maxsem Finished syncing Wikimedia installation... :
22:46 notpeter: disabling innodb-flush-log-at-trx-commit=2 on db1047 w/asher's blessing. will schedule downtime to apply
22:38 logmsgbot: maxsem Started syncing Wikimedia installation... :
22:04 notpeter: creating metrics user on all s1-7 masters
20:25 logmsgbot: reedy synchronized php-1.21wmf9/resources/mediawiki
20:22 binasher: flushed mobile varnish
20:21 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
20:19 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
20:06 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
20:05 binasher: flushed mobile varnish cache �:(
19:59 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend
19:57 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend
19:11 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Disable WikibaseClient on enwiki'
19:01 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Enable WikibaseClient on enwiki'
18:49 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:41 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:36 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:25 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
18:20 cmjohnson1: nas1001-a swapping mainboard
18:10 logmsgbot: aaron synchronized php-1.21wmf9/extensions/UploadWizard 'deployed 4548f9817d3d8025f4c19e279e9fb1f05b178087'
18:09 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Un-rollback Wikibase'
18:06 RoanKattouw: Reverted Wikibase update earlier, now reapplying
17:53 paravoid: powercycling niobium, SSH & console unresponsive
17:50 mark: Blocking modules=mobile.startup requests with 503 on bits varnish (earlier)
17:50 mark: Forced mw1149-1150 to healthy in bits.eqiad varnish
17:30 logmsgbot: catrope synchronized php-1.21wmf9/extensions/Wikibase 'Rolling back Wikibase'
16:18 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 16:18:31 UTC 2013
16:13 Jeff_Green: restarted apache on mw1152
16:01 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
16:00 Reedy: mw1161 and mw1165 are both missing mediawiki deployment files
15:58 logmsgbot: reedy synchronized php-1.21wmf9/includes/site/
15:54 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 15:54:11 UTC 2013
14:25 mutante: ran puppet on professor
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 12 02:52:42 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 12 02:28:53 UTC 2013
01:01 ^demon: stopped puppet and gerrit on manganese and formey
00:28 logmsgbot: aaron synchronized php-1.21wmf9/includes/db/Database.php 'deployed d8705542627f006a7ec9f81a9fb488fcc9a367bd'

February 11

23:19 RobH: unpooled mw1161-1188 from pybal config
22:03 logmsgbot: demon synchronized php-1.21wmf9/extensions/VisualEditor/modules/ve/init/mw/ve.init.mw.Platform.js 'Touching file'
21:08 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Wikibase client throwing fatals, disabling on enwiki pending fix'
21:04 cmjohnson1: auth-dns update
21:03 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'wikidata client on enwiki'
21:03 logmsgbot: demon synchronized wmf-config/CommonSettings.php 'wikidata client on enwiki'
20:30 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: huwiki and itwiki to 1.21wmf9
20:27 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: hewiki and itwiki to 1.21wmf9
20:22 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: enwiki to 1.21wmf9
20:11 logmsgbot: demon synchronized php-1.21wmf9/includes/site 'Updating sites code to pull in I8964a03c, I42221832'
18:39 RobH: disregard, pushing one to test, mw1161
18:36 RobH: mw1161-mw1188 being pushed into pybal pool as general apaches
16:58 logmsgbot: demon synchronized php-1.21wmf9/extensions/Wikibase/lib/includes/modules/SitesModule.php 'Touching file'
16:50 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'TorBlock to master, now with less annoying logs'
16:41 logmsgbot: demon synchronized wmf-config/InitialiseSettings.php 'Enable logging group for torblock'
16:40 logmsgbot: demon synchronized php-1.21wmf9/extensions/TorBlock 'Updating TorBlock to master'
16:27 cmjohnson1: commenting out srv226-234 on pybal/pmtpa
14:59 logmsgbot: hashar synchronized wmf-config/CommonSettings.php 'bug 44251 hardcode $wgDBuser = "wikiuser" 47074'
13:55 hashar: jenkins: regenerating all phpcs-HEAD jobs so they stop failing when no files need to be processed bug 44567 48450
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 11 02:52:39 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Mon Feb 11 02:27:44 UTC 2013

February 10

22:16 Tim: fixed syslog.eqiad.wmnet DNS entry, restarting rsyslogd on the eqiad apaches (RT 4500)
19:20 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'https://gerrit.wikimedia.org/r/#/c/48015/'
02:53 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 10 02:53:56 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sun Feb 10 02:28:46 UTC 2013

February 9

12:52 Reedy: mw1161 is missing /usr/local/apache/common-local
12:51 logmsgbot: reedy synchronized php-1.21wmf8/resources/
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 9 02:52:45 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Sat Feb 9 02:28:28 UTC 2013

February 8

23:48 logmsgbot: anomie synchronized php-1.21wmf8/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:48 logmsgbot: anomie synchronized php-1.21wmf9/extensions/Scribunto/engines/LuaCommon/LuaCommon.php
23:09 RoanKattouw: and on cerium too
23:08 RoanKattouw: Running puppet on titanium
20:20 RobH: disregard errors for mw1161-1164
20:13 RobH: mw1161-1164 offline for reallocation from job to general apache
20:10 cmjohnson1: decommissioning srvs226-234
19:43 RobH: mw1165-1188 os installed for apache use, initial puppet runs in progress
19:33 RobH: celsus back online
19:23 RobH: celsus rebooting to fix power mgmt cpu spike issue
19:02 RobH: mw1161-1164 in service as eqiad jobrunners
16:32 ottomata: updated vumi packages in apt to vumi_0.5.0~a+143-0~ppa3 and vumi-wikipedia_0.1~a+14-0~ppa3
16:09 mark: Upgraded and rebooted cr2-eqiad (both REs)
15:11 mark: Set cr2-eqiad VRRP prio to 50 to transition to backup role
15:10 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
15:08 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: srwikibooks back to 1.21wmf9 to test variant fixes
14:39 hashar: gallium installed php-luasandbox manually. Gerrit change is 48127.
05:47 logmsgbot: tstarling synchronized php-1.21wmf9/includes/context/RequestContext.php
02:54 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 8 02:54:49 UTC 2013
02:30 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Fri Feb 8 02:30:42 UTC 2013
02:02 logmsgbot: tstarling synchronized wmf-config/InitialiseSettings.php
01:48 LeslieCarr: cr2-knams is back up on the new code
01:36 LeslieCarr: cr2-knams reloading after upgrade
01:34 LeslieCarr: deactivating european bgp sessions - may cause some connectivity issues as connections reset
01:24 LeslieCarr: disabling notifications on icinga and nagios
01:03 logmsgbot: reedy synchronized php-1.21wmf9/languages/classes/
00:25 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/GettingStarted

February 7

23:31 logmsgbot: olivneh synchronized wmf-config/InitialiseSettings.php 'Enabling GuidedTour on additional wikis'
23:25 LeslieCarr: manually cleared arp and turned managmeent interface off and on of asw-c-eqiad to solve reachability issue
23:02 logmsgbot: mflaschen Finished syncing Wikimedia installation... : Deploy GuidedTour
22:47 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy GuidedTour
22:39 RobH: mw82 pooled into tampa apaches
22:04 ottomata: pulled power and rebooted amaranth toolserver for multichill. Its back online. multichill will notify whomever needs to do more work.
21:58 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/RelatedArticles/ 'Fixes 44761'
21:58 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/RelatedArticles/ 'Fixes 44761'
21:50 RobH: mw81-mw85 in pmtpa apache service
21:39 cmjohnson1: performing a hard reboot on amaranth
21:34 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/E3Experiments
21:33 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/E3Experiments
21:12 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/PostEdit
21:11 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/PostEdit
21:10 logmsgbot: olivneh synchronized php-1.21wmf8/extensions/EventLogging
21:09 logmsgbot: olivneh synchronized php-1.21wmf9/extensions/EventLogging
20:29 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: Being paranoid, node lists were in flux during last sync
20:23 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: All ku/sr wikis back to 1.21wmf8 - lang converter breakage
20:20 Nemo_bis: !Wikipedia and !Wikimedia projects are/were down, issue being investigated
20:14 logmsgbot: demon rebuilt wikiversions.cdb and synchronized wikiversions files: kuwiktionary back to 1.21wmf8
20:07 RobH: removing srv219-224 from rendering config in pmtpa
20:06 mark: Rebooting cr1-sdtpa
19:56 RobH: adding mw75-mw80 into tampa rendering pool
18:42 mark: Rebooted re0.cr1-eqiad
18:05 mark: Rebooting re1.cr1-eqiad
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:46 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
15:44 mark: Rebooting cr1-esams
15:40 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
15:39 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
14:54 mutante: granting RT group 'toolserver admins' the 'modify ticket' permission in queue esams
14:38 mutante: planet now updating hourly instead of daily
14:13 mutante: planet speaks SSL now for all languages. (it does not force http->https but it can if there is a demand)
12:00 Nemo_bis: amaranth dead/unreachable from Europe, ping dies at xe-5-3-1.cr2-eqiad.wikimedia.org (173.241.131.218)
10:08 hashar: jenkins : regenerating jobs to make phpcs to show the sniff codes being used. 47847
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Feb 7 02:52:26 UTC 2013
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Thu Feb 7 02:27:52 UTC 2013
02:23 logmsgbot: spage synchronized php-1.21wmf9/extensions/GuidedTour/modules/externals/mediawiki.libs.guiders/mediawiki.libs.guiders.submodule 'sync GuidedTour submodule omitted from 1.21wmf9'
01:57 RoanKattouw: Force-running puppet on wtp1 to figure out why it's broken
01:53 logmsgbot: krinkle synchronized wmf-config/InitialiseSettings.php 'I9fd98309'
00:41 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:20 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance
00:00 logmsgbot: reedy synchronized langlist

February 6

23:59 RobH: authdns-update to add min.wikipedia.org via addwiki automation
23:56 logmsgbot: reedy synchronized php-1.21wmf8/cache/interwiki.cdb 'Updating 1.21wmf8 interwiki cache'
23:55 logmsgbot: reedy synchronized php-1.21wmf9/cache/interwiki.cdb 'Updating 1.21wmf9 interwiki cache'
23:50 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:44 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:43 logmsgbot: reedy synchronized langlist
23:38 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:37 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:28 RobH: srv190-srv223 & srv225 should be offline, they are being decommissioned, can ignore nagios errors
23:26 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:24 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:17 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:13 RobH: specifically (i can spell, maybe)
23:13 RobH: removed from pybal specifially
23:12 cmjohnson1: removing srv192-218 and srv225 per rt 4438
23:07 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
23:05 logmsgbot: reedy synchronized php-1.21wmf9/extensions/WikimediaMaintenance/
21:36 cmjohnson1: srv193 is back online
21:21 cmjohnson1: srv193 power down to relocate to different rack
19:47 RobH: authdns-update
19:33 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wiktionary and wikibooks to 1.21wmf9
19:29 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikisource and wikiquote to 1.21wmf9
19:18 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: wikinews, wikivoyage and wikiversity to 1.21wmf9
19:12 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: special and wikimedia wikis to 1.21mwf9
19:11 cmjohnson1: powerdowing mw1 to reseat DIMM B1
19:04 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: closed, private and fishbowl wikis to 1.21wmf9
18:43 ^demon: running puppet on manganese, restarting gerrit
18:41 RobH: authdns-update
17:48 LeslieCarr: restarted morebots
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Wed Feb 6 02:51:36 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Feb 6 02:28:05 UTC 2013
02:05 logmsgbot: reedy synchronized wmf-config/
00:42 RobH: mailing lists returned to normal
00:41 RobH: done with data removal on mailing list server, rebuilding mbox
00:37 RobH: stopping mailman to scrub an entry out of archives, will restart shortly

February 5

23:29 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
23:26 logmsgbot: maxsem synchronized php-1.21wmf9/extensions/MobileFrontend/javascripts/modules/mf-photo.js 'https://gerrit.wikimedia.org/r/#/c/47670/'
22:44 logmsgbot: maxsem Finished syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:28 logmsgbot: maxsem Started syncing Wikimedia installation... : https://www.mediawiki.org/wiki/Extension:MobileFrontend/Deployments/2013-02-05
22:08 RobH: analytics1007 coming down for troubleshooting, disregard nagios errors
22:01 RobH: analytics1001 reinstalled, not puppet signed, back online
22:00 RobH: analytics1001 respects my authority.
21:33 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php 'Bug 44688 - Please disable MoodBar extension on English Wikipedia'
21:28 logmsgbot: robh synchronized wmf-config/InitialiseSettings.php
21:12 RobH: analytics1001 going down for troubleshooting/reinstallation
20:21 logmsgbot: kaldari Finished syncing Wikimedia installation... :
20:12 RobH: blog update sucessful, caching plugin update nonsuccessful, rolling it back to older plugin
20:07 logmsgbot: kaldari Started syncing Wikimedia installation... :
20:04 RobH: updating the blog software, here goes nothing
19:17 hashar: gallium : Zuul updated!
19:13 hashar: gallium : updating Zuul by cherry-picking a set of changes that let us fetch tags. ff79197..1854e32
17:05 cmjohnson1: authdns update
16:37 LeslieCarr: copied all equinix ashburn IPv4 peering sessions to their new /22 on cr2-eqiad
16:37 Jeff_Green: dist-upgrades for payments systems
16:36 Jeff_Green: restarted morebots :-P
05:37 Ryan_Lane: restarting opendj on all hosts
05:36 Ryan_Lane: remove virt1 replication agreement from all ldap servers
02:51 logmsgbot: LocalisationUpdate completed (1.21wmf9) at Tue Feb 5 02:51:10 UTC 2013
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Tue Feb 5 02:29:02 UTC 2013
01:07 logmsgbot: reedy synchronized php-1.21wmf9/extensions/TimedMediaHandler
00:51 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:39 logmsgbot: reedy synchronized php-1.21wmf8/extensions/ZeroRatedMobileAccess
00:29 logmsgbot: reedy synchronized php-1.21wmf9/extensions/ZeroRatedMobileAccess

February 4

23:49 logmsgbot: reedy synchronized multiversion/
23:46 logmsgbot: reedy synchronized multiversion/
23:45 logmsgbot: reedy synchronized wmf-config/
23:17 logmsgbot: reedy synchronized wmf-config/CommonSettings.php
21:02 hashar: gallium moved a git extension : mv /var/lib/git/rt-authen-externalauth.git /var/lib/git/operations/debs/
19:44 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
19:39 ottomata: manually gracefully reloaded eqiad apaches. See RT 4449
19:37 logmsgbot: root gracefulled all apaches
19:20 logmsgbot: reedy synchronized wmf-config/
19:16 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: testwiki, mediawikiwiki and wikidatawiki to 1.21wmf9
18:35 logmsgbot: reedy synchronized php-1.21wmf9/extensions/Wikibase
17:56 logmsgbot: reedy Finished syncing Wikimedia installation... : Build messages for 1.21wmf9
17:51 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:45 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:24 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:14 logmsgbot: reedy synchronized php-1.21wmf9/extensions/GuidedTour/
17:09 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
17:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files:
17:06 logmsgbot: reedy Started syncing Wikimedia installation... : Build messages for 1.21wmf9
16:45 logmsgbot: reedy synchronized php-1.21wmf9/ 'Initial sync'
16:22 logmsgbot: reedy synchronized wmf-config/
16:06 Ryan_Lane: force starting any glusterfs volume that has a downed process
15:12 Ryan_Lane: rebooting labstore1
15:11 Ryan_Lane: upgrading labstore1 to precise
14:33 apergos: powercycled ms-be3
14:32 Ryan_Lane: make that rebooting labstore3
14:32 Ryan_Lane: rebooting labstore2
14:07 Ryan_Lane: upgrading labstore3 to precise
11:14 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php 'cbd432f..82ed797 47393 Vector micro design on mlwiki, 29792 wikipedia translation on pawiki'
07:37 Ryan_Lane: upgrading labstore2 to precise
02:25 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Mon Feb 4 02:25:58 UTC 2013
00:42 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 3

20:58 hashar: jenkins : generating mwext-**-phpcs-HEAD jobs
17:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
17:42 logmsgbot: reedy synchronized wmf-config/
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sun Feb 3 02:28:29 UTC 2013
02:03 logmsgbot: reedy synchronized wmf-config/
02:02 logmsgbot: reedy synchronized images/sul/foundation.png
01:14 logmsgbot: reedy synchronized wmf-config/
00:48 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
00:22 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php

February 2

23:59 logmsgbot: reedy synchronized wmf-config/
23:47 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
23:34 logmsgbot: reedy synchronized wmf-config/
23:30 logmsgbot: reedy synchronized php-1.21wmf8/extensions/SearchExtraNS
23:03 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
22:35 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:02 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:54 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:39 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
16:38 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
09:23 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/PageImages/initImageData.php 'https://gerrit.wikimedia.org/r/#/c/44556/'
08:45 MaxSem: Running foreachwikiindblist wikipedia.dblist extensions/PageImages/initImageData.php --earlier-than=20121229000000
03:26 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Sat Feb 2 02:28:18 UTC 2013

February 1

23:02 K4-713: synchronized payments cluster to b421871b43822
20:00 RobH: authdns-update
19:43 RobH: authdns-update
14:45 cmjohnson1: authdns update "adding mw1161-1200 to eqiad mgmt and production zone files
12:04 cmjohnson1: authdns update adding db1051-60 to zone files
02:27 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Fri Feb 1 02:27:09 UTC 2013
01:50 K4-713: updated payments cluster to 45f8a6c6009a4bd7

January 31

23:14 andrewbogott: rebooting virt0 in a fit of optimism and/or desperation
22:44 logmsgbot: spage Finished syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:37 logmsgbot: spage Started syncing Wikimedia installation... : take 2: Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:26 logmsgbot: mflaschen Started syncing Wikimedia installation... : Deploy E3Experiments, GettingStarted, GuidedTour (new), MoodBar, and WikiEditor
22:20 ottomata: deployed tab delimiter log format change on squids, varnishes and nginxes
21:40 ottomata: reverted tab separartor change to squid
21:37 ottomata: deloyed tab separator log format changes to squids, merged the corresponding puppet changes to varnish and nginx
20:47 paravoid: deploying patched ganglia and removing .htaccess. Ganglia is public again!
14:59 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
14:58 notpeter: taking down db1047 for upgrade to precise
13:21 hashar: gallium: installed mercurial manually (puppet change is 46931 PS2)
13:19 hashar: gallium manually removed bzr: apt-get remove bzr python-keyring python-httplib2 python-launchpadlib python-zope.interface python-oauth python-bzrlib bzr python-simplejson python-configobj python-lazr.uri python-lazr.restfulclient python-wadllib
13:19 hashar: gallium: manually installed "bzr"
12:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase
10:24 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Remove strategyappswiki
02:29 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Thu Jan 31 02:29:47 UTC 2013
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Thu Jan 31 02:28:18 UTC 2013

January 30

22:40 binasher: deployed new text/upload squid confs with eqiad in proxySubnets
22:38 logmsgbot: reedy synchronized wmf-config/
22:23 logmsgbot: tstarling cleared profiling data
22:12 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/LocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
22:11 logmsgbot: aaron synchronized php-1.21wmf8/includes/filerepo/file/OldLocalFile.php 'deployed f884911f5562fc41980664c424e7037bde6ba110'
21:59 LeslieCarr: killing varnishhtcpd on cp3009 for realzies this time
21:58 RobH: mw23 offline, reinstalled with precise and signed into puppet, puppet throwing errors on initial run.
21:52 binasher: started udplog with correct mw config on fluorine
21:49 LeslieCarr: restarting varnishhtcpd on cp3010 - it appears to be locked up , netstat -l shows a huge Recv-Q for port 4827 and appears to be the previously discovered issue
21:43 binasher: restarting udp2log on fluorine
20:59 logmsgbot: reedy cleared profiling data
20:50 RobH: authdns-update
20:21 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
20:20 RobH: mw23 was stuck in installer, restarting and setting up.
20:09 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
19:45 logmsgbot: reedy synchronized wmf-config/ 'Enabling wikidata client on itwiki and hewiki'
19:42 ottomata: added updated udplog_1.8-5 .debs to apt.wikimedia.org. No real changes, these now include the packet-loss program in the package.
19:22 logmsgbot: reedy synchronized wmf-config/
19:12 logmsgbot: reedy synchronized php-1.21wmf8/extensions/Wikibase/
19:06 logmsgbot: reedy rebuilt wikiversions.cdb and synchronized wikiversions files: Everything else to 1.21wmf8
18:56 logmsgbot: reedy synchronized wmf-config/InitialiseSettings.php
18:48 Ryan_Lane: downgrading memcached to ubuntu version on virt0
17:18 Reedy: Created wb_changes_dispatch on wikidatawiki
16:55 Reedy: Created sites/site_identifier tables on hewiki and itwiki
16:34 cmjohnson1: mw1072 going to down to check h/w state currently failed rt4381
15:38 logmsgbot: hashar synchronized wmf-config/InitialiseSettings.php '(bug 44395) Allow bureaucrats to remove the translateadmin group on wikidatawiki 46011'
15:37 logmsgbot: hashar synchronized wmf-config/throttle.php '(bug 44506) raise throttle for an Israel editthon 46747'
12:54 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/GeoData
12:52 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/GeoData
05:55 Tim: rebooted kaulen, was not responding on network or serial console
03:11 mutante: wikimania.asia redirect now works, it did not before because of a bug in apache-graceful-all not actually restarting Apaches in eqiad
03:07 mutante: reloading Apaches in eqiad using manual dsh (because apache-graceful-all sanity check has a bug)
02:55 logmsgbot: catrope gracefulled all apaches
02:52 logmsgbot: LocalisationUpdate completed (1.21wmf7) at Wed Jan 30 02:52:03 UTC 2013
02:39 logmsgbot: dzahn gracefulled all apaches
02:28 logmsgbot: LocalisationUpdate completed (1.21wmf8) at Wed Jan 30 02:28:22 UTC 2013
00:44 logmsgbot: dzahn synchronized ./wmf-config/InitialiseSettings.php
00:43 mutante: sync InitialiseSettings.php for bug 44473
00:19 binasher: restarted pdns-recursor on dobson
00:13 logmsgbot: reedy synchronized wmf-config/

January 29

23:54 logmsgbot: reedy synchronized wmf-config/
23:21 logmsgbot: pgehres synchronized php-1.21wmf8/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:17 logmsgbot: pgehres synchronized php-1.21wmf7/extensions/CentralNotice/ 'Updating CentralNotice to master, resolving namespace issues'
23:00 logmsgbot: maxsem synchronized wmf-config/InitialiseSettings.php 'Enable Special:Nearby everywhere'
22:51 logmsgbot: maxsem synchronized php-1.21wmf7/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:48 logmsgbot: maxsem synchronized php-1.21wmf8/extensions/MobileFrontend 'https://gerrit.wikimedia.org/r/46630'
22:47 Tim: rotated SAL wikitech page - test logging

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s

@@ Line 1: / Line 1: @@
 == May 23 ==
+* 21:23 logmsgbot: bsitu synchronized wmf-config/InitialiseSettings.php  'Enable echo email bundling on enwiki'
 * 20:31 logmsgbot: spage synchronized php-1.22wmf4/extensions/GettingStarted  'E3 latest GettingStarted'
 * 20:29 logmsgbot: spage synchronized php-1.22wmf4/extensions/GuidedTour  'E3 latest GuidedTour'