Jump to content

Server Admin Log/Archive 78

From Wikitech


2024-04-30

  • 23:04 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7014.magru.wmnet with OS bullseye
  • 23:04 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 22:58 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7013.magru.wmnet with OS bullseye
  • 22:56 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 22:35 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7013.magru.wmnet with reason: host reimage
  • 22:33 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7014.magru.wmnet with reason: host reimage
  • 22:32 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7013.magru.wmnet with reason: host reimage
  • 22:30 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7014.magru.wmnet with reason: host reimage
  • 22:18 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1005.eqiad.wmnet with OS bullseye
  • 22:05 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7013.magru.wmnet with OS bullseye
  • 22:04 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7014.magru.wmnet with OS bullseye
  • 22:02 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage
  • 21:56 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage
  • 21:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cephadm1001.eqiad.wmnet
  • 21:50 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:50 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cephadm1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 21:49 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cephadm1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 21:37 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1005.eqiad.wmnet with OS bullseye
  • 21:33 mutante: grafana2001 - sudo -u loki /usr/bin/loki -config.file=/etc/loki/loki-local-config.yaml in an attempt to debug issue on grafana-next.wikimedia.org
  • 21:18 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1004.eqiad.wmnet with OS bullseye
  • 21:06 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS bookworm
  • 21:03 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1004.eqiad.wmnet with reason: host reimage
  • 20:58 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1004.eqiad.wmnet with reason: host reimage
  • 20:56 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 20:55 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7015.magru.wmnet with OS bullseye
  • 20:55 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 20:54 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 20:51 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts cephadm1001.eqiad.wmnet
  • 20:50 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dns7002.wikimedia.org with reason: reimaged again
  • 20:50 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dns7002.wikimedia.org with reason: reimaged again
  • 20:49 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS bookworm
  • 20:46 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 20:43 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 20:40 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lists1004.wikimedia.org with OS bookworm
  • 20:40 aokoth@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - aokoth@cumin1002"
  • 20:39 cjming: end of UTC late backport window
  • 20:39 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1004.eqiad.wmnet with OS bullseye
  • 20:38 cjming@deploy1002: Finished scap: Backport for Deploy a11y settings to testwiki (T362147) (duration: 21m 00s)
  • 20:38 aokoth@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - aokoth@cumin1002"
  • 20:37 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7016.magru.wmnet with OS bullseye
  • 20:37 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 20:36 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 20:31 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7015.magru.wmnet with reason: host reimage
  • 20:30 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS bookworm
  • 20:28 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS bookworm
  • 20:27 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7015.magru.wmnet with reason: host reimage
  • 20:26 cjming@deploy1002: ksarabia and cjming: Continuing with sync
  • 20:21 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lists1004.wikimedia.org with reason: host reimage
  • 20:20 cjming@deploy1002: ksarabia and cjming: Backport for Deploy a11y settings to testwiki (T362147) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:18 aokoth@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on lists1004.wikimedia.org with reason: host reimage
  • 20:17 cjming@deploy1002: Started scap: Backport for Deploy a11y settings to testwiki (T362147)
  • 20:16 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
  • 20:13 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7016.magru.wmnet with reason: host reimage
  • 20:11 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dns7002.wikimedia.org with reason: host reimage
  • 20:10 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7016.magru.wmnet with reason: host reimage
  • 20:10 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 20:07 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 20:03 aokoth@cumin1002: START - Cookbook sre.hosts.reimage for host lists1004.wikimedia.org with OS bookworm
  • 20:01 aokoth@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lists1004.wikimedia.org with OS bookworm
  • 19:59 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7015.magru.wmnet with OS bullseye
  • 19:55 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bookworm
  • 19:53 brett@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host ncredir1001.eqiad.wmnet with OS bookworm
  • 19:46 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7008.magru.wmnet with OS bullseye
  • 19:46 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:45 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:45 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7006.magru.wmnet with OS bullseye
  • 19:45 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:44 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7016.magru.wmnet with OS bullseye
  • 19:44 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:41 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7007.magru.wmnet with OS bullseye
  • 19:41 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:40 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 19:38 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm
  • 19:32 sukhe: sudo ipmitool -I lanplus -H "dns7002.mgmt.magru.wmnet" -U root -E chassis power cycle
  • 19:29 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7003.magru.wmnet with OS bullseye
  • 19:27 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7002.wikimedia.org with OS bookworm
  • 19:22 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bookworm
  • 19:19 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7008.magru.wmnet with reason: host reimage
  • 19:16 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7006.magru.wmnet with reason: host reimage
  • 19:15 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7008.magru.wmnet with reason: host reimage
  • 19:15 aokoth@cumin1002: START - Cookbook sre.hosts.reimage for host lists1004.wikimedia.org with OS bookworm
  • 19:14 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7007.magru.wmnet with reason: host reimage
  • 19:14 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7006.magru.wmnet with reason: host reimage
  • 19:12 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7007.magru.wmnet with reason: host reimage
  • 19:08 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lists1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 19:04 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 19:01 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 18:59 bking@deploy1002: helmfile [staging] DONE helmfile.d/services/rdf-streaming-updater: apply
  • 18:57 sukhe@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host dns7001
  • 18:56 sukhe@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host dns7001
  • 18:55 bking@deploy1002: helmfile [staging] START helmfile.d/services/rdf-streaming-updater: apply
  • 18:53 herron: updated alertmanager IRC alert text format. for details please see https://gerrit.wikimedia.org/r/c/operations/puppet/+/1019840
  • 18:49 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host dns7002.wikimedia.org with OS bookworm
  • 18:48 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7008.magru.wmnet with OS bullseye
  • 18:47 sukhe: sudo cumin -b1 -s10 'C:confd and *.esams.wmnet' 'systemctl restart confd'
  • 18:46 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7006.magru.wmnet with OS bullseye
  • 18:46 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7007.magru.wmnet with OS bullseye
  • 18:41 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) _etcd._tcp.esams.wmnet on all recursors
  • 18:41 sukhe@cumin1002: START - Cookbook sre.dns.wipe-cache _etcd._tcp.esams.wmnet on all recursors
  • 18:40 sukhe: running authdns-update for CR 1025800
  • 18:38 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp7014']
  • 18:38 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7014']
  • 18:32 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7003.magru.wmnet with OS bullseye
  • 18:31 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001.magru.wmnet']
  • 18:31 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001.magru.wmnet']
  • 18:31 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001.magru.wmnet']
  • 18:31 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001.magru.wmnet']
  • 18:30 aokoth@cumin1002: START - Cookbook sre.hosts.provision for host lists1004.mgmt.eqiad.wmnet with reboot policy FORCED
  • 18:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7014.magru.wmnet with OS bullseye
  • 18:21 xcollazo@deploy1002: Finished deploy [analytics/refinery@4836095] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4836095f] (duration: 02m 52s)
  • 18:18 aokoth@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lists1004
  • 18:18 xcollazo@deploy1002: Started deploy [analytics/refinery@4836095] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@4836095f]
  • 18:18 xcollazo@deploy1002: Finished deploy [analytics/refinery@4836095] (thin): Regular analytics weekly train THIN [analytics/refinery@4836095f] (duration: 03m 57s)
  • 18:17 aokoth@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host lists1004
  • 18:16 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:16 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge pending changes - sukhe@cumin1002"
  • 18:15 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: merge pending changes - sukhe@cumin1002"
  • 18:14 xcollazo@deploy1002: Started deploy [analytics/refinery@4836095] (thin): Regular analytics weekly train THIN [analytics/refinery@4836095f]
  • 18:13 xcollazo@deploy1002: Finished deploy [analytics/refinery@4836095]: Regular analytics weekly train [analytics/refinery@4836095f] (duration: 16m 16s)
  • 18:13 sukhe@cumin1002: START - Cookbook sre.dns.netbox
  • 18:12 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7014.magru.wmnet with OS bullseye
  • 18:11 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7014.magru.wmnet with OS bullseye
  • 18:11 sukhe@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns7001.wikimedia.org with OS bookworm
  • 18:11 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7015.magru.wmnet with OS bullseye
  • 18:09 sukhe: running cookbook -d sre.dns.netbox "test"
  • 18:06 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7015.magru.wmnet with OS bullseye
  • 18:04 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7014.magru.wmnet with OS bullseye
  • 18:03 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "running manually for cp7013 - sukhe@cumin1002"
  • 18:03 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host dns7001.wikimedia.org with OS bookworm
  • 18:02 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "running manually for cp7013 - sukhe@cumin1002"
  • 17:57 xcollazo@deploy1002: Started deploy [analytics/refinery@4836095]: Regular analytics weekly train [analytics/refinery@4836095f]
  • 17:56 swfrench@deploy1002: Unlocked for deployment [ALL REPOSITORIES]: etcd replication maintenance - T358636 (duration: 55m 11s)
  • 17:54 swfrench-wmf: putting etcd back in read-write mode for T358636
  • 17:09 swfrench-wmf: disabling etcd replication into conf2005 for T358636
  • 17:03 swfrench-wmf: putting etcd in read-only mode for T358636
  • 17:01 swfrench@deploy1002: Locking from deployment [ALL REPOSITORIES]: etcd replication maintenance - T358636
  • 16:56 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7004.magru.wmnet with OS bullseye
  • 16:55 ejegg: payments-wiki upgraded from c7ab847d to c4f43931
  • 16:53 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7013.magru.wmnet with OS bullseye
  • 16:53 sukhe@cumin1002: END (FAIL) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=99) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 16:53 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 16:52 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7005.magru.wmnet with OS bullseye
  • 16:52 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 16:51 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 16:36 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7002.magru.wmnet with OS bullseye
  • 16:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7012.magru.wmnet with OS bullseye
  • 16:33 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 16:30 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 16:30 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7013.magru.wmnet with reason: host reimage
  • 16:27 fabfur@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp7003.magru.wmnet with OS bullseye
  • 16:27 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7004.magru.wmnet with reason: host reimage
  • 16:25 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7013.magru.wmnet with reason: host reimage
  • 16:25 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7004.magru.wmnet with reason: host reimage
  • 16:24 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7005.magru.wmnet with reason: host reimage
  • 16:22 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1003.eqiad.wmnet with OS bullseye
  • 16:22 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7005.magru.wmnet with reason: host reimage
  • 16:18 stevemunene@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 16:18 stevemunene@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 16:16 elukey@puppetmaster1001: conftool action : set/pooled=true; selector: dnsdisc=inference,name=codfw
  • 16:16 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore1*.eqiad.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 16:12 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 16:10 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7002.magru.wmnet with reason: host reimage
  • 16:09 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
  • 16:08 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host dns7001.wikimedia.org with OS bullseye
  • 16:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7012.magru.wmnet with reason: host reimage
  • 16:06 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: sync on main
  • 16:05 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 16:05 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7002.magru.wmnet with reason: host reimage
  • 16:04 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7012.magru.wmnet with reason: host reimage
  • 16:02 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1003.eqiad.wmnet with reason: host reimage
  • 15:59 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7013.magru.wmnet with OS bullseye
  • 15:58 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1003.eqiad.wmnet with reason: host reimage
  • 15:58 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore1*.eqiad.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 15:57 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore200[5,6].codfw.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 15:56 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7004.magru.wmnet with OS bullseye
  • 15:56 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7005.magru.wmnet with OS bullseye
  • 15:44 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore200[5,6].codfw.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 15:40 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching sessionstore2004.codfw.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 15:39 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7012.magru.wmnet with OS bullseye
  • 15:38 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1003.eqiad.wmnet with OS bullseye
  • 15:36 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7003.magru.wmnet with OS bullseye
  • 15:36 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7002.magru.wmnet with OS bullseye
  • 15:33 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching sessionstore2004.codfw.wmnet: Move to PKI Truststore - elukey@cumin1002
  • 15:32 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 15:32 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 15:31 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host dns7001.wikimedia.org with OS bullseye
  • 15:28 elukey: move Cassandra instances on session store nodes to a new Java Truststore to support PKI - T352647
  • 15:27 elukey: depool liftwing codfw for a couple of hours to test eqiad capabilities to handle the traffic
  • 15:27 elukey@puppetmaster1001: conftool action : set/pooled=false; selector: dnsdisc=inference,name=codfw
  • 15:14 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 15:14 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 15:11 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7011.magru.wmnet with OS bullseye
  • 15:11 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 15:09 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 15:06 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['mw2382']
  • 15:06 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['mw2382']
  • 15:05 jhancock@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host mw2382.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 15:03 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1002.eqiad.wmnet with OS bullseye
  • 14:59 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7010.magru.wmnet with OS bullseye
  • 14:59 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 14:58 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7004.magru.wmnet with OS bookworm
  • 14:54 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 14:51 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 14:50 sukhe@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dns7001.wikimedia.org with OS bullseye
  • 14:49 jhancock@cumin2002: START - Cookbook sre.hosts.provision for host mw2382.mgmt.codfw.wmnet with reboot policy GRACEFUL
  • 14:45 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7011.magru.wmnet with reason: host reimage
  • 14:45 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ncredir3004.esams.wmnet
  • 14:45 vgutierrez@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ncredir3003.esams.wmnet
  • 14:43 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7001.magru.wmnet with OS bullseye
  • 14:43 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 14:42 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 14:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1002.eqiad.wmnet with reason: host reimage
  • 14:41 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7011.magru.wmnet with reason: host reimage
  • 14:39 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1002.eqiad.wmnet with reason: host reimage
  • 14:35 vgutierrez@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir3003.esams.wmnet
  • 14:34 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7010.magru.wmnet with reason: host reimage
  • 14:34 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7009.magru.wmnet with OS bullseye
  • 14:34 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 14:33 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 14:31 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7004.magru.wmnet with reason: host reimage
  • 14:31 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7010.magru.wmnet with reason: host reimage
  • 14:31 bblack@cumin1002: conftool action : set/pooled=yes; selector: name=ncredir3003.esams.wmnet
  • 14:31 bblack@cumin1002: conftool action : set/pooled=no; selector: name=ncredir3003.esams.wmnet
  • 14:29 moritzm: installing gnutls28 security updates on buster
  • 14:27 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7004.magru.wmnet with reason: host reimage
  • 14:21 moritzm: installing Java 8 security updates
  • 14:21 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7004.magru.wmnet with reason: will be reimaged soon
  • 14:21 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7004.magru.wmnet with reason: will be reimaged soon
  • 14:19 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1002.eqiad.wmnet with OS bullseye
  • 14:19 dcausse@deploy1002: Finished deploy [airflow-dags/search@ab19bcd]: wdqs: deduplicate side-output events (T362508) (duration: 00m 29s)
  • 14:19 dcausse@deploy1002: Started deploy [airflow-dags/search@ab19bcd]: wdqs: deduplicate side-output events (T362508)
  • 14:16 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7001.magru.wmnet with reason: host reimage
  • 14:15 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7011.magru.wmnet with OS bullseye
  • 14:13 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7001.magru.wmnet with reason: host reimage
  • 14:10 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 14:10 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 14:10 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7009.magru.wmnet with reason: host reimage
  • 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp[7002-7003].magru.wmnet with reason: will be reimaged soon
  • 14:07 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7009.magru.wmnet with reason: host reimage
  • 14:06 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp[7002-7003].magru.wmnet with reason: will be reimaged soon
  • 14:06 urbanecm@deploy1002: Finished scap: Backport for Turn on ParserMigration extension everywhere, Quiet ParserMigration notice for 30 days after acknowledgement (duration: 23m 40s)
  • 14:03 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7010.magru.wmnet with OS bullseye
  • 13:54 urbanecm@deploy1002: urbanecm and cscott: Continuing with sync
  • 13:54 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti7004.magru.wmnet with OS bookworm
  • 13:53 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7003.magru.wmnet with OS bookworm
  • 13:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 13:51 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host dns7001.wikimedia.org with OS bullseye
  • 13:47 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 13:47 urbanecm@deploy1002: urbanecm and cscott: Backport for Turn on ParserMigration extension everywhere, Quiet ParserMigration notice for 30 days after acknowledgement synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:44 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1233.eqiad.wmnet
  • 13:43 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7001.magru.wmnet with OS bullseye
  • 13:42 urbanecm@deploy1002: Started scap: Backport for Turn on ParserMigration extension everywhere, Quiet ParserMigration notice for 30 days after acknowledgement
  • 13:42 urbanecm@deploy1002: Finished scap: Backport for [itwiki] Create a new 'arbcom' usergroup (T363805) (duration: 20m 09s)
  • 13:39 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7009.magru.wmnet with OS bullseye
  • 13:35 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1233.eqiad.wmnet
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1222.eqiad.wmnet
  • 13:35 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7009.magru.wmnet with OS bullseye
  • 13:28 urbanecm@deploy1002: superpes and urbanecm: Continuing with sync
  • 13:27 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1222.eqiad.wmnet
  • 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7003.magru.wmnet with reason: host reimage
  • 13:26 urbanecm@deploy1002: superpes and urbanecm: Backport for [itwiki] Create a new 'arbcom' usergroup (T363805) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:26 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1197.eqiad.wmnet
  • 13:26 sfaci@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/mpic-next: apply
  • 13:26 sfaci@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
  • 13:25 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 13:25 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 13:23 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7003.magru.wmnet with reason: host reimage
  • 13:22 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 13:22 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db2114.codfw.wmnet onto db1125.eqiad.wmnet
  • 13:21 urbanecm@deploy1002: Started scap: Backport for [itwiki] Create a new 'arbcom' usergroup (T363805)
  • 13:21 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:20 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:19 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1197.eqiad.wmnet
  • 13:15 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:15 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:12 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:12 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:11 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:11 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:03 arnaudb@cumin1002: END (FAIL) - Cookbook sre.mysql.copy (exit_code=99) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:03 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 13:02 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7009.magru.wmnet with OS bullseye
  • 12:58 moritzm: installing util-linux security updates
  • 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.copy (exit_code=0) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 12:58 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.copy (exit_code=0) Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 12:58 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1125.eqiad.wmnet onto db2114.codfw.wmnet
  • 12:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1188.eqiad.wmnet
  • 12:55 moritzm: uploaded openjdk-8 8u412-ga-1~deb11u1 to bullseye-wikimedia (forward port of latest Java 8 security updates)
  • 12:51 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti7003.magru.wmnet with OS bookworm
  • 12:50 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7002.magru.wmnet with OS bookworm
  • 12:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 12:46 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 12:45 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/mpic-next: apply
  • 12:25 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7002.magru.wmnet with reason: host reimage
  • 12:24 jforrester@deploy1002: Finished deploy [integration/docroot@b88f9e1]: Update VisualEditor links, post-JSDoc (b88f9e1674) (duration: 00m 06s)
  • 12:24 jforrester@deploy1002: Started deploy [integration/docroot@b88f9e1]: Update VisualEditor links, post-JSDoc (b88f9e1674)
  • 12:23 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7002.magru.wmnet with reason: host reimage
  • 12:20 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1182.eqiad.wmnet
  • 12:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1162.eqiad.wmnet
  • 12:09 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:09 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:01 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1162.eqiad.wmnet
  • 11:50 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti7002.magru.wmnet with OS bookworm
  • 11:47 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ganeti7001.magru.wmnet with OS bookworm
  • 11:47 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 11:43 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd1001.eqiad.wmnet
  • 11:39 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jmm@cumin2002"
  • 11:37 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host cephosd1001.eqiad.wmnet
  • 11:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61495 and previous config saved to /var/cache/conftool/dbconfig/20240430-113640-root.json
  • 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61494 and previous config saved to /var/cache/conftool/dbconfig/20240430-112135-root.json
  • 11:19 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2207.codfw.wmnet
  • 11:19 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
  • 11:16 jmm@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ganeti7001.magru.wmnet with reason: host reimage
  • 11:08 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 11:07 jiji@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 11:07 jiji@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 11:06 jiji@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 11:06 jiji@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61493 and previous config saved to /var/cache/conftool/dbconfig/20240430-110629-root.json
  • 11:06 jiji@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 11:04 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2207.codfw.wmnet
  • 11:03 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2204.codfw.wmnet
  • 10:54 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2204.codfw.wmnet
  • 10:54 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1001.eqiad.wmnet with OS bullseye
  • 10:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2189.codfw.wmnet
  • 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61492 and previous config saved to /var/cache/conftool/dbconfig/20240430-105124-root.json
  • 10:48 Dreamy_Jazz: Security deploy finished
  • 10:47 logmsgbot: dreamyjazz Deployed security patch for T338419
  • 10:45 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2189.codfw.wmnet
  • 10:43 jmm@cumin2002: START - Cookbook sre.hosts.reimage for host ganeti7001.magru.wmnet with OS bookworm
  • 10:39 aokoth@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lists1004.eqiad.wmnet
  • 10:39 aokoth@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:39 aokoth@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lists1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
  • 10:38 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2175.codfw.wmnet
  • 10:38 aokoth@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lists1004.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - aokoth@cumin1002"
  • 10:37 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61491 and previous config saved to /var/cache/conftool/dbconfig/20240430-103618-root.json
  • 10:35 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 10:33 logmsgbot: dreamyjazz Deployed security patch for T338419
  • 10:32 aokoth@cumin1002: START - Cookbook sre.dns.netbox
  • 10:27 aokoth@cumin1002: START - Cookbook sre.hosts.decommission for hosts lists1004.eqiad.wmnet
  • 10:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2175.codfw.wmnet
  • 10:24 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2148.codfw.wmnet
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61490 and previous config saved to /var/cache/conftool/dbconfig/20240430-102113-root.json
  • 10:16 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1001.eqiad.wmnet with OS bullseye
  • 10:15 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2148.codfw.wmnet
  • 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2138.codfw.wmnet
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1158 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61489 and previous config saved to /var/cache/conftool/dbconfig/20240430-100607-root.json
  • 10:00 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2138.codfw.wmnet
  • 09:58 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2125.codfw.wmnet
  • 09:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T360332)', diff saved to https://phabricator.wikimedia.org/P61488 and previous config saved to /var/cache/conftool/dbconfig/20240430-095745-arnaudb.json
  • 09:53 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'sync'.
  • 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'Push es6 codfw config T355424', diff saved to https://phabricator.wikimedia.org/P61487 and previous config saved to /var/cache/conftool/dbconfig/20240430-095119-marostegui.json
  • 09:47 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2125.codfw.wmnet
  • 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Push es6 eqiad section T355285', diff saved to https://phabricator.wikimedia.org/P61486 and previous config saved to /var/cache/conftool/dbconfig/20240430-094635-marostegui.json
  • 09:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P61485 and previous config saved to /var/cache/conftool/dbconfig/20240430-094237-arnaudb.json
  • 09:41 jayme@cumin1002: conftool action : set/pooled=inactive; selector: name=mw2382.codfw.wmnet
  • 09:39 Dreamy_Jazz: Starting security deploy on tmux session
  • 09:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1158.eqiad.wmnet with OS bookworm
  • 09:30 marostegui@deploy1002: Finished scap: Backport for etcd.php: Add es6 (T355285 T355424) (duration: 15m 01s)
  • 09:29 jiji@deploy1002: helmfile [codfw] START helmfile.d/admin 'sync'.
  • 09:29 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:28 jiji@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:28 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on mw2382.codfw.wmnet with reason: Degraded RAID/storage controller issues
  • 09:28 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on mw2382.codfw.wmnet with reason: Degraded RAID/storage controller issues
  • 09:28 jiji@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 09:27 jiji@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 09:27 jiji@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:27 jiji@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 09:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179', diff saved to https://phabricator.wikimedia.org/P61484 and previous config saved to /var/cache/conftool/dbconfig/20240430-092729-arnaudb.json
  • 09:26 jayme: draining mw2382.codfw.wmnet - T362938
  • 09:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 100%: Post replag', diff saved to https://phabricator.wikimedia.org/P61483 and previous config saved to /var/cache/conftool/dbconfig/20240430-092230-arnaudb.json
  • 09:18 volans@cumin1002: END (PASS) - Cookbook sre.deploy.python-code (exit_code=0) netbox to netbox-dev2002.codfw.wmnet with reason: Update Netbox dependencies for netbox-next - volans@cumin1002
  • 09:17 marostegui@deploy1002: marostegui: Continuing with sync
  • 09:17 marostegui@deploy1002: marostegui: Backport for etcd.php: Add es6 (T355285 T355424) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T360332)', diff saved to https://phabricator.wikimedia.org/P61482 and previous config saved to /var/cache/conftool/dbconfig/20240430-091556-arnaudb.json
  • 09:15 marostegui@deploy1002: Started scap: Backport for etcd.php: Add es6 (T355285 T355424)
  • 09:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2179 (T360332)', diff saved to https://phabricator.wikimedia.org/P61481 and previous config saved to /var/cache/conftool/dbconfig/20240430-091221-arnaudb.json
  • 09:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage
  • 09:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1157 (T352010)', diff saved to https://phabricator.wikimedia.org/P61480 and previous config saved to /var/cache/conftool/dbconfig/20240430-091049-ladsgroup.json
  • 09:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 09:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2179 (T360332)', diff saved to https://phabricator.wikimedia.org/P61479 and previous config saved to /var/cache/conftool/dbconfig/20240430-091002-arnaudb.json
  • 09:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 09:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2179.codfw.wmnet with reason: Maintenance
  • 09:08 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1158.eqiad.wmnet with reason: host reimage
  • 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 75%: Post replag', diff saved to https://phabricator.wikimedia.org/P61478 and previous config saved to /var/cache/conftool/dbconfig/20240430-090724-arnaudb.json
  • 09:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P61477 and previous config saved to /var/cache/conftool/dbconfig/20240430-090048-arnaudb.json
  • 08:54 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1158.eqiad.wmnet with OS bookworm
  • 08:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61476 and previous config saved to /var/cache/conftool/dbconfig/20240430-085441-root.json
  • 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 50%: Post replag', diff saved to https://phabricator.wikimedia.org/P61475 and previous config saved to /var/cache/conftool/dbconfig/20240430-085219-arnaudb.json
  • 08:51 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1158', diff saved to https://phabricator.wikimedia.org/P61474 and previous config saved to /var/cache/conftool/dbconfig/20240430-085129-root.json
  • 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61473 and previous config saved to /var/cache/conftool/dbconfig/20240430-084926-root.json
  • 08:48 volans@cumin1002: START - Cookbook sre.deploy.python-code netbox to netbox-dev2002.codfw.wmnet with reason: Update Netbox dependencies for netbox-next - volans@cumin1002
  • 08:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2121', diff saved to https://phabricator.wikimedia.org/P61472 and previous config saved to /var/cache/conftool/dbconfig/20240430-084541-arnaudb.json
  • 08:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61471 and previous config saved to /var/cache/conftool/dbconfig/20240430-083935-root.json
  • 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 25%: Post replag', diff saved to https://phabricator.wikimedia.org/P61470 and previous config saved to /var/cache/conftool/dbconfig/20240430-083713-arnaudb.json
  • 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61469 and previous config saved to /var/cache/conftool/dbconfig/20240430-083420-root.json
  • 08:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2121 (T360332)', diff saved to https://phabricator.wikimedia.org/P61468 and previous config saved to /var/cache/conftool/dbconfig/20240430-083033-arnaudb.json
  • 08:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2121 (T360332)', diff saved to https://phabricator.wikimedia.org/P61467 and previous config saved to /var/cache/conftool/dbconfig/20240430-082753-arnaudb.json
  • 08:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 08:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2121.codfw.wmnet with reason: Maintenance
  • 08:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61466 and previous config saved to /var/cache/conftool/dbconfig/20240430-082430-root.json
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2151 (re)pooling @ 5%: Post replag', diff saved to https://phabricator.wikimedia.org/P61465 and previous config saved to /var/cache/conftool/dbconfig/20240430-082208-arnaudb.json
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Removes db2114, repools db2151', diff saved to https://phabricator.wikimedia.org/P61464 and previous config saved to /var/cache/conftool/dbconfig/20240430-082200-arnaudb.json
  • 08:21 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.3 refs T361397
  • 08:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61463 and previous config saved to /var/cache/conftool/dbconfig/20240430-081915-root.json
  • 08:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61462 and previous config saved to /var/cache/conftool/dbconfig/20240430-080924-root.json
  • 08:08 godog: bounce prometheus@k8s in eqiad - T343529
  • 08:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61461 and previous config saved to /var/cache/conftool/dbconfig/20240430-080409-root.json
  • 07:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61460 and previous config saved to /var/cache/conftool/dbconfig/20240430-075418-root.json
  • 07:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61459 and previous config saved to /var/cache/conftool/dbconfig/20240430-074903-root.json
  • 07:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61458 and previous config saved to /var/cache/conftool/dbconfig/20240430-073912-root.json
  • 07:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61457 and previous config saved to /var/cache/conftool/dbconfig/20240430-073358-root.json
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1157 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61456 and previous config saved to /var/cache/conftool/dbconfig/20240430-072406-root.json
  • 07:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1174.eqiad.wmnet with OS bookworm
  • 07:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1174 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61455 and previous config saved to /var/cache/conftool/dbconfig/20240430-071852-root.json
  • 07:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 07:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1174.eqiad.wmnet with reason: host reimage
  • 07:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1174.eqiad.wmnet with reason: host reimage
  • 06:48 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1174.eqiad.wmnet with OS bookworm
  • 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1174', diff saved to https://phabricator.wikimedia.org/P61454 and previous config saved to /var/cache/conftool/dbconfig/20240430-064720-root.json
  • 05:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1157.eqiad.wmnet with OS bookworm
  • 05:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2129 from API serving', diff saved to https://phabricator.wikimedia.org/P61453 and previous config saved to /var/cache/conftool/dbconfig/20240430-054943-arnaudb.json
  • 05:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 05:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1157.eqiad.wmnet with reason: host reimage
  • 05:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1157.eqiad.wmnet with reason: host reimage
  • 05:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1157.eqiad.wmnet with OS bookworm
  • 05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1157 T363672', diff saved to https://phabricator.wikimedia.org/P61452 and previous config saved to /var/cache/conftool/dbconfig/20240430-051419-root.json
  • 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1223 to s3 primary and set section read-write T363672', diff saved to https://phabricator.wikimedia.org/P61451 and previous config saved to /var/cache/conftool/dbconfig/20240430-051332-root.json
  • 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Set s3 eqiad as read-only for maintenance - T363672', diff saved to https://phabricator.wikimedia.org/P61450 and previous config saved to /var/cache/conftool/dbconfig/20240430-051312-root.json
  • 05:12 marostegui: Starting s3 eqiad failover from db1157 to db1223 - T363672
  • 05:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 05:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 04:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 24 hosts with reason: Primary switchover s3 T363672
  • 04:55 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1223 with weight 0 T363672', diff saved to https://phabricator.wikimedia.org/P61449 and previous config saved to /var/cache/conftool/dbconfig/20240430-045541-marostegui.json
  • 04:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 24 hosts with reason: Primary switchover s3 T363672
  • 04:05 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.43.0-wmf.3 refs T361397 (duration: 59m 27s)
  • 03:05 mwpresync@deploy1002: Started scap: testwikis wikis to 1.43.0-wmf.3 refs T361397
  • 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.26 (duration: 03m 03s)
  • 02:16 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7004.magru.wmnet with OS bullseye
  • 02:16 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 02:15 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 01:55 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7004.magru.wmnet with reason: host reimage
  • 01:53 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7004.magru.wmnet with reason: host reimage
  • 01:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7004.magru.wmnet with OS bullseye
  • 00:45 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7004.magru.wmnet with OS bullseye

2024-04-29

  • 23:56 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7004.magru.wmnet with OS bullseye
  • 22:57 eileen: civicrm upgraded from e95e03d9 to 393e1deb
  • 21:19 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['lvs7002']
  • 21:14 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7004.mgmt.magru.wmnet with reboot policy FORCED
  • 21:06 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7004.mgmt.magru.wmnet with reboot policy FORCED
  • 21:06 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7001.mgmt.magru.wmnet with reboot policy FORCED
  • 21:06 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs7002']
  • 21:06 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['lvs7002']
  • 21:06 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs7002']
  • 21:06 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7002.mgmt.magru.wmnet with reboot policy FORCED
  • 21:06 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7003.mgmt.magru.wmnet with reboot policy FORCED
  • 21:02 cjming: end of UTC late backport window
  • 21:02 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs7002.mgmt.magru.wmnet with reboot policy FORCED
  • 21:01 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dns7001']
  • 20:58 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7003.mgmt.magru.wmnet with reboot policy FORCED
  • 20:58 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:58 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:58 mforns@deploy1002: Finished deploy [airflow-dags/analytics@8c9c32c]: (no justification provided) (duration: 00m 30s)
  • 20:57 mforns@deploy1002: Started deploy [airflow-dags/analytics@8c9c32c]: (no justification provided)
  • 20:54 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001']
  • 20:41 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['lvs7003']
  • 20:41 robh@cumin2002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 20:39 robh@cumin2002: START - Cookbook sre.dns.netbox
  • 20:39 cjming@deploy1002: Finished scap: Backport for Turn off DiscussionTools A/B test, and enable features on those wikis (T341491) (duration: 16m 06s)
  • 20:37 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7003.magru.wmnet with OS bullseye
  • 20:37 fabfur@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 20:35 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs7003']
  • 20:29 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti7004']
  • 20:29 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['lvs7001']
  • 20:28 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001']
  • 20:28 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001']
  • 20:28 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti7003']
  • 20:28 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti7002']
  • 20:27 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['ganeti7001']
  • 20:27 topranks: re-announcing magru prefixes to from EdgeUno
  • 20:27 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['dns7002']
  • 20:26 cjming@deploy1002: cjming and esanders: Continuing with sync
  • 20:26 robh@cumin2002: START - Cookbook sre.hosts.provision for host lvs7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:26 cjming@deploy1002: cjming and esanders: Backport for Turn off DiscussionTools A/B test, and enable features on those wikis (T341491) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:24 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['lvs7002']
  • 20:24 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs7002']
  • 20:23 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['lvs7001']
  • 20:23 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti7004']
  • 20:23 cjming@deploy1002: Started scap: Backport for Turn off DiscussionTools A/B test, and enable features on those wikis (T341491)
  • 20:23 fabfur@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - fabfur@cumin1002"
  • 20:22 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti7003']
  • 20:22 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti7002']
  • 20:22 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['ganeti7001']
  • 20:22 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:22 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lvs7003.mgmt.magru.wmnet with reboot policy FORCED
  • 20:21 cjming@deploy1002: Finished scap: Backport for Shift writes to SUP, 1st batch (T363475) (duration: 16m 17s)
  • 20:20 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001']
  • 20:20 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001']
  • 20:19 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001']
  • 20:19 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001']
  • 20:19 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7002']
  • 20:19 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['dns7001']
  • 20:19 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['dns7001']
  • 20:17 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7004.mgmt.magru.wmnet with reboot policy FORCED
  • 20:16 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:16 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7003.mgmt.magru.wmnet with reboot policy FORCED
  • 20:15 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host ganeti7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:15 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dns7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:14 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host dns7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:09 cjming@deploy1002: cjming and pfischer: Continuing with sync
  • 20:09 cjming@deploy1002: cjming and pfischer: Backport for Shift writes to SUP, 1st batch (T363475) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:09 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp7002.magru.wmnet with OS bullseye
  • 20:09 sukhe@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 20:07 topranks: withdrawing prefixes from EdgeUno transit in magru to test paths via second transit
  • 20:06 robh@cumin2002: START - Cookbook sre.hosts.provision for host lvs7003.mgmt.magru.wmnet with reboot policy FORCED
  • 20:05 cjming@deploy1002: Started scap: Backport for Shift writes to SUP, 1st batch (T363475)
  • 20:05 robh@cumin2002: START - Cookbook sre.hosts.provision for host lvs7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:05 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7004.mgmt.magru.wmnet with reboot policy FORCED
  • 20:04 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7003.mgmt.magru.wmnet with reboot policy FORCED
  • 20:04 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:03 robh@cumin2002: START - Cookbook sre.hosts.provision for host ganeti7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:03 robh@cumin2002: START - Cookbook sre.hosts.provision for host dns7002.mgmt.magru.wmnet with reboot policy FORCED
  • 20:03 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 20:02 robh@cumin2002: START - Cookbook sre.hosts.provision for host dns7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:00 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7003.magru.wmnet with reason: host reimage
  • 19:51 sukhe@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - sukhe@cumin1002"
  • 19:46 robh@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:46 robh@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru misc hosts - robh@cumin2002"
  • 19:45 robh@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru misc hosts - robh@cumin2002"
  • 19:41 robh@cumin2002: START - Cookbook sre.dns.netbox
  • 19:33 mforns@deploy1002: Finished deploy [analytics/refinery@1693892] (hadoop-test): Fixes queries for Commons Impact MEtrics dumps TEST [analytics/refinery@1693892a] (duration: 03m 22s)
  • 19:31 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7003.magru.wmnet with OS bullseye
  • 19:31 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp7002.magru.wmnet with reason: host reimage
  • 19:30 mforns@deploy1002: Started deploy [analytics/refinery@1693892] (hadoop-test): Fixes queries for Commons Impact MEtrics dumps TEST [analytics/refinery@1693892a]
  • 19:29 mforns@deploy1002: Finished deploy [analytics/refinery@1693892] (thin): Fixes queries for Commons Impact MEtrics dumps THIN [analytics/refinery@1693892a] (duration: 03m 46s)
  • 19:27 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp7002.magru.wmnet with reason: host reimage
  • 19:26 mforns@deploy1002: Started deploy [analytics/refinery@1693892] (thin): Fixes queries for Commons Impact MEtrics dumps THIN [analytics/refinery@1693892a]
  • 19:25 mforns@deploy1002: Finished deploy [analytics/refinery@1693892]: Fixes to queries for Commons Impact Metrics dumps [analytics/refinery@1693892a] (duration: 13m 58s)
  • 19:11 mforns@deploy1002: Started deploy [analytics/refinery@1693892]: Fixes to queries for Commons Impact Metrics dumps [analytics/refinery@1693892a]
  • 19:08 fabfur@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7003.magru.wmnet with OS bullseye
  • 18:58 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp7002.magru.wmnet with OS bullseye
  • 18:50 swfrench-wmf: running authdns-update on dns1004 for T361835
  • 18:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 18:43 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 18:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 18:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T361627)', diff saved to https://phabricator.wikimedia.org/P61447 and previous config saved to /var/cache/conftool/dbconfig/20240429-183903-marostegui.json
  • 18:25 denisse: Manually delete unused TLS certificates for thanos-query as part of the CFSSL migration - T360414
  • 18:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P61446 and previous config saved to /var/cache/conftool/dbconfig/20240429-182355-marostegui.json
  • 18:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P61445 and previous config saved to /var/cache/conftool/dbconfig/20240429-180848-marostegui.json
  • 18:02 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7016']
  • 18:02 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7014']
  • 18:02 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7015']
  • 18:01 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7013']
  • 18:00 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7012']
  • 17:59 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7011']
  • 17:59 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7010']
  • 17:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7016']
  • 17:56 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7015']
  • 17:55 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7014']
  • 17:54 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7013']
  • 17:54 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7012']
  • 17:53 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp7003.magru.wmnet with OS bullseye
  • 17:53 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7011']
  • 17:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T361627)', diff saved to https://phabricator.wikimedia.org/P61444 and previous config saved to /var/cache/conftool/dbconfig/20240429-175340-marostegui.json
  • 17:53 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7010']
  • 17:49 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7009']
  • 17:49 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7007']
  • 17:49 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7005']
  • 17:49 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7004']
  • 17:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T361627)', diff saved to https://phabricator.wikimedia.org/P61443 and previous config saved to /var/cache/conftool/dbconfig/20240429-174856-marostegui.json
  • 17:48 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7008']
  • 17:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 17:48 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7006']
  • 17:48 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7003']
  • 17:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T361627)', diff saved to https://phabricator.wikimedia.org/P61442 and previous config saved to /var/cache/conftool/dbconfig/20240429-174834-marostegui.json
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7009']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7008']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7007']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7006']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7005']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7004']
  • 17:42 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7003']
  • 17:41 root@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase10[29-42]*: Move Cassandra to PKI - root@cumin1002
  • 17:40 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7006.mgmt.magru.wmnet with reboot policy FORCED
  • 17:38 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7014.mgmt.magru.wmnet with reboot policy FORCED
  • 17:38 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7016.mgmt.magru.wmnet with reboot policy FORCED
  • 17:37 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7012.mgmt.magru.wmnet with reboot policy FORCED
  • 17:37 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7008.mgmt.magru.wmnet with reboot policy FORCED
  • 17:36 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7004.mgmt.magru.wmnet with reboot policy FORCED
  • 17:36 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1005.eqiad.wmnet with OS bullseye
  • 17:36 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7010.mgmt.magru.wmnet with reboot policy FORCED
  • 17:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P61441 and previous config saved to /var/cache/conftool/dbconfig/20240429-173326-marostegui.json
  • 17:29 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7006.mgmt.magru.wmnet with reboot policy FORCED
  • 17:28 ebernhardson@deploy1002: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:28 ebernhardson@deploy1002: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:26 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7016.mgmt.magru.wmnet with reboot policy FORCED
  • 17:26 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7014.mgmt.magru.wmnet with reboot policy FORCED
  • 17:25 robh@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp7006.mgmt.magru.wmnet with reboot policy FORCED
  • 17:25 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7012.mgmt.magru.wmnet with reboot policy FORCED
  • 17:24 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7010.mgmt.magru.wmnet with reboot policy FORCED
  • 17:24 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7008.mgmt.magru.wmnet with reboot policy FORCED
  • 17:24 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7006.mgmt.magru.wmnet with reboot policy FORCED
  • 17:24 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7004.mgmt.magru.wmnet with reboot policy FORCED
  • 17:22 robh@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:22 robh@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rack b4 cp hosts - robh@cumin2002"
  • 17:22 robh@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rack b4 cp hosts - robh@cumin2002"
  • 17:20 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage
  • 17:19 robh@cumin2002: START - Cookbook sre.dns.netbox
  • 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P61440 and previous config saved to /var/cache/conftool/dbconfig/20240429-171818-marostegui.json
  • 17:17 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7015.mgmt.magru.wmnet with reboot policy FORCED
  • 17:17 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1005.eqiad.wmnet with reason: host reimage
  • 17:15 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7013.mgmt.magru.wmnet with reboot policy FORCED
  • 17:14 ebernhardson@deploy1002: helmfile [codfw] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:13 ebernhardson@deploy1002: helmfile [codfw] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:13 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7002.magru.wmnet with OS bullseye
  • 17:11 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7011.mgmt.magru.wmnet with reboot policy FORCED
  • 17:06 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7015.mgmt.magru.wmnet with reboot policy FORCED
  • 17:03 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7013.mgmt.magru.wmnet with reboot policy FORCED
  • 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T361627)', diff saved to https://phabricator.wikimedia.org/P61439 and previous config saved to /var/cache/conftool/dbconfig/20240429-170311-marostegui.json
  • 17:02 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7009.mgmt.magru.wmnet with reboot policy FORCED
  • 16:59 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1005.eqiad.wmnet with OS bullseye
  • 16:58 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7011.mgmt.magru.wmnet with reboot policy FORCED
  • 16:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T361627)', diff saved to https://phabricator.wikimedia.org/P61438 and previous config saved to /var/cache/conftool/dbconfig/20240429-165728-marostegui.json
  • 16:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 16:57 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7007.mgmt.magru.wmnet with reboot policy FORCED
  • 16:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 16:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T361627)', diff saved to https://phabricator.wikimedia.org/P61437 and previous config saved to /var/cache/conftool/dbconfig/20240429-165705-marostegui.json
  • 16:50 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7009.mgmt.magru.wmnet with reboot policy FORCED
  • 16:49 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7005.mgmt.magru.wmnet with reboot policy FORCED
  • 16:45 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7007.mgmt.magru.wmnet with reboot policy FORCED
  • 16:44 robh@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7003.mgmt.magru.wmnet with reboot policy FORCED
  • 16:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P61436 and previous config saved to /var/cache/conftool/dbconfig/20240429-164158-marostegui.json
  • 16:37 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7005.mgmt.magru.wmnet with reboot policy FORCED
  • 16:32 robh@cumin2002: START - Cookbook sre.hosts.provision for host cp7003.mgmt.magru.wmnet with reboot policy FORCED
  • 16:30 robh@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:30 robh@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rack b3 cp hosts - robh@cumin2002"
  • 16:29 robh@cumin2002: START - Cookbook sre.hosts.reimage for host cp7002.magru.wmnet with OS bullseye
  • 16:29 robh@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: rack b3 cp hosts - robh@cumin2002"
  • 16:27 robh@cumin2002: START - Cookbook sre.dns.netbox
  • 16:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P61435 and previous config saved to /var/cache/conftool/dbconfig/20240429-162650-marostegui.json
  • 16:26 robh@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7002']
  • 16:23 jayme@cumin1002: conftool action : set/pooled=yes; selector: name=kubestagemaster2003.codfw.wmnet
  • 16:23 jayme@cumin1002: conftool action : set/weight=10; selector: name=kubestagemaster2003.codfw.wmnet
  • 16:20 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7002']
  • 16:20 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=1) upgrade firmware for hosts ['cp7002']
  • 16:19 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7002']
  • 16:19 robh@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp7002']
  • 16:19 robh@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7002']
  • 16:18 robh@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp7002.magru.wmnet with OS bullseye
  • 16:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T361627)', diff saved to https://phabricator.wikimedia.org/P61434 and previous config saved to /var/cache/conftool/dbconfig/20240429-161143-marostegui.json
  • 16:10 jdrewniak@deploy1002: Synchronized portals: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 14m 10s)
  • 16:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T361627)', diff saved to https://phabricator.wikimedia.org/P61433 and previous config saved to /var/cache/conftool/dbconfig/20240429-160859-marostegui.json
  • 16:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 16:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T361627)', diff saved to https://phabricator.wikimedia.org/P61432 and previous config saved to /var/cache/conftool/dbconfig/20240429-160836-marostegui.json
  • 16:06 robh@cumin2002: START - Cookbook sre.hosts.reimage for host cp7002.magru.wmnet with OS bullseye
  • 15:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2114 T363713', diff saved to https://phabricator.wikimedia.org/P61431 and previous config saved to /var/cache/conftool/dbconfig/20240429-155838-arnaudb.json
  • 15:56 jdrewniak@deploy1002: Synchronized portals/wikipedia.org/assets: Wikimedia Portals Update: Bumping portals to master (T128546) (duration: 14m 38s)
  • 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2129 to s6 primary T363713', diff saved to https://phabricator.wikimedia.org/P61430 and previous config saved to /var/cache/conftool/dbconfig/20240429-155557-arnaudb.json
  • 15:55 arnaudb: Starting s6 codfw failover from db2114 to db2129 - T363713
  • 15:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P61429 and previous config saved to /var/cache/conftool/dbconfig/20240429-155328-marostegui.json
  • 15:44 swfrench@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:43 swfrench@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:39 swfrench@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P61428 and previous config saved to /var/cache/conftool/dbconfig/20240429-153821-marostegui.json
  • 15:37 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2200.codfw.wmnet with reason: Maintenance
  • 15:37 swfrench@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:37 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2200.codfw.wmnet with reason: Maintenance
  • 15:36 swfrench@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 15:35 swfrench@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 15:35 root@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase10[29-42]*: Move Cassandra to PKI - root@cumin1002
  • 15:34 swfrench@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:32 swfrench@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 15:28 arnaudb@cumin1002: dbctl commit (dc=all): 'depool db2151', diff saved to https://phabricator.wikimedia.org/P61427 and previous config saved to /var/cache/conftool/dbconfig/20240429-152809-arnaudb.json
  • 15:25 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase1028.eqiad.wmnet: Move to PKI TLS certs - elukey@cumin1002
  • 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T361627)', diff saved to https://phabricator.wikimedia.org/P61426 and previous config saved to /var/cache/conftool/dbconfig/20240429-152314-marostegui.json
  • 15:23 robh@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['cp7002']
  • 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T361627)', diff saved to https://phabricator.wikimedia.org/P61425 and previous config saved to /var/cache/conftool/dbconfig/20240429-152029-marostegui.json
  • 15:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T361627)', diff saved to https://phabricator.wikimedia.org/P61424 and previous config saved to /var/cache/conftool/dbconfig/20240429-152006-marostegui.json
  • 15:19 robh@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7002']
  • 15:15 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase1028.eqiad.wmnet: Move to PKI TLS certs - elukey@cumin1002
  • 15:14 robh@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7002.mgmt.magru.wmnet with reboot policy FORCED
  • 15:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P61423 and previous config saved to /var/cache/conftool/dbconfig/20240429-150459-marostegui.json
  • 15:03 robh@cumin1002: START - Cookbook sre.hosts.provision for host cp7002.mgmt.magru.wmnet with reboot policy FORCED
  • 15:02 robh@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:02 robh@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7002 setup - robh@cumin1002"
  • 15:01 robh@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7002 setup - robh@cumin1002"
  • 14:59 robh@cumin1002: START - Cookbook sre.dns.netbox
  • 14:54 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase20[23-35]*: Roll out PKI TLS certs - elukey@cumin1002
  • 14:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61422 and previous config saved to /var/cache/conftool/dbconfig/20240429-145306-arnaudb.json
  • 14:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T363713
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2129 with weight 0 T363713', diff saved to https://phabricator.wikimedia.org/P61421 and previous config saved to /var/cache/conftool/dbconfig/20240429-145203-arnaudb.json
  • 14:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T363713
  • 14:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P61420 and previous config saved to /var/cache/conftool/dbconfig/20240429-144951-marostegui.json
  • 14:41 taavi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt2002-dev.codfw.wmnet with OS bookworm
  • 14:38 godog: add 120G to prometheus/k8s in codfw
  • 14:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61418 and previous config saved to /var/cache/conftool/dbconfig/20240429-143800-arnaudb.json
  • 14:37 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 14:36 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 14:36 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 14:35 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 14:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T361627)', diff saved to https://phabricator.wikimedia.org/P61417 and previous config saved to /var/cache/conftool/dbconfig/20240429-143444-marostegui.json
  • 14:32 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1004.eqiad.wmnet with OS bullseye
  • 14:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T361627)', diff saved to https://phabricator.wikimedia.org/P61416 and previous config saved to /var/cache/conftool/dbconfig/20240429-143053-marostegui.json
  • 14:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 14:30 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 14:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T361627)', diff saved to https://phabricator.wikimedia.org/P61415 and previous config saved to /var/cache/conftool/dbconfig/20240429-143030-marostegui.json
  • 14:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61414 and previous config saved to /var/cache/conftool/dbconfig/20240429-142254-arnaudb.json
  • 14:17 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1004.eqiad.wmnet with reason: host reimage
  • 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P61413 and previous config saved to /var/cache/conftool/dbconfig/20240429-141523-marostegui.json
  • 14:14 reedy@deploy1002: Synchronized php-1.43.0-wmf.2/extensions/TimedMediaHandler/: T363550 (duration: 14m 42s)
  • 14:14 taavi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 14:11 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1004.eqiad.wmnet with reason: host reimage
  • 14:11 taavi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt2002-dev.codfw.wmnet with reason: host reimage
  • 14:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61412 and previous config saved to /var/cache/conftool/dbconfig/20240429-140748-arnaudb.json
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P61411 and previous config saved to /var/cache/conftool/dbconfig/20240429-140015-marostegui.json
  • 13:53 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1004.eqiad.wmnet with OS bullseye
  • 13:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61410 and previous config saved to /var/cache/conftool/dbconfig/20240429-135241-arnaudb.json
  • 13:50 taavi@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt2002-dev.codfw.wmnet with OS bookworm
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T361627)', diff saved to https://phabricator.wikimedia.org/P61409 and previous config saved to /var/cache/conftool/dbconfig/20240429-134507-marostegui.json
  • 13:43 dcausse@deploy1002: Finished scap: Backport for Revert "cirrus: Shift autocomplete traffic to codfw" (T363516) (duration: 16m 02s)
  • 13:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T361627)', diff saved to https://phabricator.wikimedia.org/P61408 and previous config saved to /var/cache/conftool/dbconfig/20240429-134115-marostegui.json
  • 13:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 13:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 13:40 sukhe: sudo cumin -b1 -s10 "A:cp-text" "run-puppet-agent --enable 'merging CR 1025357'"
  • 13:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 13:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 13:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2179 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61407 and previous config saved to /var/cache/conftool/dbconfig/20240429-133736-arnaudb.json
  • 13:30 dcausse@deploy1002: dcausse: Continuing with sync
  • 13:29 dcausse@deploy1002: dcausse: Backport for Revert "cirrus: Shift autocomplete traffic to codfw" (T363516) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2179.codfw.wmnet with OS bookworm
  • 13:27 dcausse@deploy1002: Started scap: Backport for Revert "cirrus: Shift autocomplete traffic to codfw" (T363516)
  • 13:26 sukhe: sudo cumin "A:cp-text" "disable-puppet 'merging CR 1025357'"
  • 13:24 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1003.eqiad.wmnet with OS bullseye
  • 13:23 dcausse@deploy1002: Finished scap: Backport for CommonSettings: change jobrunner xff to mw-jobrunner (duration: 16m 10s)
  • 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61406 and previous config saved to /var/cache/conftool/dbconfig/20240429-131728-root.json
  • 13:09 dcausse@deploy1002: hnowlan and dcausse: Continuing with sync
  • 13:09 dcausse@deploy1002: hnowlan and dcausse: Backport for CommonSettings: change jobrunner xff to mw-jobrunner synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:08 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1087.eqiad.wmnet
  • 13:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2179.codfw.wmnet with reason: host reimage
  • 13:07 dcausse@deploy1002: Started scap: Backport for CommonSettings: change jobrunner xff to mw-jobrunner
  • 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'fix weights', diff saved to https://phabricator.wikimedia.org/P61405 and previous config saved to /var/cache/conftool/dbconfig/20240429-130652-arnaudb.json
  • 13:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2179.codfw.wmnet with reason: host reimage
  • 13:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61403 and previous config saved to /var/cache/conftool/dbconfig/20240429-130222-root.json
  • 13:02 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1087.eqiad.wmnet
  • 12:58 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase20[23-35]*: Roll out PKI TLS certs - elukey@cumin1002
  • 12:57 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1003.eqiad.wmnet with reason: host reimage
  • 12:54 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1003.eqiad.wmnet with reason: host reimage
  • 12:53 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2022.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002
  • 12:49 brouberol@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:48 brouberol@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 12:48 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:47 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:47 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2179.codfw.wmnet with OS bookworm
  • 12:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61402 and previous config saved to /var/cache/conftool/dbconfig/20240429-124716-root.json
  • 12:47 brouberol@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:46 brouberol@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:24 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61398 and previous config saved to /var/cache/conftool/dbconfig/20240429-122246-root.json
  • 12:18 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 12:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61397 and previous config saved to /var/cache/conftool/dbconfig/20240429-121704-root.json
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2140 with weight 0 T363688', diff saved to https://phabricator.wikimedia.org/P61396 and previous config saved to /var/cache/conftool/dbconfig/20240429-121559-arnaudb.json
  • 12:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 33 hosts with reason: Primary switchover s4 T363688
  • 12:14 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 12:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 33 hosts with reason: Primary switchover s4 T363688
  • 12:10 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cephosd1002.eqiad.wmnet
  • 12:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61395 and previous config saved to /var/cache/conftool/dbconfig/20240429-120740-root.json
  • 12:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61394 and previous config saved to /var/cache/conftool/dbconfig/20240429-120159-root.json
  • 12:01 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host cephosd1002.eqiad.wmnet
  • 11:52 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61393 and previous config saved to /var/cache/conftool/dbconfig/20240429-115234-root.json
  • 11:38 marostegui@cumin1002: dbctl commit (dc=all): 'db1180 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61392 and previous config saved to /var/cache/conftool/dbconfig/20240429-113850-root.json
  • 11:37 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61391 and previous config saved to /var/cache/conftool/dbconfig/20240429-113728-root.json
  • 11:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61390 and previous config saved to /var/cache/conftool/dbconfig/20240429-113640-root.json
  • 11:35 marostegui@cumin1002: END (ERROR) - Cookbook sre.mysql.clone (exit_code=97) of db1180.eqiad.wmnet onto es1036.eqiad.wmnet
  • 11:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T361627)', diff saved to https://phabricator.wikimedia.org/P61389 and previous config saved to /var/cache/conftool/dbconfig/20240429-113445-marostegui.json
  • 11:32 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1002.eqiad.wmnet with OS bullseye
  • 11:22 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61388 and previous config saved to /var/cache/conftool/dbconfig/20240429-112223-root.json
  • 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61387 and previous config saved to /var/cache/conftool/dbconfig/20240429-112134-root.json
  • 11:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P61386 and previous config saved to /var/cache/conftool/dbconfig/20240429-111938-marostegui.json
  • 11:07 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61385 and previous config saved to /var/cache/conftool/dbconfig/20240429-110717-root.json
  • 11:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61384 and previous config saved to /var/cache/conftool/dbconfig/20240429-110628-root.json
  • 11:05 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Added kubestagemaster2003 - jayme@cumin1002"
  • 11:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P61383 and previous config saved to /var/cache/conftool/dbconfig/20240429-110430-marostegui.json
  • 11:03 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Added kubestagemaster2003 - jayme@cumin1002"
  • 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1191.eqiad.wmnet with OS bookworm
  • 10:54 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephadm1001.eqiad.wmnet with OS bookworm
  • 10:54 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1002.eqiad.wmnet with reason: host reimage
  • 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'db1191 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61382 and previous config saved to /var/cache/conftool/dbconfig/20240429-105212-root.json
  • 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61381 and previous config saved to /var/cache/conftool/dbconfig/20240429-105122-root.json
  • 10:51 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1002.eqiad.wmnet with reason: host reimage
  • 10:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T361627)', diff saved to https://phabricator.wikimedia.org/P61380 and previous config saved to /var/cache/conftool/dbconfig/20240429-104923-marostegui.json
  • 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2211 (T361627)', diff saved to https://phabricator.wikimedia.org/P61379 and previous config saved to /var/cache/conftool/dbconfig/20240429-104501-marostegui.json
  • 10:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 10:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 10:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2201.codfw.wmnet with reason: Maintenance
  • 10:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2201.codfw.wmnet with reason: Maintenance
  • 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T361627)', diff saved to https://phabricator.wikimedia.org/P61378 and previous config saved to /var/cache/conftool/dbconfig/20240429-104152-marostegui.json
  • 10:40 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephadm1001.eqiad.wmnet with reason: host reimage
  • 10:37 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephadm1001.eqiad.wmnet with reason: host reimage
  • 10:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
  • 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61377 and previous config saved to /var/cache/conftool/dbconfig/20240429-103617-root.json
  • 10:35 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 10:35 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db1180.eqiad.wmnet onto es1036.eqiad.wmnet
  • 10:35 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1180', diff saved to https://phabricator.wikimedia.org/P61376 and previous config saved to /var/cache/conftool/dbconfig/20240429-103436-marostegui.json
  • 10:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1191.eqiad.wmnet with reason: host reimage
  • 10:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1002.eqiad.wmnet with OS bullseye
  • 10:26 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephadm1001.eqiad.wmnet with OS bookworm
  • 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P61375 and previous config saved to /var/cache/conftool/dbconfig/20240429-102644-marostegui.json
  • 10:26 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cephadm1001.eqiad.wmnet with OS bullseye
  • 10:25 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1001.eqiad.wmnet with OS bullseye
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61374 and previous config saved to /var/cache/conftool/dbconfig/20240429-102111-root.json
  • 10:20 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1191.eqiad.wmnet with OS bookworm
  • 10:19 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1191 T362745', diff saved to https://phabricator.wikimedia.org/P61373 and previous config saved to /var/cache/conftool/dbconfig/20240429-101908-marostegui.json
  • 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61372 and previous config saved to /var/cache/conftool/dbconfig/20240429-101525-root.json
  • 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P61371 and previous config saved to /var/cache/conftool/dbconfig/20240429-101137-marostegui.json
  • 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2121.codfw.wmnet with OS bookworm
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'db2121 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61370 and previous config saved to /var/cache/conftool/dbconfig/20240429-100605-root.json
  • 10:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61369 and previous config saved to /var/cache/conftool/dbconfig/20240429-100018-root.json
  • 09:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T361627)', diff saved to https://phabricator.wikimedia.org/P61368 and previous config saved to /var/cache/conftool/dbconfig/20240429-095629-marostegui.json
  • 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2192 (T361627)', diff saved to https://phabricator.wikimedia.org/P61367 and previous config saved to /var/cache/conftool/dbconfig/20240429-095259-marostegui.json
  • 09:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 09:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T361627)', diff saved to https://phabricator.wikimedia.org/P61366 and previous config saved to /var/cache/conftool/dbconfig/20240429-095237-marostegui.json
  • 09:49 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: apply
  • 09:48 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: apply
  • 09:47 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2121.codfw.wmnet with reason: host reimage
  • 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61365 and previous config saved to /var/cache/conftool/dbconfig/20240429-094512-root.json
  • 09:43 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 09:43 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2121.codfw.wmnet with reason: host reimage
  • 09:42 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifeeds: apply
  • 09:42 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/wikifeeds: apply
  • 09:41 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 09:39 akosiaris@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifeeds: apply
  • 09:39 akosiaris@deploy1002: helmfile [staging] START helmfile.d/services/wikifeeds: apply
  • 09:38 ladsgroup@deploy1002: Finished scap: Backport for rdbms: Protect against stale cache in LB::getMaxLag() (T361824) (duration: 20m 15s)
  • 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P61364 and previous config saved to /var/cache/conftool/dbconfig/20240429-093729-marostegui.json
  • 09:36 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/image-suggestion: apply
  • 09:36 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/image-suggestion: apply
  • 09:35 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/image-suggestion: apply
  • 09:35 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/image-suggestion: apply
  • 09:31 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/image-suggestion: apply
  • 09:31 btullis@deploy1002: helmfile [staging] START helmfile.d/services/image-suggestion: apply
  • 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61363 and previous config saved to /var/cache/conftool/dbconfig/20240429-093007-root.json
  • 09:25 ladsgroup@deploy1002: ladsgroup: Continuing with sync
  • 09:25 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2121.codfw.wmnet with OS bookworm
  • 09:24 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1001.eqiad.wmnet with OS bullseye
  • 09:23 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephadm1001.eqiad.wmnet with OS bullseye
  • 09:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P61362 and previous config saved to /var/cache/conftool/dbconfig/20240429-092222-marostegui.json
  • 09:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2218 from api', diff saved to https://phabricator.wikimedia.org/P61361 and previous config saved to /var/cache/conftool/dbconfig/20240429-092213-marostegui.json
  • 09:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2121 T363668', diff saved to https://phabricator.wikimedia.org/P61360 and previous config saved to /var/cache/conftool/dbconfig/20240429-092104-root.json
  • 09:20 ladsgroup@deploy1002: ladsgroup: Backport for rdbms: Protect against stale cache in LB::getMaxLag() (T361824) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db2218 to s7 primary T363668', diff saved to https://phabricator.wikimedia.org/P61359 and previous config saved to /var/cache/conftool/dbconfig/20240429-092029-marostegui.json
  • 09:20 marostegui: Starting s7 codfw failover from db2121 to db2218 - T363668
  • 09:18 ladsgroup@deploy1002: Started scap: Backport for rdbms: Protect against stale cache in LB::getMaxLag() (T361824)
  • 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61358 and previous config saved to /var/cache/conftool/dbconfig/20240429-091500-root.json
  • 09:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T361627)', diff saved to https://phabricator.wikimedia.org/P61357 and previous config saved to /var/cache/conftool/dbconfig/20240429-090701-marostegui.json
  • 09:04 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 09:03 jayme@cumin1002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 09:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2178 (T361627)', diff saved to https://phabricator.wikimedia.org/P61356 and previous config saved to /var/cache/conftool/dbconfig/20240429-090329-marostegui.json
  • 09:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 09:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T361627)', diff saved to https://phabricator.wikimedia.org/P61355 and previous config saved to /var/cache/conftool/dbconfig/20240429-090317-marostegui.json
  • 09:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363668
  • 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2218 with weight 0 T363668', diff saved to https://phabricator.wikimedia.org/P61354 and previous config saved to /var/cache/conftool/dbconfig/20240429-090046-marostegui.json
  • 09:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s7 T363668
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61353 and previous config saved to /var/cache/conftool/dbconfig/20240429-085953-root.json
  • 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61352 and previous config saved to /var/cache/conftool/dbconfig/20240429-085829-root.json
  • 08:54 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubestagemaster2003.codfw.wmnet
  • 08:54 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 08:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P61351 and previous config saved to /var/cache/conftool/dbconfig/20240429-084808-marostegui.json
  • 08:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1223.eqiad.wmnet with OS bookworm
  • 08:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1223 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61350 and previous config saved to /var/cache/conftool/dbconfig/20240429-084447-root.json
  • 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61349 and previous config saved to /var/cache/conftool/dbconfig/20240429-084323-root.json
  • 08:40 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 08:37 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 08:33 taavi@deploy1002: Finished scap: Backport for Fix disabling TOTP keys with scratch tokens (T363548) (duration: 15m 27s)
  • 08:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P61348 and previous config saved to /var/cache/conftool/dbconfig/20240429-083301-marostegui.json
  • 08:29 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61347 and previous config saved to /var/cache/conftool/dbconfig/20240429-082817-root.json
  • 08:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1223.eqiad.wmnet with reason: host reimage
  • 08:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1223.eqiad.wmnet with reason: host reimage
  • 08:21 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 08:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61346 and previous config saved to /var/cache/conftool/dbconfig/20240429-082056-root.json
  • 08:20 taavi@deploy1002: taavi: Continuing with sync
  • 08:20 taavi@deploy1002: taavi: Backport for Fix disabling TOTP keys with scratch tokens (T363548) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:18 taavi@deploy1002: Started scap: Backport for Fix disabling TOTP keys with scratch tokens (T363548)
  • 08:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T361627)', diff saved to https://phabricator.wikimedia.org/P61345 and previous config saved to /var/cache/conftool/dbconfig/20240429-081754-marostegui.json
  • 08:16 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 08:15 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 08:15 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2003.codfw.wmnet on all recursors
  • 08:15 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache kubestagemaster2003.codfw.wmnet on all recursors
  • 08:15 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:15 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 08:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61344 and previous config saved to /var/cache/conftool/dbconfig/20240429-081455-root.json
  • 08:14 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2171 (T361627)', diff saved to https://phabricator.wikimedia.org/P61343 and previous config saved to /var/cache/conftool/dbconfig/20240429-081323-marostegui.json
  • 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61342 and previous config saved to /var/cache/conftool/dbconfig/20240429-081312-root.json
  • 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 08:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 08:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T361627)', diff saved to https://phabricator.wikimedia.org/P61341 and previous config saved to /var/cache/conftool/dbconfig/20240429-081254-marostegui.json
  • 08:11 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 08:11 jayme@cumin1002: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2003.codfw.wmnet
  • 08:09 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1223.eqiad.wmnet with OS bookworm
  • 08:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1223', diff saved to https://phabricator.wikimedia.org/P61340 and previous config saved to /var/cache/conftool/dbconfig/20240429-080710-root.json
  • 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61339 and previous config saved to /var/cache/conftool/dbconfig/20240429-080550-root.json
  • 08:04 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 08:04 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 08:04 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 08:04 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 08:04 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 08:04 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 08:00 dcausse: restarting blazegraph on wdqs1019 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 07:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61338 and previous config saved to /var/cache/conftool/dbconfig/20240429-075949-root.json
  • 07:59 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 07:59 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 07:59 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 07:58 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 07:58 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 07:58 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61337 and previous config saved to /var/cache/conftool/dbconfig/20240429-075806-root.json
  • 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P61336 and previous config saved to /var/cache/conftool/dbconfig/20240429-075746-marostegui.json
  • 07:52 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 07:52 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 07:52 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 07:52 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 07:52 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 07:52 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 07:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61335 and previous config saved to /var/cache/conftool/dbconfig/20240429-075045-root.json
  • 07:49 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 07:48 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 07:48 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 07:48 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 07:48 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 07:47 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61334 and previous config saved to /var/cache/conftool/dbconfig/20240429-074444-root.json
  • 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61333 and previous config saved to /var/cache/conftool/dbconfig/20240429-074301-root.json
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P61332 and previous config saved to /var/cache/conftool/dbconfig/20240429-074238-marostegui.json
  • 07:37 marostegui: Drop machinevision tables on commonswiki T362229
  • 07:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61331 and previous config saved to /var/cache/conftool/dbconfig/20240429-073539-root.json
  • 07:35 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 07:34 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 07:34 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 07:34 marostegui: Drop machinevision tables on testcommonswiki T362229
  • 07:34 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 07:34 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 07:33 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61330 and previous config saved to /var/cache/conftool/dbconfig/20240429-072937-root.json
  • 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2159 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61329 and previous config saved to /var/cache/conftool/dbconfig/20240429-072755-root.json
  • 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T361627)', diff saved to https://phabricator.wikimedia.org/P61328 and previous config saved to /var/cache/conftool/dbconfig/20240429-072731-marostegui.json
  • 07:26 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2159.codfw.wmnet with OS bookworm
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2157 (T361627)', diff saved to https://phabricator.wikimedia.org/P61327 and previous config saved to /var/cache/conftool/dbconfig/20240429-072404-marostegui.json
  • 07:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T361627)', diff saved to https://phabricator.wikimedia.org/P61326 and previous config saved to /var/cache/conftool/dbconfig/20240429-072341-marostegui.json
  • 07:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61325 and previous config saved to /var/cache/conftool/dbconfig/20240429-072033-root.json
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61324 and previous config saved to /var/cache/conftool/dbconfig/20240429-071431-root.json
  • 07:13 slyngs: Upgrade idm.wikimedia.org to Bitu 0.7.0
  • 07:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
  • 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P61323 and previous config saved to /var/cache/conftool/dbconfig/20240429-070834-marostegui.json
  • 07:07 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2159.codfw.wmnet with reason: host reimage
  • 07:05 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61322 and previous config saved to /var/cache/conftool/dbconfig/20240429-070527-root.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61321 and previous config saved to /var/cache/conftool/dbconfig/20240429-065926-root.json
  • 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P61320 and previous config saved to /var/cache/conftool/dbconfig/20240429-065326-marostegui.json
  • 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'es1023 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61319 and previous config saved to /var/cache/conftool/dbconfig/20240429-065022-root.json
  • 06:48 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2159.codfw.wmnet with OS bookworm
  • 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2159', diff saved to https://phabricator.wikimedia.org/P61318 and previous config saved to /var/cache/conftool/dbconfig/20240429-064717-root.json
  • 06:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1212.eqiad.wmnet with OS bookworm
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1212 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61317 and previous config saved to /var/cache/conftool/dbconfig/20240429-064420-root.json
  • 06:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host es1023.eqiad.wmnet with OS bookworm
  • 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T361627)', diff saved to https://phabricator.wikimedia.org/P61316 and previous config saved to /var/cache/conftool/dbconfig/20240429-063819-marostegui.json
  • 06:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2128 (T361627)', diff saved to https://phabricator.wikimedia.org/P61315 and previous config saved to /var/cache/conftool/dbconfig/20240429-063450-marostegui.json
  • 06:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T361627)', diff saved to https://phabricator.wikimedia.org/P61314 and previous config saved to /var/cache/conftool/dbconfig/20240429-063412-marostegui.json
  • 06:24 marostegui: Restart sanitarium instances in eqiad T363276
  • 06:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
  • 06:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on es1023.eqiad.wmnet with reason: host reimage
  • 06:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1212.eqiad.wmnet with reason: host reimage
  • 06:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P61313 and previous config saved to /var/cache/conftool/dbconfig/20240429-061905-marostegui.json
  • 06:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on es1023.eqiad.wmnet with reason: host reimage
  • 06:14 marostegui: Restart sanitarium instances in codfw T363276
  • 06:06 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1212.eqiad.wmnet with OS bookworm
  • 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1212', diff saved to https://phabricator.wikimedia.org/P61312 and previous config saved to /var/cache/conftool/dbconfig/20240429-060423-root.json
  • 06:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123', diff saved to https://phabricator.wikimedia.org/P61311 and previous config saved to /var/cache/conftool/dbconfig/20240429-060358-marostegui.json
  • 06:02 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host es1023.eqiad.wmnet with OS bookworm
  • 05:58 marostegui@deploy1002: Finished scap: Backport for Revert "db-production.php: Disable writes on es5" (duration: 14m 47s)
  • 05:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2123 (T361627)', diff saved to https://phabricator.wikimedia.org/P61310 and previous config saved to /var/cache/conftool/dbconfig/20240429-054850-marostegui.json
  • 05:46 marostegui@deploy1002: marostegui: Continuing with sync
  • 05:46 marostegui@deploy1002: marostegui: Backport for Revert "db-production.php: Disable writes on es5" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 05:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2123 (T361627)', diff saved to https://phabricator.wikimedia.org/P61309 and previous config saved to /var/cache/conftool/dbconfig/20240429-054519-marostegui.json
  • 05:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 05:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 05:44 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 05:44 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 05:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T352010)', diff saved to https://phabricator.wikimedia.org/P61308 and previous config saved to /var/cache/conftool/dbconfig/20240429-054413-ladsgroup.json
  • 05:43 marostegui@deploy1002: Started scap: Backport for Revert "db-production.php: Disable writes on es5"
  • 05:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1023 T361548', diff saved to https://phabricator.wikimedia.org/P61306 and previous config saved to /var/cache/conftool/dbconfig/20240429-054158-marostegui.json
  • 05:40 marostegui@cumin1002: dbctl commit (dc=all): 'Promote es1024 to es5 primary T361548', diff saved to https://phabricator.wikimedia.org/P61305 and previous config saved to /var/cache/conftool/dbconfig/20240429-054035-marostegui.json
  • 05:40 marostegui: Starting es5 eqiad failover from es1023 to es1024 T361548
  • 05:35 marostegui@deploy1002: Finished scap: Backport for db-production.php: Disable writes on es5 (T361548) (duration: 26m 58s)
  • 05:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P61304 and previous config saved to /var/cache/conftool/dbconfig/20240429-052906-ladsgroup.json
  • 05:23 marostegui@cumin1002: dbctl commit (dc=all): 'Set es1024 with weight 0 T361548', diff saved to https://phabricator.wikimedia.org/P61303 and previous config saved to /var/cache/conftool/dbconfig/20240429-052311-root.json
  • 05:22 marostegui@deploy1002: marostegui: Continuing with sync
  • 05:22 marostegui@deploy1002: marostegui: Backport for db-production.php: Disable writes on es5 (T361548) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 05:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P61302 and previous config saved to /var/cache/conftool/dbconfig/20240429-051359-ladsgroup.json
  • 05:08 marostegui@deploy1002: Started scap: Backport for db-production.php: Disable writes on es5 (T361548)
  • 05:05 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 05:04 root@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Primary switchover es5 T322187
  • 04:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T352010)', diff saved to https://phabricator.wikimedia.org/P61301 and previous config saved to /var/cache/conftool/dbconfig/20240429-045851-ladsgroup.json
  • 03:11 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1003.eqiad.wmnet with OS bookworm
  • 02:17 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
  • 02:14 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1003.eqiad.wmnet with reason: host reimage
  • 01:42 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1003.eqiad.wmnet with OS bookworm

2024-04-28

  • 20:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2195 (T352010)', diff saved to https://phabricator.wikimedia.org/P61300 and previous config saved to /var/cache/conftool/dbconfig/20240428-200522-ladsgroup.json
  • 20:05 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 20:05 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 20:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T352010)', diff saved to https://phabricator.wikimedia.org/P61299 and previous config saved to /var/cache/conftool/dbconfig/20240428-200500-ladsgroup.json
  • 19:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P61298 and previous config saved to /var/cache/conftool/dbconfig/20240428-194952-ladsgroup.json
  • 19:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P61297 and previous config saved to /var/cache/conftool/dbconfig/20240428-193445-ladsgroup.json
  • 19:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T352010)', diff saved to https://phabricator.wikimedia.org/P61296 and previous config saved to /var/cache/conftool/dbconfig/20240428-191938-ladsgroup.json
  • 07:45 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2181 (T352010)', diff saved to https://phabricator.wikimedia.org/P61295 and previous config saved to /var/cache/conftool/dbconfig/20240428-074511-ladsgroup.json
  • 07:45 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 07:44 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 07:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T352010)', diff saved to https://phabricator.wikimedia.org/P61294 and previous config saved to /var/cache/conftool/dbconfig/20240428-074448-ladsgroup.json
  • 07:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T352010)', diff saved to https://phabricator.wikimedia.org/P61293 and previous config saved to /var/cache/conftool/dbconfig/20240428-073827-ladsgroup.json
  • 07:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P61292 and previous config saved to /var/cache/conftool/dbconfig/20240428-072941-ladsgroup.json
  • 07:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P61291 and previous config saved to /var/cache/conftool/dbconfig/20240428-072320-ladsgroup.json
  • 07:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P61290 and previous config saved to /var/cache/conftool/dbconfig/20240428-071434-ladsgroup.json
  • 07:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P61289 and previous config saved to /var/cache/conftool/dbconfig/20240428-070812-ladsgroup.json
  • 06:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T352010)', diff saved to https://phabricator.wikimedia.org/P61288 and previous config saved to /var/cache/conftool/dbconfig/20240428-065927-ladsgroup.json
  • 06:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T352010)', diff saved to https://phabricator.wikimedia.org/P61287 and previous config saved to /var/cache/conftool/dbconfig/20240428-065305-ladsgroup.json

2024-04-27

  • 23:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T352010)', diff saved to https://phabricator.wikimedia.org/P61286 and previous config saved to /var/cache/conftool/dbconfig/20240427-231136-ladsgroup.json
  • 23:11 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 23:11 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 23:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61285 and previous config saved to /var/cache/conftool/dbconfig/20240427-231112-ladsgroup.json
  • 22:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P61284 and previous config saved to /var/cache/conftool/dbconfig/20240427-225604-ladsgroup.json
  • 22:44 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp7001.magru.wmnet with OS bullseye
  • 22:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P61283 and previous config saved to /var/cache/conftool/dbconfig/20240427-224057-ladsgroup.json
  • 22:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61282 and previous config saved to /var/cache/conftool/dbconfig/20240427-222548-ladsgroup.json
  • 21:16 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cp7001.magru.wmnet with OS bullseye
  • 21:14 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7001']
  • 21:06 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7001']
  • 21:01 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cp7001']
  • 20:42 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cp7001']
  • 20:41 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cp7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:29 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2167 (T352010)', diff saved to https://phabricator.wikimedia.org/P61281 and previous config saved to /var/cache/conftool/dbconfig/20240427-202602-ladsgroup.json
  • 20:25 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 20:25 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 20:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61280 and previous config saved to /var/cache/conftool/dbconfig/20240427-202539-ladsgroup.json
  • 20:25 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.provision (exit_code=99) for host cp7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:25 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cp7001.mgmt.magru.wmnet with reboot policy FORCED
  • 20:23 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:23 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7001 DNS add - pt1979@cumin2002"
  • 20:22 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cp7001 DNS add - pt1979@cumin2002"
  • 20:20 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 20:15 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:14 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 20:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P61279 and previous config saved to /var/cache/conftool/dbconfig/20240427-201031-ladsgroup.json
  • 19:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P61278 and previous config saved to /var/cache/conftool/dbconfig/20240427-195524-ladsgroup.json
  • 19:46 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:46 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002"
  • 19:45 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002"
  • 19:43 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 19:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61277 and previous config saved to /var/cache/conftool/dbconfig/20240427-194017-ladsgroup.json
  • 19:39 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 19:39 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002"
  • 19:38 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add DNS for magru PDUs - cmooney@cumin1002"
  • 19:34 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 17:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru oob - ayounsi@cumin1002"
  • 17:42 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru oob - ayounsi@cumin1002"
  • 17:39 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 17:06 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:06 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru - ayounsi@cumin1002"
  • 17:03 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: magru - ayounsi@cumin1002"
  • 17:01 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 14:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2212 (T352010)', diff saved to https://phabricator.wikimedia.org/P61275 and previous config saved to /var/cache/conftool/dbconfig/20240427-144642-ladsgroup.json
  • 14:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 14:46 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 09:51 volans: manually upgraded wmflib in netbox1002/2002's Netbox's venv
  • 08:58 volans: restarted uwsgi on netbox1002 to pickup the latest wmflib with magru
  • 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2166 (T352010)', diff saved to https://phabricator.wikimedia.org/P61274 and previous config saved to /var/cache/conftool/dbconfig/20240427-075233-ladsgroup.json
  • 07:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 07:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 07:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T352010)', diff saved to https://phabricator.wikimedia.org/P61273 and previous config saved to /var/cache/conftool/dbconfig/20240427-075210-ladsgroup.json
  • 07:43 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: Maintenance
  • 07:43 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2202.codfw.wmnet with reason: Maintenance
  • 07:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P61272 and previous config saved to /var/cache/conftool/dbconfig/20240427-074250-ladsgroup.json
  • 07:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P61271 and previous config saved to /var/cache/conftool/dbconfig/20240427-073703-ladsgroup.json
  • 07:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P61270 and previous config saved to /var/cache/conftool/dbconfig/20240427-072742-ladsgroup.json
  • 07:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P61269 and previous config saved to /var/cache/conftool/dbconfig/20240427-072155-ladsgroup.json
  • 07:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P61268 and previous config saved to /var/cache/conftool/dbconfig/20240427-071235-ladsgroup.json
  • 07:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T352010)', diff saved to https://phabricator.wikimedia.org/P61267 and previous config saved to /var/cache/conftool/dbconfig/20240427-070648-ladsgroup.json
  • 06:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P61266 and previous config saved to /var/cache/conftool/dbconfig/20240427-065728-ladsgroup.json
  • 00:51 urandom: rebooting puppetserver1001.eqiad.wmnet via drac

2024-04-26

  • 23:13 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T352010)', diff saved to https://phabricator.wikimedia.org/P61265 and previous config saved to /var/cache/conftool/dbconfig/20240426-231316-ladsgroup.json
  • 23:13 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 23:13 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 23:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T352010)', diff saved to https://phabricator.wikimedia.org/P61264 and previous config saved to /var/cache/conftool/dbconfig/20240426-231252-ladsgroup.json
  • 22:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P61263 and previous config saved to /var/cache/conftool/dbconfig/20240426-225744-ladsgroup.json
  • 22:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P61262 and previous config saved to /var/cache/conftool/dbconfig/20240426-224235-ladsgroup.json
  • 22:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T352010)', diff saved to https://phabricator.wikimedia.org/P61261 and previous config saved to /var/cache/conftool/dbconfig/20240426-222728-ladsgroup.json
  • 22:24 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lists2001.wikimedia.org with OS bookworm
  • 22:07 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lists2001.wikimedia.org with reason: host reimage
  • 22:04 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lists2001.wikimedia.org with reason: host reimage
  • 21:43 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bookworm
  • 21:38 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host lists2001.wikimedia.org with OS bullseye
  • 21:38 dzahn@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin2002"
  • 21:37 dzahn@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - dzahn@cumin2002"
  • 21:25 amastilovic@deploy1002: Finished deploy [airflow-dags/analytics@33b39d9]: (no justification provided) (duration: 00m 28s)
  • 21:24 amastilovic@deploy1002: Started deploy [airflow-dags/analytics@33b39d9]: (no justification provided)
  • 21:21 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on lists2001.wikimedia.org with reason: host reimage
  • 21:18 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on lists2001.wikimedia.org with reason: host reimage
  • 21:01 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bullseye
  • 19:11 mutante: LDAP - added linafaridwmde to groups wmde and nda (T362959)
  • 19:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2164 (T352010)', diff saved to https://phabricator.wikimedia.org/P61260 and previous config saved to /var/cache/conftool/dbconfig/20240426-190909-ladsgroup.json
  • 19:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 19:09 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 19:09 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 19:09 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 19:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61259 and previous config saved to /var/cache/conftool/dbconfig/20240426-190842-ladsgroup.json
  • 18:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P61258 and previous config saved to /var/cache/conftool/dbconfig/20240426-185335-ladsgroup.json
  • 18:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P61257 and previous config saved to /var/cache/conftool/dbconfig/20240426-183827-ladsgroup.json
  • 18:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61256 and previous config saved to /var/cache/conftool/dbconfig/20240426-182320-ladsgroup.json
  • 17:57 dancy@deploy1002: Finished scap: Testing T325530 (duration: 09m 14s)
  • 17:48 dancy@deploy1002: Started scap: Testing T325530
  • 17:47 dancy@deploy1002: Installation of scap version "4.80.0" completed for 325 hosts
  • 17:47 dancy@deploy1002: Installing scap version "4.80.0" for 325 hosts
  • 17:27 bking@cumin2002: conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet
  • 17:15 eoghan@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host lists2001.wikimedia.org with OS bookworm
  • 17:02 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 16:57 bking@cumin2002: conftool action : set/pooled=no; selector: name=elastic1107\.eqiad\.wmnet
  • 16:36 bking@cumin2002: conftool action : set/weight=20:pooled=yes; selector: name=elastic1107\.eqiad\.wmnet
  • 16:35 bking@cumin2002: conftool action : set/pooled=no; selector: name=elastic1106\.eqiad\.wmnet
  • 16:33 bking@cumin2002: conftool action : set/weight=20:pooled=yes; selector: name=elastic1106\.eqiad\.wmnet
  • 16:32 bking@cumin2002: conftool action : set/pooled=no; selector: name=elastic1105\.eqiad\.wmnet
  • 16:30 denisse: Delete the unused Prometheus PoP TLS certificates in the private repository as part of the cergen to CFSSL migration - T360414
  • 16:22 eoghan@cumin1002: START - Cookbook sre.hosts.reimage for host lists2001.wikimedia.org with OS bookworm
  • 16:20 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 16:20 eoghan@cumin1002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host lists2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 16:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 16:18 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 16:18 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 16:17 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 16:17 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 16:16 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:06 vgutierrez: repool ncredir6001
  • 15:56 bking@cumin2002: conftool action : set/pooled=no; selector: name=elastic1104\.eqiad\.wmnet
  • 15:56 bking@cumin2002: conftool action : set/pooled=no; selector: name=elastic1103\.eqiad\.wmnet
  • 15:55 vgutierrez: depool ncredir6001
  • 15:53 eoghan@cumin1002: START - Cookbook sre.hosts.provision for host lists2001.mgmt.codfw.wmnet with reboot policy FORCED
  • 15:46 denisse: Enabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 15:43 bking@cumin2002: conftool action : set/weight=20:pooled=yes; selector: name=elastic1105\.eqiad\.wmnet
  • 15:36 eoghan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:34 eoghan@cumin1002: START - Cookbook sre.dns.netbox
  • 15:31 bking@cumin2002: conftool action : set/weight=20:pooled=yes; selector: name=elastic1104\.eqiad\.wmnet
  • 15:29 denisse: testing patch #1023917 on prometheus6002 - T360414
  • 15:28 denisse: testing patch #1023917 on prometheus6002
  • 15:26 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kubestagemaster2003.codfw.wmnet
  • 15:26 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:25 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
  • 15:25 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts part of the cergen to CFSSL migration - T360414
  • 15:25 denisse: Disabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 15:24 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts part of the cergen to CFSSL migration - T360414
  • 15:23 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
  • 15:22 denisse: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 15:20 cmooney@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host lists2001
  • 15:19 cmooney@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host lists2001
  • 15:19 cmooney@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 15:14 bking@cumin2002: conftool action : set/weight=20:pooled=yes; selector: name=elastic1103\.eqiad\.wmnet
  • 15:12 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 15:07 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kubestagemaster2003.codfw.wmnet
  • 14:48 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2021.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002
  • 14:38 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2021.codfw.wmnet: Move to PKI TLS certs - elukey@cumin1002
  • 14:15 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephadm1001.eqiad.wmnet with OS bookworm
  • 14:10 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1014.eqiad.wmnet
  • 14:03 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephadm1001.eqiad.wmnet with reason: host reimage
  • 14:02 eevans@cumin1002: START - Cookbook sre.hosts.reboot-single for host aqs1014.eqiad.wmnet
  • 13:57 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephadm1001.eqiad.wmnet with reason: host reimage
  • 13:48 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 13:47 jayme@cumin1002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 13:45 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephadm1001.eqiad.wmnet with OS bookworm
  • 13:28 akosiaris@cumin1002: conftool action : set/pooled=no; selector: name=elastic110[3-7]\.eqiad\.wmnet
  • 13:28 eoghan@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts lists2001.codfw.wmnet
  • 13:28 eoghan@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:27 eoghan@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lists2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1002"
  • 13:25 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 13:23 eoghan@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: lists2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - eoghan@cumin1002"
  • 13:21 eoghan@cumin1002: START - Cookbook sre.dns.netbox
  • 13:14 eoghan@cumin1002: START - Cookbook sre.hosts.decommission for hosts lists2001.codfw.wmnet
  • 12:52 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 12:44 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 12:44 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 12:27 btullis@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host cephadm1001.eqiad.wmnet
  • 12:26 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cephadm1001.eqiad.wmnet with OS bookworm
  • 12:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T352010)', diff saved to https://phabricator.wikimedia.org/P61251 and previous config saved to /var/cache/conftool/dbconfig/20240426-121951-ladsgroup.json
  • 12:20 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 12:20 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 12:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T352010)', diff saved to https://phabricator.wikimedia.org/P61250 and previous config saved to /var/cache/conftool/dbconfig/20240426-121939-ladsgroup.json
  • 12:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P61249 and previous config saved to /var/cache/conftool/dbconfig/20240426-120431-ladsgroup.json
  • 11:53 claime: Silencing all alerts matching parse1002.* for 4 days - T363086
  • 11:53 moritzm: uploaded debdeploy 0.0.99.14 to apt.wikimedia.org (for buster/bullseye/bookworm)
  • 11:50 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubestagemaster2003.codfw.wmnet
  • 11:50 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 11:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P61248 and previous config saved to /var/cache/conftool/dbconfig/20240426-114923-ladsgroup.json
  • 11:43 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephadm1001.eqiad.wmnet with OS bookworm
  • 11:43 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cephadm1001.eqiad.wmnet - btullis@cumin1002"
  • 11:43 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cephadm1001.eqiad.wmnet - btullis@cumin1002"
  • 11:43 btullis@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cephadm1001.eqiad.wmnet on all recursors
  • 11:42 btullis@cumin1002: START - Cookbook sre.dns.wipe-cache cephadm1001.eqiad.wmnet on all recursors
  • 11:42 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:42 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cephadm1001.eqiad.wmnet - btullis@cumin1002"
  • 11:39 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cephadm1001.eqiad.wmnet - btullis@cumin1002"
  • 11:36 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 11:36 btullis@cumin1002: START - Cookbook sre.ganeti.makevm for new host cephadm1001.eqiad.wmnet
  • 11:35 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 11:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T352010)', diff saved to https://phabricator.wikimedia.org/P61247 and previous config saved to /var/cache/conftool/dbconfig/20240426-113416-ladsgroup.json
  • 11:33 claime: Forcing puppet run on O:alerting_host - T363086
  • 11:32 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 11:29 claime: Forcing puppet run on deploy server - T363086
  • 11:28 claime: Deactivating puppet for parse1002 - T363086
  • 11:19 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 11:19 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 11:19 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2002.wikimedia.org
  • 11:18 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 11:18 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2003.codfw.wmnet on all recursors
  • 11:17 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache kubestagemaster2003.codfw.wmnet on all recursors
  • 11:17 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:16 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 11:15 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 11:13 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 11:13 jayme@cumin1002: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2003.codfw.wmnet
  • 11:12 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts kubestagemaster2003.codfw.wmnet
  • 11:11 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:11 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
  • 11:10 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab2002.wikimedia.org
  • 11:10 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: kubestagemaster2003.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jayme@cumin1002"
  • 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 10:59 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 10:55 jayme@cumin1002: START - Cookbook sre.hosts.decommission for hosts kubestagemaster2003.codfw.wmnet
  • 10:54 jayme@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 10:46 jmm@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
  • 10:46 jmm@cumin2002: START - Cookbook sre.ganeti.resource-report
  • 10:08 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 10:07 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 10:06 jayme@cumin1002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 10:04 jayme@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=93) for new host kubestagemaster2003.codfw.wmnet
  • 10:03 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2003.codfw.wmnet on all recursors
  • 10:02 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache kubestagemaster2003.codfw.wmnet on all recursors
  • 10:00 jayme@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=99)
  • 09:58 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 09:57 jayme@cumin1002: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2003.codfw.wmnet
  • 09:54 dcausse@deploy1002: Finished scap: Backport for cirrus: Shift autocomplete traffic to codfw (T363516) (duration: 17m 57s)
  • 09:51 joal@deploy1002: Finished deploy [airflow-dags/analytics@e57ae00]: Deploy of Analytics airflow dags for browser-metrics [airflow-dags/analytics@e57ae006] (duration: 00m 27s)
  • 09:50 joal@deploy1002: Started deploy [airflow-dags/analytics@e57ae00]: Deploy of Analytics airflow dags for browser-metrics [airflow-dags/analytics@e57ae006]
  • 09:47 jayme: repooled mw2391.codfw.wmnet
  • 09:42 dcausse@deploy1002: dcausse and ebernhardson: Continuing with sync
  • 09:41 dcausse@deploy1002: dcausse and ebernhardson: Backport for cirrus: Shift autocomplete traffic to codfw (T363516) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:40 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.changedisk (exit_code=0) for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 09:40 jayme@cumin1002: START - Cookbook sre.ganeti.changedisk for changing disk type of kubestagemaster2003.codfw.wmnet to plain
  • 09:36 dcausse@deploy1002: Started scap: Backport for cirrus: Shift autocomplete traffic to codfw (T363516)
  • 08:57 hashar: Restarted Gerrit
  • 08:41 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 08:34 hashar: Restarted Gerrit replica
  • 08:34 jayme@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host kubestagemaster2003.codfw.wmnet
  • 08:33 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 08:18 jayme: depooled mw2391.codfw.wmnet for etcd benchmark
  • 07:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61246 and previous config saved to /var/cache/conftool/dbconfig/20240426-075748-arnaudb.json
  • 07:48 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 07:45 jayme@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on kubestagemaster2003.codfw.wmnet with reason: host reimage
  • 07:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61245 and previous config saved to /var/cache/conftool/dbconfig/20240426-074243-arnaudb.json
  • 07:30 jayme@cumin1002: START - Cookbook sre.hosts.reimage for host kubestagemaster2003.codfw.wmnet with OS bullseye
  • 07:29 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 07:29 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 07:29 jayme@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) kubestagemaster2003.codfw.wmnet on all recursors
  • 07:28 jayme@cumin1002: START - Cookbook sre.dns.wipe-cache kubestagemaster2003.codfw.wmnet on all recursors
  • 07:28 jayme@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 07:28 jayme@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 07:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61244 and previous config saved to /var/cache/conftool/dbconfig/20240426-072737-arnaudb.json
  • 07:21 jayme@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM kubestagemaster2003.codfw.wmnet - jayme@cumin1002"
  • 07:19 jayme@cumin1002: START - Cookbook sre.dns.netbox
  • 07:18 jayme@cumin1002: START - Cookbook sre.ganeti.makevm for new host kubestagemaster2003.codfw.wmnet
  • 07:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61243 and previous config saved to /var/cache/conftool/dbconfig/20240426-071231-arnaudb.json
  • 07:08 hashar: Restarting CI Jenkins
  • 07:05 gmodena@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventstreams: apply
  • 07:05 gmodena@deploy1002: helmfile [eqiad] START helmfile.d/services/eventstreams: apply
  • 07:03 gmodena@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventstreams: apply
  • 07:02 gmodena@deploy1002: helmfile [codfw] START helmfile.d/services/eventstreams: apply
  • 07:01 gmodena@deploy1002: helmfile [staging] DONE helmfile.d/services/eventstreams: apply
  • 07:01 gmodena@deploy1002: helmfile [staging] START helmfile.d/services/eventstreams: apply
  • 06:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61242 and previous config saved to /var/cache/conftool/dbconfig/20240426-065726-arnaudb.json
  • 06:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61241 and previous config saved to /var/cache/conftool/dbconfig/20240426-064220-arnaudb.json
  • 06:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2163 (T352010)', diff saved to https://phabricator.wikimedia.org/P61240 and previous config saved to /var/cache/conftool/dbconfig/20240426-062340-ladsgroup.json
  • 06:23 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 06:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 06:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T352010)', diff saved to https://phabricator.wikimedia.org/P61239 and previous config saved to /var/cache/conftool/dbconfig/20240426-062317-ladsgroup.json
  • 06:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P61238 and previous config saved to /var/cache/conftool/dbconfig/20240426-060810-ladsgroup.json
  • 05:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P61237 and previous config saved to /var/cache/conftool/dbconfig/20240426-055303-ladsgroup.json
  • 05:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T352010)', diff saved to https://phabricator.wikimedia.org/P61236 and previous config saved to /var/cache/conftool/dbconfig/20240426-053756-ladsgroup.json
  • 01:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T352010)', diff saved to https://phabricator.wikimedia.org/P61235 and previous config saved to /var/cache/conftool/dbconfig/20240426-015212-ladsgroup.json
  • 01:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 01:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 01:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T352010)', diff saved to https://phabricator.wikimedia.org/P61234 and previous config saved to /var/cache/conftool/dbconfig/20240426-015149-ladsgroup.json
  • 01:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P61233 and previous config saved to /var/cache/conftool/dbconfig/20240426-013642-ladsgroup.json
  • 01:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P61232 and previous config saved to /var/cache/conftool/dbconfig/20240426-012135-ladsgroup.json
  • 01:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T352010)', diff saved to https://phabricator.wikimedia.org/P61231 and previous config saved to /var/cache/conftool/dbconfig/20240426-010628-ladsgroup.json

2024-04-25

  • 23:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61230 and previous config saved to /var/cache/conftool/dbconfig/20240425-231201-ladsgroup.json
  • 22:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P61229 and previous config saved to /var/cache/conftool/dbconfig/20240425-225654-ladsgroup.json
  • 22:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P61228 and previous config saved to /var/cache/conftool/dbconfig/20240425-224146-ladsgroup.json
  • 22:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61227 and previous config saved to /var/cache/conftool/dbconfig/20240425-222638-ladsgroup.json
  • 22:23 brett: Extend prometheus1005 and prometheus1006 logical volume by an extra 60G due to disk filling up
  • 19:33 ebernhardson: T363516 started manual rebuild of enwiki titlesuggest indices in eqiad
  • 19:12 dancy@deploy1002: Finished scap: Testing (duration: 08m 47s)
  • 19:03 dancy@deploy1002: Started scap: Testing
  • 18:23 brennen@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.43.0-wmf.2 refs T361396
  • 18:22 logmsgbot: nshahquinn-wmf@deploy1002 Finished deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided) (duration: 00m 07s)
  • 18:22 logmsgbot: nshahquinn-wmf@deploy1002 Started deploy [airflow-dags/analytics_product@0e9fd9a]: (no justification provided)
  • 18:08 brennen: train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group2
  • 17:58 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2162 (T352010)', diff saved to https://phabricator.wikimedia.org/P61223 and previous config saved to /var/cache/conftool/dbconfig/20240425-175802-ladsgroup.json
  • 17:58 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 17:57 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 17:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61222 and previous config saved to /var/cache/conftool/dbconfig/20240425-175739-ladsgroup.json
  • 17:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P61221 and previous config saved to /var/cache/conftool/dbconfig/20240425-174233-ladsgroup.json
  • 17:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P61220 and previous config saved to /var/cache/conftool/dbconfig/20240425-172725-ladsgroup.json
  • 17:16 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cephosd1001.eqiad.wmnet with OS bullseye
  • 17:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2147 (T352010)', diff saved to https://phabricator.wikimedia.org/P61219 and previous config saved to /var/cache/conftool/dbconfig/20240425-171329-ladsgroup.json
  • 17:13 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 17:13 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 17:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61218 and previous config saved to /var/cache/conftool/dbconfig/20240425-171218-ladsgroup.json
  • 16:34 mutante: releases1003 - docker and containerd restarted by manually starting wmf_auto_restart services
  • 15:38 dancy@deploy1002: Finished scap: Testing (duration: 08m 44s)
  • 15:34 mforns@deploy1002: Finished deploy [airflow-dags/analytics@b17acd0]: (no justification provided) (duration: 00m 27s)
  • 15:33 mforns@deploy1002: Started deploy [airflow-dags/analytics@b17acd0]: (no justification provided)
  • 15:29 dancy@deploy1002: Started scap: Testing
  • 15:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2155.codfw.wmnet with OS bookworm
  • 15:27 dancy@deploy1002: sync-world aborted: Testing (duration: 01m 33s)
  • 15:26 klausman@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-eqiad: Java 11 security updates - klausman@cumin1002
  • 15:25 dancy@deploy1002: Started scap: Testing
  • 15:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T352010)', diff saved to https://phabricator.wikimedia.org/P61216 and previous config saved to /var/cache/conftool/dbconfig/20240425-151120-ladsgroup.json
  • 15:12 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 15:12 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 15:11 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 15:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T352010)', diff saved to https://phabricator.wikimedia.org/P61215 and previous config saved to /var/cache/conftool/dbconfig/20240425-151041-ladsgroup.json
  • 15:07 klausman@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-eqiad: Java 11 security updates - klausman@cumin1002
  • 15:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2155.codfw.wmnet with reason: host reimage
  • 15:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2155.codfw.wmnet with reason: host reimage
  • 14:59 klausman@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin1002
  • 14:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P61214 and previous config saved to /var/cache/conftool/dbconfig/20240425-145534-ladsgroup.json
  • 14:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 20:00:00 on db2187.codfw.wmnet with reason: Host has hardware issues
  • 14:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 20:00:00 on db2187.codfw.wmnet with reason: Host has hardware issues
  • 14:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: Host has hardware issues
  • 14:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1234.eqiad.wmnet with reason: Host has hardware issues
  • 14:41 klausman@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:ml-cache-codfw: Java 11 security updates - klausman@cumin1002
  • 14:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P61213 and previous config saved to /var/cache/conftool/dbconfig/20240425-144027-ladsgroup.json
  • 14:29 moritzm: installing Java 11 security updates
  • 14:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T352010)', diff saved to https://phabricator.wikimedia.org/P61212 and previous config saved to /var/cache/conftool/dbconfig/20240425-142520-ladsgroup.json
  • 14:21 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bookworm
  • 14:15 arnaudb@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=93) for host db2155.codfw.wmnet with OS bookworm
  • 14:10 root@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host backup1005.eqiad.wmnet with OS bullseye
  • 14:10 root@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - root@cumin1002"
  • 13:47 claime: UTC afternoon backports window closed
  • 13:45 root@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - root@cumin1002"
  • 13:44 cgoubert@deploy1002: Finished scap: Backport for Set conflicting gadget settings for the Cite extension (T362771) (duration: 21m 33s)
  • 13:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2155.codfw.wmnet with reason: host reimage
  • 13:26 cgoubert@deploy1002: cgoubert and wmde-fisch: Continuing with sync
  • 13:26 cgoubert@deploy1002: cgoubert and wmde-fisch: Backport for Set conflicting gadget settings for the Cite extension (T362771) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2155.codfw.wmnet with reason: host reimage
  • 13:23 root@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
  • 13:23 cgoubert@deploy1002: Started scap: Backport for Set conflicting gadget settings for the Cite extension (T362771)
  • 13:20 root@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on backup1005.eqiad.wmnet with reason: host reimage
  • 13:19 cgoubert@deploy1002: Finished scap: Backport for ClusterConfigTest: Add mw-on-k8s specific tests (duration: 14m 54s)
  • 13:09 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bookworm
  • 13:08 arnaudb@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host db2155.codfw.wmnet with OS bullseye
  • 13:07 cgoubert@deploy1002: cgoubert: Continuing with sync
  • 13:07 cgoubert@deploy1002: cgoubert: Backport for ClusterConfigTest: Add mw-on-k8s specific tests synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:04 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bullseye
  • 13:04 cgoubert@deploy1002: Started scap: Backport for ClusterConfigTest: Add mw-on-k8s specific tests
  • 12:44 arnaudb@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2155.codfw.wmnet with OS bookworm
  • 12:05 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2155.codfw.wmnet with OS bookworm
  • 12:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2155', diff saved to https://phabricator.wikimedia.org/P61211 and previous config saved to /var/cache/conftool/dbconfig/20240425-120409-arnaudb.json
  • 12:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db[2155,2187].codfw.wmnet with reason: T362746
  • 12:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on db[2155,2187].codfw.wmnet with reason: T362746
  • 12:02 root@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host backup1005.eqiad.wmnet with OS bullseye
  • 11:37 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 11:37 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 11:37 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 11:36 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 11:17 root@cumin1002: START - Cookbook sre.hosts.reimage for host backup1005.eqiad.wmnet with OS bullseye
  • 11:15 root@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host backup1005.eqiad.wmnet with OS bookworm
  • 11:10 root@cumin1002: START - Cookbook sre.hosts.reimage for host backup1005.eqiad.wmnet with OS bookworm
  • 10:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61208 and previous config saved to /var/cache/conftool/dbconfig/20240425-103802-arnaudb.json
  • 10:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61207 and previous config saved to /var/cache/conftool/dbconfig/20240425-102255-arnaudb.json
  • 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61206 and previous config saved to /var/cache/conftool/dbconfig/20240425-100748-arnaudb.json
  • 09:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61205 and previous config saved to /var/cache/conftool/dbconfig/20240425-095459-arnaudb.json
  • 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61204 and previous config saved to /var/cache/conftool/dbconfig/20240425-095242-arnaudb.json
  • 09:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61203 and previous config saved to /var/cache/conftool/dbconfig/20240425-093954-arnaudb.json
  • 09:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61202 and previous config saved to /var/cache/conftool/dbconfig/20240425-093735-arnaudb.json
  • 09:36 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1004.wikimedia.org
  • 09:29 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab1004.wikimedia.org
  • 09:29 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab1003.wikimedia.org
  • 09:24 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61201 and previous config saved to /var/cache/conftool/dbconfig/20240425-092448-arnaudb.json
  • 09:24 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 09:22 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab1003.wikimedia.org
  • 09:22 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host gitlab2003.wikimedia.org
  • 09:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db1160 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61200 and previous config saved to /var/cache/conftool/dbconfig/20240425-092229-arnaudb.json
  • 09:21 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cephosd1001.eqiad.wmnet with reason: host reimage
  • 09:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1160.eqiad.wmnet with OS bookworm
  • 09:17 jmm@cumin2002: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:cloudelastic
  • 09:16 jelto@cumin1002: START - Cookbook sre.hosts.reboot-single for host gitlab2003.wikimedia.org
  • 09:16 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics: apply
  • 09:15 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics: apply
  • 09:15 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.reboot-runner (exit_code=0) rolling reboot on A:gitlab-runner
  • 09:13 jmm@cumin2002: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:cloudelastic
  • 09:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61199 and previous config saved to /var/cache/conftool/dbconfig/20240425-090942-arnaudb.json
  • 09:07 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas-eqiad
  • 09:06 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics: apply
  • 09:06 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas-eqiad
  • 09:05 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics: apply
  • 09:04 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 09:04 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-analytics-external: apply
  • 09:02 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host cephosd1001.eqiad.wmnet with OS bullseye
  • 09:01 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 08:59 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-analytics-external: apply
  • 08:58 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-main: apply
  • 08:57 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-main: apply
  • 08:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1160.eqiad.wmnet with reason: host reimage
  • 08:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61198 and previous config saved to /var/cache/conftool/dbconfig/20240425-085437-arnaudb.json
  • 08:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1160.eqiad.wmnet with reason: host reimage
  • 08:54 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-main: apply
  • 08:54 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-main: apply
  • 08:51 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/eventgate-logging-external: apply
  • 08:50 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/eventgate-logging-external: apply
  • 08:48 jmm@cumin2002: END (PASS) - Cookbook sre.ldap.roll-restart-reboot-replica (exit_code=0) rolling restart_daemons on A:ldap-replicas-codfw
  • 08:47 jmm@cumin2002: START - Cookbook sre.ldap.roll-restart-reboot-replica rolling restart_daemons on A:ldap-replicas-codfw
  • 08:42 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1160.eqiad.wmnet with OS bookworm
  • 08:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1160.eqiad.wmnet with reason: T362746
  • 08:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1160.eqiad.wmnet with reason: T362746
  • 08:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1160', diff saved to https://phabricator.wikimedia.org/P61197 and previous config saved to /var/cache/conftool/dbconfig/20240425-083956-arnaudb.json
  • 08:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db1241 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61196 and previous config saved to /var/cache/conftool/dbconfig/20240425-083931-arnaudb.json
  • 08:38 jelto@cumin1002: START - Cookbook sre.gitlab.reboot-runner rolling reboot on A:gitlab-runner
  • 08:34 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/eventgate-logging-external: apply
  • 08:32 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/eventgate-logging-external: apply
  • 08:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1241.eqiad.wmnet with OS bookworm
  • 08:23 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-logging-external: apply
  • 08:23 jayme@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-logging-external: apply
  • 08:23 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics-external: apply
  • 08:22 jayme@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-analytics-external: apply
  • 08:22 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-analytics: apply
  • 08:21 jayme@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-analytics: apply
  • 08:20 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/eventgate-main: apply
  • 08:19 jayme@deploy1002: helmfile [staging] START helmfile.d/services/eventgate-main: apply
  • 08:15 jmm@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-eqiad
  • 08:11 jmm@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-eqiad
  • 08:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1241.eqiad.wmnet with reason: host reimage
  • 08:02 hashar@deploy1002: Finished scap: Backport for logging: do not explicitly set blackhole handler (T228838) (duration: 16m 17s)
  • 08:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1241.eqiad.wmnet with reason: host reimage
  • 07:59 jmm@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies (exit_code=0) rolling restart_daemons on A:swift-fe-codfw
  • 07:56 jmm@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-ms-proxies rolling restart_daemons on A:swift-fe-codfw
  • 07:50 hashar@deploy1002: hashar: Continuing with sync
  • 07:48 hashar@deploy1002: hashar: Backport for logging: do not explicitly set blackhole handler (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:47 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1241.eqiad.wmnet with OS bookworm
  • 07:45 hashar@deploy1002: Started scap: Backport for logging: do not explicitly set blackhole handler (T228838)
  • 07:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1241', diff saved to https://phabricator.wikimedia.org/P61195 and previous config saved to /var/cache/conftool/dbconfig/20240425-074516-arnaudb.json
  • 07:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1241.eqiad.wmnet with reason: T362746
  • 07:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1241.eqiad.wmnet with reason: T362746
  • 07:38 jmm@cumin2002: END (PASS) - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies (exit_code=0) rolling restart_daemons on A:thanos-fe
  • 07:33 jmm@cumin2002: START - Cookbook sre.swift.roll-restart-reboot-swift-thanos-proxies rolling restart_daemons on A:thanos-fe
  • 07:15 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 07:08 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 06:58 moritzm: installing glibc security updates
  • 06:34 moritzm: uninstalling redis on netbox hosts, it uses the central Redis servers for a while now
  • 05:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T352010)', diff saved to https://phabricator.wikimedia.org/P61194 and previous config saved to /var/cache/conftool/dbconfig/20240425-055431-ladsgroup.json
  • 05:54 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 05:54 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 05:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61193 and previous config saved to /var/cache/conftool/dbconfig/20240425-055408-ladsgroup.json
  • 05:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P61192 and previous config saved to /var/cache/conftool/dbconfig/20240425-053901-ladsgroup.json
  • 05:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2161 (T352010)', diff saved to https://phabricator.wikimedia.org/P61191 and previous config saved to /var/cache/conftool/dbconfig/20240425-053608-ladsgroup.json
  • 05:36 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 05:35 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 05:35 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61190 and previous config saved to /var/cache/conftool/dbconfig/20240425-053545-ladsgroup.json
  • 05:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P61189 and previous config saved to /var/cache/conftool/dbconfig/20240425-052354-ladsgroup.json
  • 05:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P61188 and previous config saved to /var/cache/conftool/dbconfig/20240425-052038-ladsgroup.json
  • 05:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61187 and previous config saved to /var/cache/conftool/dbconfig/20240425-050845-ladsgroup.json
  • 05:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P61186 and previous config saved to /var/cache/conftool/dbconfig/20240425-050531-ladsgroup.json
  • 04:50 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61185 and previous config saved to /var/cache/conftool/dbconfig/20240425-045023-ladsgroup.json

2024-04-24

  • 21:52 dzahn@cumin2002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: security release T363349
  • 21:36 ryankemper: [Elastic] T361268 Pooled new hosts: `elastic110[3-7]`
  • 21:35 ryankemper@puppetmaster1001: conftool action : set/weight=10:pooled=yes; selector: name=elastic110[3-7]\.eqiad\.wmnet
  • 20:38 denisse: Disabling Puppet on the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 20:38 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 20:37 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on prometheus6002.drmrs.wmnet,prometheus5002.eqsin.wmnet,prometheus3003.esams.wmnet,prometheus4002.ulsfo.wmnet with reason: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 20:37 denisse: Downtiming the Prometheus PoP hosts as part of the cergen to CFSSL migration - T360414
  • 20:24 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:24 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:32 dzahn@cumin2002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: security release T363349
  • 19:27 cstone: payments-wiki upgraded from 1895e43b to c7ab847d
  • 19:15 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:15 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:14 eevans@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002
  • 19:08 inflatador: bking@deploy1002 stop `consumer-cloudelastic` release to test alerting T359213
  • 19:07 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:06 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:03 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T352010)', diff saved to https://phabricator.wikimedia.org/P61181 and previous config saved to /var/cache/conftool/dbconfig/20240424-190237-ladsgroup.json
  • 19:03 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 19:02 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 19:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61180 and previous config saved to /var/cache/conftool/dbconfig/20240424-190214-ladsgroup.json
  • 18:57 amastilovic@deploy1002: Finished deploy [airflow-dags/analytics@3f994d5]: (no justification provided) (duration: 00m 28s)
  • 18:57 amastilovic@deploy1002: Started deploy [airflow-dags/analytics@3f994d5]: (no justification provided)
  • 18:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61179 and previous config saved to /var/cache/conftool/dbconfig/20240424-184707-ladsgroup.json
  • 18:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P61178 and previous config saved to /var/cache/conftool/dbconfig/20240424-183200-ladsgroup.json
  • 18:20 brennen@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.2 refs T361396
  • 18:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61177 and previous config saved to /var/cache/conftool/dbconfig/20240424-181653-ladsgroup.json
  • 18:14 cstone: payments-wiki upgraded from fb0367a4 to 1895e43b
  • 18:03 brennen: train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group1
  • 17:41 btullis@cumin1002: END (PASS) - Cookbook sre.wdqs.restart (exit_code=0)
  • 17:20 btullis@cumin1002: START - Cookbook sre.wdqs.restart
  • 17:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2154 (T352010)', diff saved to https://phabricator.wikimedia.org/P61176 and previous config saved to /var/cache/conftool/dbconfig/20240424-170444-ladsgroup.json
  • 17:04 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 17:04 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 17:04 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61175 and previous config saved to /var/cache/conftool/dbconfig/20240424-170421-ladsgroup.json
  • 17:03 hmonroy@deploy1002: Finished scap: Backport for [hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211) (duration: 20m 36s)
  • 16:51 eevans@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-eqiad: Apply truststore changes — T352647 - eevans@cumin1002
  • 16:50 hmonroy@deploy1002: musikanimal and hmonroy: Continuing with sync
  • 16:49 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host stat1010.eqiad.wmnet
  • 16:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P61174 and previous config saved to /var/cache/conftool/dbconfig/20240424-164914-ladsgroup.json
  • 16:45 hmonroy@deploy1002: musikanimal and hmonroy: Backport for [hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 16:43 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 16:42 hmonroy@deploy1002: Started scap: Backport for [hewiki] enable CodeMirrorV6 and CodeMirrorLineNumberingNamespaces (T357795 T347211)
  • 16:42 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1010.eqiad.wmnet
  • 16:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P61173 and previous config saved to /var/cache/conftool/dbconfig/20240424-163407-ladsgroup.json
  • 16:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61172 and previous config saved to /var/cache/conftool/dbconfig/20240424-161859-ladsgroup.json
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61171 and previous config saved to /var/cache/conftool/dbconfig/20240424-161112-arnaudb.json
  • 16:05 sukhe: running authdns-update
  • 16:04 elukey@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 16:01 elukey@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 15:59 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 15:58 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns6002.wikimedia.org
  • 15:56 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns6002.wikimedia.org
  • 15:56 jclark@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts aqs1014.eqiad.wmnet
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61170 and previous config saved to /var/cache/conftool/dbconfig/20240424-155607-arnaudb.json
  • 15:55 jclark@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1014.eqiad.wmnet
  • 15:55 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns5004.wikimedia.org
  • 15:54 jclark@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts aqs1014.eqiad.wmnet
  • 15:54 SandraEbele_: Deployed refinery using scap, then deployed onto hdfs.
  • 15:54 jclark@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts aqs1014.eqiad.wmnet
  • 15:53 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns5004.wikimedia.org
  • 15:51 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns5003.wikimedia.org
  • 15:50 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns5003.wikimedia.org
  • 15:48 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns4004.wikimedia.org
  • 15:47 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns4004.wikimedia.org
  • 15:45 jmm@cumin2002: END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector
  • 15:44 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns4003.wikimedia.org
  • 15:42 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns4003.wikimedia.org
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61169 and previous config saved to /var/cache/conftool/dbconfig/20240424-154101-arnaudb.json
  • 15:40 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns3004.wikimedia.org
  • 15:38 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns3004.wikimedia.org
  • 15:37 jmm@cumin2002: START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector
  • 15:37 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
  • 15:35 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns3003.wikimedia.org
  • 15:34 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns3003.wikimedia.org
  • 15:32 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns2006.wikimedia.org
  • 15:31 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns2006.wikimedia.org
  • 15:29 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns2005.wikimedia.org
  • 15:28 ebysans@deploy1002: Finished deploy [analytics/refinery@a5f2b25] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a5f2b252] (duration: 02m 51s)
  • 15:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61168 and previous config saved to /var/cache/conftool/dbconfig/20240424-152811-arnaudb.json
  • 15:27 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns2005.wikimedia.org
  • 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61167 and previous config saved to /var/cache/conftool/dbconfig/20240424-152556-arnaudb.json
  • 15:26 ebysans@deploy1002: Started deploy [analytics/refinery@a5f2b25] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@a5f2b252]
  • 15:25 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns2004.wikimedia.org
  • 15:23 ebysans@deploy1002: Finished deploy [analytics/refinery@a5f2b25] (thin): Regular analytics weekly train THIN [analytics/refinery@a5f2b252] (duration: 03m 36s)
  • 15:23 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns2004.wikimedia.org
  • 15:21 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns1006.wikimedia.org
  • 15:20 ebysans@deploy1002: Started deploy [analytics/refinery@a5f2b25] (thin): Regular analytics weekly train THIN [analytics/refinery@a5f2b252]
  • 15:20 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns1006.wikimedia.org
  • 15:18 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns1005.wikimedia.org
  • 15:16 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns1005.wikimedia.org
  • 15:14 sukhe@cumin1002: conftool action : set/pooled=yes; selector: name=dns1004.wikimedia.org
  • 15:13 sukhe@cumin1002: conftool action : set/pooled=no; selector: name=dns1004.wikimedia.org
  • 15:13 ebysans@deploy1002: Finished deploy [analytics/refinery@a5f2b25]: Regular analytics weekly train [analytics/refinery@a5f2b252] (duration: 12m 13s)
  • 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61166 and previous config saved to /var/cache/conftool/dbconfig/20240424-151304-arnaudb.json
  • 15:11 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 15:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61165 and previous config saved to /var/cache/conftool/dbconfig/20240424-151050-arnaudb.json
  • 15:10 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 15:10 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 15:10 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:10 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp4037.ulsfo.wmnet
  • 15:09 fabfur: depooling cp4037 to test tls connection to kafka cluster (T358109)
  • 15:01 ebysans@deploy1002: Started deploy [analytics/refinery@a5f2b25]: Regular analytics weekly train [analytics/refinery@a5f2b252]
  • 15:00 SandraEbele_: starting refinery deployment
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61164 and previous config saved to /var/cache/conftool/dbconfig/20240424-145758-arnaudb.json
  • 14:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db1190 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61163 and previous config saved to /var/cache/conftool/dbconfig/20240424-145545-arnaudb.json
  • 14:55 dancy@deploy1002: Installation of scap version "4.79.0" completed for 325 hosts
  • 14:54 dancy@deploy1002: Installing scap version "4.79.0" for 325 hosts
  • 14:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1190.eqiad.wmnet with OS bookworm
  • 14:52 moritzm: installing exim4/spamassassin on MXes
  • 14:45 moritzm: installing php7.4 security updates (as shipped in Debian, not our internal component)
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61162 and previous config saved to /var/cache/conftool/dbconfig/20240424-144252-arnaudb.json
  • 14:38 sukhe: rolling restart of haproxy, pdns-rec and ntp on A:dnsbox
  • 14:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1190.eqiad.wmnet with reason: host reimage
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61160 and previous config saved to /var/cache/conftool/dbconfig/20240424-142905-arnaudb.json
  • 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61159 and previous config saved to /var/cache/conftool/dbconfig/20240424-142747-arnaudb.json
  • 14:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1190.eqiad.wmnet with reason: host reimage
  • 14:26 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 14:21 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns6001.wikimedia.org
  • 14:20 sukhe: restarting pdns-rec on dns6001
  • 14:19 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns6001.wikimedia.org
  • 14:19 moritzm: import djangorestframework 3.14.0-2+wmf12u1 to apt.wikimedia.org (bug fix needed for Bitu 0.7.0, https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1068747)
  • 14:14 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1190.eqiad.wmnet with OS bookworm
  • 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61158 and previous config saved to /var/cache/conftool/dbconfig/20240424-141400-arnaudb.json
  • 14:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1190.eqiad.wmnet with reason: T362746
  • 14:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1190.eqiad.wmnet with reason: T362746
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1190', diff saved to https://phabricator.wikimedia.org/P61157 and previous config saved to /var/cache/conftool/dbconfig/20240424-141305-arnaudb.json
  • 14:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db1199 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61156 and previous config saved to /var/cache/conftool/dbconfig/20240424-141241-arnaudb.json
  • 14:11 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:59 urbanecm@deploy1002: Finished scap: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620), Revert "Updated uzwiktionary project namespace name and site name to follow" (T362620) (duration: 14m 08s)
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61155 and previous config saved to /var/cache/conftool/dbconfig/20240424-135854-arnaudb.json
  • 13:56 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:restbase-codfw: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1199.eqiad.wmnet with OS bookworm
  • 13:49 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching restbase2021.codfw.wmnet: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:48 urbanecm@deploy1002: urbanecm and nmw03: Continuing with sync
  • 13:48 urbanecm@deploy1002: urbanecm and nmw03: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620), Revert "Updated uzwiktionary project namespace name and site name to follow" (T362620) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdeb
  • 13:47 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns (exit_code=0) rolling restart_daemons on A:wikidough and A:wikidough
  • 13:45 urbanecm@deploy1002: Started scap: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620), Revert "Updated uzwiktionary project namespace name and site name to follow" (T362620)
  • 13:44 urbanecm@deploy1002: Sync cancelled.
  • 13:44 urbanecm@deploy1002: urbanecm and nmw03: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620), Revert "Updated uzwiktionary project namespace name and site name to follow" (T362620) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdeb
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61153 and previous config saved to /var/cache/conftool/dbconfig/20240424-134349-arnaudb.json
  • 13:42 urbanecm@deploy1002: Started scap: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620), Revert "Updated uzwiktionary project namespace name and site name to follow" (T362620)
  • 13:40 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2021.codfw.wmnet: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:39 urbanecm@deploy1002: Sync cancelled.
  • 13:38 elukey@cumin1002: END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching restbase2021.codfw.wmnet: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:37 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 13:37 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 13:37 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching restbase2021.codfw.wmnet: Deploy new TLS Truststore for PKI - elukey@cumin1002
  • 13:36 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 13:36 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 13:35 urbanecm@deploy1002: urbanecm and nmw03: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1199.eqiad.wmnet with reason: host reimage
  • 13:34 sukhe@cumin1002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum
  • 13:32 urbanecm@deploy1002: Started scap: Backport for Enabled subpages for main namespace in ptwikimedia (T362300), Updated uzwiktionary project namespace name and site name to follow Uzbek grammar (T362620)
  • 13:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1199.eqiad.wmnet with reason: host reimage
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61152 and previous config saved to /var/cache/conftool/dbconfig/20240424-132841-arnaudb.json
  • 13:25 urbanecm@deploy1002: Finished scap: Backport for Growth: Enable Levelling up features on all wikis (T348086), WikiEduDashboard: allow removal when course is not synced (T363187), WikiEduDashboard: allow removal when course is not synced (T363187) (duration: 20m 21s)
  • 13:23 jmm@cumin2002: END (PASS) - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors (exit_code=0) rolling restart_daemons on A:logstash-collector
  • 13:18 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1199.eqiad.wmnet with OS bookworm
  • 13:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1199.eqiad.wmnet with reason: T362746
  • 13:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1199.eqiad.wmnet with reason: T362746
  • 13:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1199', diff saved to https://phabricator.wikimedia.org/P61151 and previous config saved to /var/cache/conftool/dbconfig/20240424-131702-arnaudb.json
  • 13:15 jmm@cumin2002: START - Cookbook sre.o11y.roll-restart-reboot-logstash-collectors rolling restart_daemons on A:logstash-collector
  • 13:14 jmm@cumin2002: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:elastic-eqiad
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1242 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61150 and previous config saved to /var/cache/conftool/dbconfig/20240424-131336-arnaudb.json
  • 13:12 urbanecm@deploy1002: daimona and urbanecm: Continuing with sync
  • 13:09 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum
  • 13:07 urbanecm@deploy1002: daimona and urbanecm: Backport for Growth: Enable Levelling up features on all wikis (T348086), WikiEduDashboard: allow removal when course is not synced (T363187), WikiEduDashboard: allow removal when course is not synced (T363187) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:04 urbanecm@deploy1002: Started scap: Backport for Growth: Enable Levelling up features on all wikis (T348086), WikiEduDashboard: allow removal when course is not synced (T363187), WikiEduDashboard: allow removal when course is not synced (T363187)
  • 13:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1242.eqiad.wmnet with OS bookworm
  • 12:59 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 12:58 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 12:57 sukhe@cumin1002: START - Cookbook sre.dns.roll-restart-reboot-wikimedia-dns rolling restart_daemons on A:wikidough and A:wikidough
  • 12:52 jmm@cumin2002: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:elastic-eqiad
  • 12:26 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1242.eqiad.wmnet with OS bookworm
  • 12:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1242', diff saved to https://phabricator.wikimedia.org/P61149 and previous config saved to /var/cache/conftool/dbconfig/20240424-122520-arnaudb.json
  • 12:25 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1242.eqiad.wmnet with reason: T362746
  • 12:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1242.eqiad.wmnet with reason: T362746
  • 12:23 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 12:23 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 12:20 jmm@cumin2002: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:elastic-codfw
  • 12:10 jmm@cumin2002: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:elastic-canary
  • 12:10 jmm@cumin2002: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:elastic-canary
  • 11:50 jmm@cumin2002: END (PASS) - Cookbook sre.elasticsearch.restart-nginx (exit_code=0) rolling restart_daemons on A:relforge
  • 11:49 jmm@cumin2002: START - Cookbook sre.elasticsearch.restart-nginx rolling restart_daemons on A:relforge
  • 11:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61148 and previous config saved to /var/cache/conftool/dbconfig/20240424-113241-arnaudb.json
  • 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61147 and previous config saved to /var/cache/conftool/dbconfig/20240424-111735-arnaudb.json
  • 11:05 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database kawikisource (T363242)
  • 11:02 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61146 and previous config saved to /var/cache/conftool/dbconfig/20240424-110230-arnaudb.json
  • 10:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61145 and previous config saved to /var/cache/conftool/dbconfig/20240424-104724-arnaudb.json
  • 10:40 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database kawikisource (T363242)
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61144 and previous config saved to /var/cache/conftool/dbconfig/20240424-103922-arnaudb.json
  • 10:39 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database mswikisource (T363249)
  • 10:39 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database mswikisource (T363249)
  • 10:38 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database kaawiktionary (T363255)
  • 10:38 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database kaawiktionary (T363255)
  • 10:37 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database iglwiki (T363262)
  • 10:37 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database iglwiki (T363262)
  • 10:32 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database mywikisource (T363269)
  • 10:32 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database mywikisource (T363269)
  • 10:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61143 and previous config saved to /var/cache/conftool/dbconfig/20240424-103218-arnaudb.json
  • 10:30 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0) for database bewwiki
  • 10:30 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki for database bewwiki
  • 10:24 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61142 and previous config saved to /var/cache/conftool/dbconfig/20240424-102416-arnaudb.json
  • 10:22 taavi@cumin1002: END (PASS) - Cookbook sre.wikireplicas.add-wiki (exit_code=0)
  • 10:22 taavi@cumin1002: Added views for new wiki: kuswiki T360302
  • 10:21 taavi@cumin1002: START - Cookbook sre.wikireplicas.add-wiki
  • 10:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 151326
  • 10:18 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 151326
  • 10:17 arnaudb@cumin1002: dbctl commit (dc=all): 'db1247 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61141 and previous config saved to /var/cache/conftool/dbconfig/20240424-101713-arnaudb.json
  • 10:09 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61140 and previous config saved to /var/cache/conftool/dbconfig/20240424-100910-arnaudb.json
  • 10:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1247.eqiad.wmnet with OS bookworm
  • 09:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61139 and previous config saved to /var/cache/conftool/dbconfig/20240424-095405-arnaudb.json
  • 09:45 taavi: echo "https://en.wikipedia.org/static/images/mobile/copyright/wikipedia-tagline-ca-750k.svg" | mwscript purgeList.php --wiki enwiki # T363057
  • 09:44 taavi@deploy1002: Finished scap: Backport for logos: Update cawiki 750k logo tagline (T363057) (duration: 14m 53s)
  • 09:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1247.eqiad.wmnet with reason: host reimage
  • 09:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1247.eqiad.wmnet with reason: host reimage
  • 09:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T352010)', diff saved to https://phabricator.wikimedia.org/P61138 and previous config saved to /var/cache/conftool/dbconfig/20240424-094027-ladsgroup.json
  • 09:40 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 09:40 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 09:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T352010)', diff saved to https://phabricator.wikimedia.org/P61137 and previous config saved to /var/cache/conftool/dbconfig/20240424-094004-ladsgroup.json
  • 09:39 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61136 and previous config saved to /var/cache/conftool/dbconfig/20240424-093859-arnaudb.json
  • 09:33 taavi@deploy1002: taavi: Continuing with sync
  • 09:32 taavi@deploy1002: taavi: Backport for logos: Update cawiki 750k logo tagline (T363057) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:29 taavi@deploy1002: Started scap: Backport for logos: Update cawiki 750k logo tagline (T363057)
  • 09:29 claime: 80% of external traffix to mw-on-k8s - T362323
  • 09:28 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1247.eqiad.wmnet with OS bookworm
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1247', diff saved to https://phabricator.wikimedia.org/P61135 and previous config saved to /var/cache/conftool/dbconfig/20240424-092540-arnaudb.json
  • 09:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P61134 and previous config saved to /var/cache/conftool/dbconfig/20240424-092457-ladsgroup.json
  • 09:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1247.eqiad.wmnet with reason: T362746
  • 09:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1247.eqiad.wmnet with reason: T362746
  • 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db1248 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61133 and previous config saved to /var/cache/conftool/dbconfig/20240424-092353-arnaudb.json
  • 09:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1248.eqiad.wmnet with OS bookworm
  • 09:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P61132 and previous config saved to /var/cache/conftool/dbconfig/20240424-090950-ladsgroup.json
  • 09:08 elukey: run 'kill `pgrep -u dbad2021`' on all stat nodes to unblock puppet
  • 08:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage
  • 08:54 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 08:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T352010)', diff saved to https://phabricator.wikimedia.org/P61131 and previous config saved to /var/cache/conftool/dbconfig/20240424-085442-ladsgroup.json
  • 08:54 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 08:54 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 08:54 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 08:53 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 08:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 08:53 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 08:53 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 08:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1248.eqiad.wmnet with reason: host reimage
  • 08:52 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-eqiad
  • 08:51 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-eqiad
  • 08:47 jmm@cumin2002: END (PASS) - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas (exit_code=0) rolling restart_daemons on A:schema-codfw
  • 08:46 jmm@cumin2002: START - Cookbook sre.misc-clusters.roll-restart-reboot-eventschemas rolling restart_daemons on A:schema-codfw
  • 08:39 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1248.eqiad.wmnet with OS bookworm
  • 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1248', diff saved to https://phabricator.wikimedia.org/P61130 and previous config saved to /var/cache/conftool/dbconfig/20240424-083736-arnaudb.json
  • 08:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1248.eqiad.wmnet with reason: T362746
  • 08:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1248.eqiad.wmnet with reason: T362746
  • 08:08 filippo@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 08:08 filippo@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 06:17 moritzm: installing glibc security updates
  • 04:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2152 (T352010)', diff saved to https://phabricator.wikimedia.org/P61129 and previous config saved to /var/cache/conftool/dbconfig/20240424-045230-ladsgroup.json
  • 04:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 04:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 00:15 zabe@deploy1002: Finished scap: Backport for Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270), Update interwiki cache (duration: 13m 56s)
  • 00:04 zabe@deploy1002: zabe: Continuing with sync
  • 00:03 zabe@deploy1002: zabe: Backport for Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270), Update interwiki cache synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 00:01 zabe@deploy1002: Started scap: Backport for Set timezones for new wikis (T360310 T360303 T363263 T363256 T363250 T363243 T363270), Update interwiki cache

2024-04-23

  • 23:58 eileen: config revision changed from 75af3eb6 to dc53becd
  • 23:57 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=kawikisource --cluster=all 2>&1 | tee /tmp/kawikisource.UpdateSearchIndexConfig.log # T363085
  • 23:56 zabe@deploy1002: Finished scap: Creating kawikisource (T363085) (duration: 14m 40s)
  • 23:45 zabe@deploy1002: zabe: Continuing with sync
  • 23:44 zabe@deploy1002: zabe: Creating kawikisource (T363085) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 23:42 zabe@deploy1002: Started scap: Creating kawikisource (T363085)
  • 23:41 zabe: create Wikisource Georgian # T363085
  • 23:39 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=mswikisource --cluster=all 2>&1 | tee /tmp/mswikisource.UpdateSearchIndexConfig.log # T363039
  • 23:39 zabe@deploy1002: Finished scap: Creating mswikisource (T363039) (duration: 15m 00s)
  • 23:34 eileen: config revision changed from 974afe9c to 75af3eb6
  • 23:27 zabe@deploy1002: zabe: Continuing with sync
  • 23:26 zabe@deploy1002: zabe: Creating mswikisource (T363039) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 23:24 zabe@deploy1002: Started scap: Creating mswikisource (T363039)
  • 23:23 zabe: create Wikisource Malay # T363039
  • 23:22 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=kaawiktionary --cluster=all 2>&1 | tee /tmp/kaawiktionary.UpdateSearchIndexConfig.log # T362135
  • 23:20 zabe@deploy1002: Finished scap: Creating kaawiktionary (T362135) (duration: 13m 34s)
  • 23:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T352010)', diff saved to https://phabricator.wikimedia.org/P61128 and previous config saved to /var/cache/conftool/dbconfig/20240423-231923-ladsgroup.json
  • 23:19 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 23:19 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 23:10 zabe@deploy1002: zabe: Continuing with sync
  • 23:10 zabe@deploy1002: zabe: Creating kaawiktionary (T362135) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 23:07 zabe@deploy1002: Started scap: Creating kaawiktionary (T362135)
  • 23:06 zabe: create Wiktionary Karakalpak # T362135
  • 23:06 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host aqs1013.eqiad.wmnet
  • 23:04 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=iglwiki --cluster=all 2>&1 | tee /tmp/iglwiki.UpdateSearchIndexConfig.log # T361644
  • 23:04 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=iglwiki --cluster=all 2>&1 | tee /tmp/iglwiki.UpdateSearchIndexConfig.log # T362135
  • 23:03 zabe@deploy1002: Finished scap: Creating iglwiki (T361644) (duration: 13m 32s)
  • 23:00 eevans@cumin1002: START - Cookbook sre.hosts.reboot-single for host aqs1013.eqiad.wmnet
  • 22:52 zabe@deploy1002: zabe: Continuing with sync
  • 22:52 zabe@deploy1002: zabe: Creating iglwiki (T361644) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 22:49 zabe@deploy1002: Started scap: Creating iglwiki (T361644)
  • 22:49 zabe: create Wikipedia Igala # T361644
  • 22:47 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=mywikisource --cluster=all 2>&1 | tee /tmp/mywikisource.UpdateSearchIndexConfig.log # T361085
  • 22:47 zabe@deploy1002: Finished scap: Creating mywikisource (T361085) (duration: 13m 45s)
  • 22:36 zabe@deploy1002: zabe: Continuing with sync
  • 22:36 zabe@deploy1002: zabe: Creating mywikisource (T361085) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 22:33 zabe@deploy1002: Started scap: Creating mywikisource (T361085)
  • 22:33 zabe: create Wikisource Burmese # T361085
  • 22:29 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=kuswiki --cluster=all 2>&1 | tee /tmp/kuswiki.UpdateSearchIndexConfig.log # T359757
  • 22:28 zabe@deploy1002: Finished scap: Creating kuswiki (T359757) (duration: 14m 10s)
  • 22:18 zabe@deploy1002: zabe: Continuing with sync
  • 22:17 zabe@deploy1002: zabe: Creating kuswiki (T359757) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 22:14 zabe@deploy1002: Started scap: Creating kuswiki (T359757)
  • 22:14 zabe: create Wikipedia Kusaal # T359757
  • 22:11 zabe: zabe@mwmaint1002:~$ mwscript extensions/CirrusSearch/maintenance/UpdateSearchIndexConfig.php --wiki=bewwiki --cluster=all 2>&1 | tee /tmp/bewwiki.UpdateSearchIndexConfig.log # T357866
  • 22:11 zabe@deploy1002: Finished scap: Creating bewwiki (T357866) (duration: 12m 59s)
  • 21:58 zabe@deploy1002: Started scap: Creating bewwiki (T357866)
  • 21:58 zabe@deploy1002: Sync cancelled.
  • 21:56 zabe@deploy1002: zabe: Creating bewwiki (T357866) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:53 zabe@deploy1002: Started scap: Creating bewwiki (T357866)
  • 21:52 zabe: create Wikipedia Betawi # T357866
  • 21:46 brennen@deploy1002: Finished scap: Backport for Add afl_var_dump to AbuseLogPager::getQueryInfo (T363213) (duration: 16m 05s)
  • 21:35 brennen@deploy1002: brennen: Continuing with sync
  • 21:33 brennen@deploy1002: brennen: Backport for Add afl_var_dump to AbuseLogPager::getQueryInfo (T363213) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:30 brennen@deploy1002: Started scap: Backport for Add afl_var_dump to AbuseLogPager::getQueryInfo (T363213)
  • 21:18 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:18 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: telxius magru-eqiad - ayounsi@cumin1002"
  • 21:17 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: telxius magru-eqiad - ayounsi@cumin1002"
  • 21:17 zabe: zabe@mwmaint1002:~$ mwscript namespaceDupes.php azwikiquote --fix # T362645
  • 21:16 zabe@deploy1002: Finished scap: Backport for Added namespace alias for Azerbaijani Wikiquote (T362645) (duration: 14m 58s)
  • 21:13 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 21:06 zabe@deploy1002: zabe and nmw03: Continuing with sync
  • 21:04 zabe@deploy1002: zabe and nmw03: Backport for Added namespace alias for Azerbaijani Wikiquote (T362645) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:01 zabe@deploy1002: Started scap: Backport for Added namespace alias for Azerbaijani Wikiquote (T362645)
  • 21:01 zabe@deploy1002: Finished scap: Backport for .nvmrc: Update version from 18.17.0 to 18.20.2, Use dedicated Codex style modules (T362986), Use dedicated Codex style modules (T362986) (duration: 24m 32s)
  • 20:50 zabe@deploy1002: zabe and jdlrobson: Continuing with sync
  • 20:39 zabe@deploy1002: zabe and jdlrobson: Backport for .nvmrc: Update version from 18.17.0 to 18.20.2, Use dedicated Codex style modules (T362986), Use dedicated Codex style modules (T362986) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:36 zabe@deploy1002: Started scap: Backport for .nvmrc: Update version from 18.17.0 to 18.20.2, Use dedicated Codex style modules (T362986), Use dedicated Codex style modules (T362986)
  • 20:35 zabe@deploy1002: Finished scap: Backport for Enable night mode styles on Vector 2022 skin (T362726) (duration: 26m 08s)
  • 20:24 zabe@deploy1002: jdlrobson and zabe: Continuing with sync
  • 20:12 zabe@deploy1002: jdlrobson and zabe: Backport for Enable night mode styles on Vector 2022 skin (T362726) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:09 zabe@deploy1002: Started scap: Backport for Enable night mode styles on Vector 2022 skin (T362726)
  • 19:31 brennen@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.2 refs T361396
  • 19:15 brennen@deploy1002: Finished scap: testwikis wikis to 1.43.0-wmf.2 refs T361396 (duration: 56m 50s)
  • 18:18 brennen@deploy1002: Started scap: testwikis wikis to 1.43.0-wmf.2 refs T361396
  • 18:04 brennen: train 1.43.0-wmf.2 (T361396) status: no current blockers, rolling to group0
  • 18:03 jynus: db1208 aka matomo db (data engineering)
  • 18:02 jynus: add backup user to db1208 T349397
  • 17:09 inflatador: bking@mw1461 "restart rsyslog to reclaim fds T357616"
  • 16:46 zabe@deploy1002: Finished scap: T361041 T362529 (duration: 06m 28s)
  • 16:42 denisse@deploy1002: Finished deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.4.0 - T363141 (duration: 00m 12s)
  • 16:42 denisse@deploy1002: Started deploy [librenms/librenms@f049593]: Upgrade LibreNMS to 24.4.0 - T363141
  • 16:41 denisse: Upgrading LibreNMS in production - T363141
  • 16:41 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 16:39 zabe@deploy1002: Started scap: T361041 T362529
  • 16:37 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 16:30 taavi: disable puppet on P:mediawiki::webserver to deploy https://gerrit.wikimedia.org/r/c/operations/puppet/+/1020920 and https://gerrit.wikimedia.org/r/c/operations/puppet/+/1023436
  • 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61125 and previous config saved to /var/cache/conftool/dbconfig/20240423-162709-arnaudb.json
  • 16:13 denisse: Backing up LibreNMS DB - T363141
  • 16:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61124 and previous config saved to /var/cache/conftool/dbconfig/20240423-161204-arnaudb.json
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61123 and previous config saved to /var/cache/conftool/dbconfig/20240423-155657-arnaudb.json
  • 15:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:51 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 15:49 brennen@deploy1002: Finished deploy [phabricator/deployment@12abb76]: deploy phab1004 for T363174 (duration: 00m 32s)
  • 15:49 brennen@deploy1002: Started deploy [phabricator/deployment@12abb76]: deploy phab1004 for T363174
  • 15:48 brennen@deploy1002: Finished deploy [phabricator/deployment@12abb76]: test deploy phab2002 for T363174 (duration: 00m 32s)
  • 15:48 brennen@deploy1002: Started deploy [phabricator/deployment@12abb76]: test deploy phab2002 for T363174
  • 15:46 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab.wmfusercontent.org with reason: T363174
  • 15:46 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab.wmfusercontent.org with reason: T363174
  • 15:46 dzahn@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on phabricator.wikimedia.org with reason: T363174
  • 15:45 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on phabricator.wikimedia.org with reason: T363174
  • 15:45 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab2002.codfw.wmnet with reason: T363174
  • 15:45 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab2002.codfw.wmnet with reason: T363174
  • 15:44 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: T363174
  • 15:44 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: T363174
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61122 and previous config saved to /var/cache/conftool/dbconfig/20240423-154152-arnaudb.json
  • 15:30 klausman@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:30 klausman@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 15:28 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 15:27 ladsgroup@deploy1002: Finished scap: Backport for logos: revert back the tagline (T363165) (duration: 13m 30s)
  • 15:27 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 15:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T352010)', diff saved to https://phabricator.wikimedia.org/P61121 and previous config saved to /var/cache/conftool/dbconfig/20240423-152725-ladsgroup.json
  • 15:27 klausman@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 15:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61120 and previous config saved to /var/cache/conftool/dbconfig/20240423-152646-arnaudb.json
  • 15:19 jmm@cumin2002: END (FAIL) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=1) rolling restart_daemons on A:durum
  • 15:19 moritzm: restarting FPM on phab1004
  • 15:16 ladsgroup@deploy1002: ladsgroup: Continuing with sync
  • 15:16 ladsgroup@deploy1002: ladsgroup: Backport for logos: revert back the tagline (T363165) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:14 ladsgroup@deploy1002: Started scap: Backport for logos: revert back the tagline (T363165)
  • 15:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 100%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61119 and previous config saved to /var/cache/conftool/dbconfig/20240423-151240-arnaudb.json
  • 15:12 ladsgroup@deploy1002: Finished scap: Backport for logos: Add the override for 1M variant of fawiki (T363165) (duration: 14m 28s)
  • 15:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P61118 and previous config saved to /var/cache/conftool/dbconfig/20240423-151216-ladsgroup.json
  • 15:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2136 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61117 and previous config saved to /var/cache/conftool/dbconfig/20240423-151140-arnaudb.json
  • 15:10 arnaudb@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host db2136.codfw.wmnet with OS bookworm
  • 15:08 jmm@cumin2002: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum
  • 15:05 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 15:03 jmm@cumin2002: END (PASS) - Cookbook sre.dns.roll-restart-reboot-durum (exit_code=0) rolling restart_daemons on A:durum-drmrs
  • 15:01 ladsgroup@deploy1002: ladsgroup: Continuing with sync
  • 15:01 ladsgroup@deploy1002: ladsgroup: Backport for logos: Add the override for 1M variant of fawiki (T363165) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:59 vgutierrez: repool ncredir6001
  • 14:58 ladsgroup@deploy1002: Started scap: Backport for logos: Add the override for 1M variant of fawiki (T363165)
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 75%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61116 and previous config saved to /var/cache/conftool/dbconfig/20240423-145734-arnaudb.json
  • 14:57 ladsgroup@deploy1002: Finished scap: Backport for logos: Add fawiki logo for 1,000,000 article (T363165) (duration: 17m 38s)
  • 14:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P61115 and previous config saved to /var/cache/conftool/dbconfig/20240423-145709-ladsgroup.json
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 100%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61114 and previous config saved to /var/cache/conftool/dbconfig/20240423-145603-arnaudb.json
  • 14:53 jmm@cumin2002: START - Cookbook sre.dns.roll-restart-reboot-durum rolling restart_daemons on A:durum-drmrs
  • 14:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2136.codfw.wmnet with reason: host reimage
  • 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2136.codfw.wmnet with reason: host reimage
  • 14:47 jclark@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts parse1002.eqiad.wmnet
  • 14:46 ladsgroup@deploy1002: ladsgroup: Continuing with sync
  • 14:44 vgutierrez: depool ncredir6001
  • 14:44 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon1003.wikimedia.org
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 50%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61113 and previous config saved to /var/cache/conftool/dbconfig/20240423-144229-arnaudb.json
  • 14:42 ladsgroup@deploy1002: ladsgroup: Backport for logos: Add fawiki logo for 1,000,000 article (T363165) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T352010)', diff saved to https://phabricator.wikimedia.org/P61112 and previous config saved to /var/cache/conftool/dbconfig/20240423-144202-ladsgroup.json
  • 14:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 75%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61111 and previous config saved to /var/cache/conftool/dbconfig/20240423-144057-arnaudb.json
  • 14:39 ladsgroup@deploy1002: Started scap: Backport for logos: Add fawiki logo for 1,000,000 article (T363165)
  • 14:38 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon1003.wikimedia.org
  • 14:35 jclark@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts parse1002.eqiad.wmnet
  • 14:35 jclark@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['parse1002.eqiad.wmnet']
  • 14:35 jclark@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['parse1002.eqiad.wmnet']
  • 14:30 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host netmon2002.wikimedia.org
  • 14:29 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2136.codfw.wmnet with OS bookworm
  • 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2140 (re)pooling @ 25%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61110 and previous config saved to /var/cache/conftool/dbconfig/20240423-142723-arnaudb.json
  • 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2136', diff saved to https://phabricator.wikimedia.org/P61109 and previous config saved to /var/cache/conftool/dbconfig/20240423-142630-arnaudb.json
  • 14:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2136.codfw.wmnet with reason: T362746
  • 14:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2136.codfw.wmnet with reason: T362746
  • 14:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 50%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61108 and previous config saved to /var/cache/conftool/dbconfig/20240423-142551-arnaudb.json
  • 14:25 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2140.codfw.wmnet with OS bookworm
  • 14:24 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host netmon2002.wikimedia.org
  • 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir
  • 14:14 zabe@deploy1002: Finished scap: Backport for Update interwiki cache (T363093) (duration: 13m 41s)
  • 14:14 jmm@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir
  • 14:13 effie: upload prometheus-memcached-exporter_0.14.2-2~wmf1_amd64 to bookworm-wikimedia - T350807
  • 14:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 25%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61107 and previous config saved to /var/cache/conftool/dbconfig/20240423-141045-arnaudb.json
  • 14:10 jmm@cumin2002: END (PASS) - Cookbook sre.cdn.roll-restart-reboot-ncredir (exit_code=0) rolling restart_daemons on A:ncredir-ulsfo
  • 14:09 jmm@cumin2002: START - Cookbook sre.cdn.roll-restart-reboot-ncredir rolling restart_daemons on A:ncredir-ulsfo
  • 14:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2140.codfw.wmnet with reason: host reimage
  • 14:03 zabe@deploy1002: zabe: Continuing with sync
  • 14:03 zabe@deploy1002: zabe: Backport for Update interwiki cache (T363093) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2140.codfw.wmnet with reason: host reimage
  • 14:00 zabe@deploy1002: Started scap: Backport for Update interwiki cache (T363093)
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 10%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61106 and previous config saved to /var/cache/conftool/dbconfig/20240423-135540-arnaudb.json
  • 13:43 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2140.codfw.wmnet with OS bookworm
  • 13:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2140.codfw.wmnet with reason: T362746
  • 13:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2140.codfw.wmnet with reason: T362746
  • 13:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2147 (re)pooling @ 5%: Post reimage', diff saved to https://phabricator.wikimedia.org/P61105 and previous config saved to /var/cache/conftool/dbconfig/20240423-134034-arnaudb.json
  • 13:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2147.codfw.wmnet with OS bookworm
  • 13:38 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 13:37 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 13:36 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 13:36 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 13:35 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 13:35 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 13:35 moritzm: installing glibc security updates
  • 13:34 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 13:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 100%: Sanitarium master', diff saved to https://phabricator.wikimedia.org/P61103 and previous config saved to /var/cache/conftool/dbconfig/20240423-132633-arnaudb.json
  • 13:19 sukhe: running authdns-update for T362921
  • 13:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2147.codfw.wmnet with reason: host reimage
  • 13:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2147.codfw.wmnet with reason: host reimage
  • 13:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 75%: Sanitarium master', diff saved to https://phabricator.wikimedia.org/P61102 and previous config saved to /var/cache/conftool/dbconfig/20240423-131128-arnaudb.json
  • 12:58 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2147.codfw.wmnet with OS bookworm
  • 12:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2147.codfw.wmnet with reason: T362746
  • 12:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2147.codfw.wmnet with reason: T362746
  • 12:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2147', diff saved to https://phabricator.wikimedia.org/P61101 and previous config saved to /var/cache/conftool/dbconfig/20240423-125703-arnaudb.json
  • 12:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 (re)pooling @ 50%: Sanitarium master', diff saved to https://phabricator.wikimedia.org/P61100 and previous config saved to /var/cache/conftool/dbconfig/20240423-125622-arnaudb.json
  • 12:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db2155.codfw.wmnet with reason: Reimage db2155
  • 12:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on db2155.codfw.wmnet with reason: Reimage db2155
  • 12:54 arnaudb@cumin1002: dbctl commit (dc=all): 'db2155 depool', diff saved to https://phabricator.wikimedia.org/P61099 and previous config saved to /var/cache/conftool/dbconfig/20240423-125430-arnaudb.json
  • 12:45 hashar@deploy1002: Finished deploy [gerrit/gerrit@ff51759]: Remove registerStyleModule() for Gerrit 3.8 - T354886 (duration: 00m 07s)
  • 12:17 taavi@deploy1002: taavi: Continuing with sync
  • 12:17 taavi@deploy1002: taavi: Backport for Add cawiki 750k logo (T363057) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 12:13 taavi@deploy1002: Started scap: Backport for Add cawiki 750k logo (T363057)
  • 11:47 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1414.eqiad.wmnet|mw1415.eqiad.wmnet|mw1416.eqiad.wmnet|mw1448.eqiad.wmnet|mw1449.eqiad.wmnet),cluster=kubernetes,service=kubesvc
  • 11:47 claime: Pooling and uncordoning mw1414.eqiad.wmnet,mw1415.eqiad.wmnet,mw1416.eqiad.wmnet,mw1448.eqiad.wmnet,mw1449.eqiad.wmnet - T351074
  • 11:39 claime: Running homer 'cr*eqiad*' commit 'T351074'
  • 11:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host has hardware issues
  • 11:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host has hardware issues
  • 11:38 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1415.eqiad.wmnet with OS bullseye
  • 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1448.eqiad.wmnet with OS bullseye
  • 11:33 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1416.eqiad.wmnet with OS bullseye
  • 11:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1449.eqiad.wmnet with OS bullseye
  • 11:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1414.eqiad.wmnet with OS bullseye
  • 11:21 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1415.eqiad.wmnet with reason: host reimage
  • 11:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1448.eqiad.wmnet with reason: host reimage
  • 11:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1416.eqiad.wmnet with reason: host reimage
  • 11:12 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1449.eqiad.wmnet with reason: host reimage
  • 11:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1414.eqiad.wmnet with reason: host reimage
  • 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1449.eqiad.wmnet with reason: host reimage
  • 11:07 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1448.eqiad.wmnet with reason: host reimage
  • 11:07 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1416.eqiad.wmnet with reason: host reimage
  • 11:06 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1415.eqiad.wmnet with reason: host reimage
  • 11:06 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1414.eqiad.wmnet with reason: host reimage
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 100%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61098 and previous config saved to /var/cache/conftool/dbconfig/20240423-105812-arnaudb.json
  • 10:55 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1449.eqiad.wmnet with OS bullseye
  • 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1448.eqiad.wmnet with OS bullseye
  • 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1416.eqiad.wmnet with OS bullseye
  • 10:53 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1415.eqiad.wmnet with OS bullseye
  • 10:53 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1414.eqiad.wmnet with OS bullseye
  • 10:45 claime: Depooling mw1414.eqiad.wmnet,mw1415.eqiad.wmnet,mw1416.eqiad.wmnet,mw1448.eqiad.wmnet,mw1449.eqiad.wmnet for reimage to kubernetes - T351074
  • 10:43 jayme: kubectl cordon parse1002.eqiad.wmnet - T363086
  • 10:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 75%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61097 and previous config saved to /var/cache/conftool/dbconfig/20240423-104306-arnaudb.json
  • 10:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2196.codfw.wmnet
  • 10:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 50%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61094 and previous config saved to /var/cache/conftool/dbconfig/20240423-102801-arnaudb.json
  • 10:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1245.eqiad.wmnet with reason: T360116
  • 10:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1245.eqiad.wmnet with reason: T360116
  • 10:22 btullis@deploy1002: Finished deploy [analytics/hdfs-tools/deploy@3618aab]: (no justification provided) (duration: 00m 11s)
  • 10:22 btullis@deploy1002: Started deploy [analytics/hdfs-tools/deploy@3618aab]: (no justification provided)
  • 10:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2196.codfw.wmnet
  • 10:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1220.eqiad.wmnet
  • 10:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 25%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61093 and previous config saved to /var/cache/conftool/dbconfig/20240423-101255-arnaudb.json
  • 09:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 15%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61092 and previous config saved to /var/cache/conftool/dbconfig/20240423-095749-arnaudb.json
  • 09:52 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 09:52 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 09:44 hashar@deploy1002: Finished scap: Backport for logging: always register udp2log handlers (T228838) (duration: 15m 11s)
  • 09:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 10%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61091 and previous config saved to /var/cache/conftool/dbconfig/20240423-094244-arnaudb.json
  • 09:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P61090 and previous config saved to /var/cache/conftool/dbconfig/20240423-094030-arnaudb.json
  • 09:39 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1220.eqiad.wmnet
  • 09:33 hashar@deploy1002: hashar: Continuing with sync
  • 09:31 hashar@deploy1002: hashar: Backport for logging: always register udp2log handlers (T228838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:29 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1179.eqiad.wmnet
  • 09:29 hashar@deploy1002: Started scap: Backport for logging: always register udp2log handlers (T228838)
  • 09:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db2172 (re)pooling @ 5%: post upgrade repool', diff saved to https://phabricator.wikimedia.org/P61089 and previous config saved to /var/cache/conftool/dbconfig/20240423-092738-arnaudb.json
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P61088 and previous config saved to /var/cache/conftool/dbconfig/20240423-092525-arnaudb.json
  • 09:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2172.codfw.wmnet with OS bookworm
  • 09:18 hashar@deploy1002: Finished deploy [gerrit/gerrit@8b4ae00]: wm-zuul-status: filter based solely on change number - T358253 (duration: 00m 07s)
  • 09:18 hashar@deploy1002: Started deploy [gerrit/gerrit@8b4ae00]: wm-zuul-status: filter based solely on change number - T358253
  • 09:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P61087 and previous config saved to /var/cache/conftool/dbconfig/20240423-091019-arnaudb.json
  • 09:04 hashar: Backport & config window completed
  • 09:04 godog: delete tags for docker-registry.discovery.wmnet/jaeger-es-index-cleaner - T344953
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2172.codfw.wmnet with reason: host reimage
  • 09:02 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1179.eqiad.wmnet
  • 09:00 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2172.codfw.wmnet with reason: host reimage
  • 08:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2215.codfw.wmnet
  • 08:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 25%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P61086 and previous config saved to /var/cache/conftool/dbconfig/20240423-085514-arnaudb.json
  • 08:54 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc2002.wikimedia.org
  • 08:50 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc2002.wikimedia.org
  • 08:43 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2172.codfw.wmnet with OS bookworm
  • 08:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2172.codfw.wmnet with reason: T362746
  • 08:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2172.codfw.wmnet with reason: T362746
  • 08:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2172', diff saved to https://phabricator.wikimedia.org/P61085 and previous config saved to /var/cache/conftool/dbconfig/20240423-084146-arnaudb.json
  • 08:40 hashar@deploy1002: Finished scap: Backport for ParserOutput: don't complain if TOCHTML is unset from ParserCache (T363107) (duration: 16m 13s)
  • 08:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db2206 (re)pooling @ 10%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P61084 and previous config saved to /var/cache/conftool/dbconfig/20240423-084008-arnaudb.json
  • 08:33 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2215.codfw.wmnet
  • 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2206.codfw.wmnet
  • 08:28 hashar@deploy1002: cscott and hashar: Continuing with sync
  • 08:28 hashar@deploy1002: cscott and hashar: Backport for ParserOutput: don't complain if TOCHTML is unset from ParserCache (T363107) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:28 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2206.codfw.wmnet
  • 08:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2206.codfw.wmnet with reason: T362746
  • 08:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2206.codfw.wmnet with reason: T362746
  • 08:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2206', diff saved to https://phabricator.wikimedia.org/P61083 and previous config saved to /var/cache/conftool/dbconfig/20240423-082621-arnaudb.json
  • 08:24 hashar@deploy1002: Started scap: Backport for ParserOutput: don't complain if TOCHTML is unset from ParserCache (T363107)
  • 08:20 hashar@deploy1002: Sync cancelled.
  • 08:20 hashar@deploy1002: hashar and cscott: Backport for ParserOutput: don't complain if TOCHTML is unset from ParserCache (T363107) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:16 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host irc1002.wikimedia.org
  • 08:15 hashar@deploy1002: Started scap: Backport for ParserOutput: don't complain if TOCHTML is unset from ParserCache (T363107)
  • 08:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2191.codfw.wmnet
  • 08:12 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host irc1002.wikimedia.org
  • 08:12 kartik@deploy1002: Finished scap: Backport for CX: Initialize publishNamespace for CXTarget (T349959) (duration: 44m 53s)
  • 08:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2191.codfw.wmnet
  • 08:04 godog: restore sre business hour escalation policy - T350192
  • 07:59 kartik@deploy1002: kartik: Continuing with sync
  • 07:58 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2131.codfw.wmnet
  • 07:42 kartik@deploy1002: kartik: Backport for CX: Initialize publishNamespace for CXTarget (T349959) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:38 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2131.codfw.wmnet
  • 07:37 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 07:37 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 07:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T352010)', diff saved to https://phabricator.wikimedia.org/P61082 and previous config saved to /var/cache/conftool/dbconfig/20240423-073658-ladsgroup.json
  • 07:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2115.codfw.wmnet
  • 07:27 kartik@deploy1002: Started scap: Backport for CX: Initialize publishNamespace for CXTarget (T349959)
  • 07:22 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2115.codfw.wmnet
  • 07:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P61081 and previous config saved to /var/cache/conftool/dbconfig/20240423-072151-ladsgroup.json
  • 07:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P61080 and previous config saved to /var/cache/conftool/dbconfig/20240423-070643-ladsgroup.json
  • 06:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T352010)', diff saved to https://phabricator.wikimedia.org/P61079 and previous config saved to /var/cache/conftool/dbconfig/20240423-065136-ladsgroup.json
  • 03:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2130 (T352010)', diff saved to https://phabricator.wikimedia.org/P61078 and previous config saved to /var/cache/conftool/dbconfig/20240423-034652-ladsgroup.json
  • 03:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 03:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T352010)', diff saved to https://phabricator.wikimedia.org/P61077 and previous config saved to /var/cache/conftool/dbconfig/20240423-034628-ladsgroup.json
  • 03:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P61076 and previous config saved to /var/cache/conftool/dbconfig/20240423-033120-ladsgroup.json
  • 03:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P61075 and previous config saved to /var/cache/conftool/dbconfig/20240423-031613-ladsgroup.json
  • 03:05 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.25 (duration: 05m 37s)
  • 03:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T352010)', diff saved to https://phabricator.wikimedia.org/P61074 and previous config saved to /var/cache/conftool/dbconfig/20240423-030106-ladsgroup.json
  • 00:24 sukhe@cumin2002: dbctl commit (dc=all): 'depool db1246', diff saved to https://phabricator.wikimedia.org/P61073 and previous config saved to /var/cache/conftool/dbconfig/20240423-002413-sukhe.json

2024-04-22

  • 22:28 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T352010)', diff saved to https://phabricator.wikimedia.org/P61072 and previous config saved to /var/cache/conftool/dbconfig/20240422-222830-ladsgroup.json
  • 22:28 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 22:28 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 16:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db2116 (T352010)', diff saved to https://phabricator.wikimedia.org/P61071 and previous config saved to /var/cache/conftool/dbconfig/20240422-162340-ladsgroup.json
  • 16:23 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 16:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 14:58 mforns@deploy1002: Finished deploy [airflow-dags/analytics@70946de]: (no justification provided) (duration: 00m 27s)
  • 14:57 mforns@deploy1002: Started deploy [airflow-dags/analytics@70946de]: (no justification provided)
  • 14:41 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1234.eqiad.wmnet with reason: Down
  • 14:41 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1234.eqiad.wmnet with reason: Down
  • 14:10 mforns@deploy1002: Finished deploy [analytics/refinery@a7af5a6] (hadoop-test): Deploying Commons Impact Metrics dump queries TEST [analytics/refinery@a7af5a6b] (duration: 02m 36s)
  • 14:08 mforns@deploy1002: Started deploy [analytics/refinery@a7af5a6] (hadoop-test): Deploying Commons Impact Metrics dump queries TEST [analytics/refinery@a7af5a6b]
  • 14:05 mforns@deploy1002: Finished deploy [analytics/refinery@a7af5a6] (thin): Deploy Commons Impact Metrics dumps queries THIN [analytics/refinery@a7af5a6b] (duration: 03m 34s)
  • 14:01 mforns@deploy1002: Started deploy [analytics/refinery@a7af5a6] (thin): Deploy Commons Impact Metrics dumps queries THIN [analytics/refinery@a7af5a6b]
  • 14:00 mforns@deploy1002: Finished deploy [analytics/refinery@a7af5a6]: Deploying Commons Impact Metrics dumps queries [analytics/refinery@a7af5a6b] (duration: 13m 02s)
  • 13:47 mforns@deploy1002: Started deploy [analytics/refinery@a7af5a6]: Deploying Commons Impact Metrics dumps queries [analytics/refinery@a7af5a6b]
  • 11:26 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 11:26 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 11:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T352010)', diff saved to https://phabricator.wikimedia.org/P61069 and previous config saved to /var/cache/conftool/dbconfig/20240422-112625-ladsgroup.json
  • 11:11 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P61068 and previous config saved to /var/cache/conftool/dbconfig/20240422-111117-ladsgroup.json
  • 10:56 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P61067 and previous config saved to /var/cache/conftool/dbconfig/20240422-105610-ladsgroup.json
  • 10:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T352010)', diff saved to https://phabricator.wikimedia.org/P61066 and previous config saved to /var/cache/conftool/dbconfig/20240422-104102-ladsgroup.json
  • 05:30 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 05:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 05:30 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P61065 and previous config saved to /var/cache/conftool/dbconfig/20240422-053006-ladsgroup.json
  • 05:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P61064 and previous config saved to /var/cache/conftool/dbconfig/20240422-051459-ladsgroup.json
  • 04:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P61063 and previous config saved to /var/cache/conftool/dbconfig/20240422-045952-ladsgroup.json
  • 04:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P61062 and previous config saved to /var/cache/conftool/dbconfig/20240422-044444-ladsgroup.json
  • 01:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T352010)', diff saved to https://phabricator.wikimedia.org/P61061 and previous config saved to /var/cache/conftool/dbconfig/20240422-010520-ladsgroup.json
  • 01:05 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 01:05 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 01:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T352010)', diff saved to https://phabricator.wikimedia.org/P61060 and previous config saved to /var/cache/conftool/dbconfig/20240422-010457-ladsgroup.json
  • 00:49 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P61059 and previous config saved to /var/cache/conftool/dbconfig/20240422-004950-ladsgroup.json
  • 00:34 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P61058 and previous config saved to /var/cache/conftool/dbconfig/20240422-003442-ladsgroup.json
  • 00:19 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T352010)', diff saved to https://phabricator.wikimedia.org/P61057 and previous config saved to /var/cache/conftool/dbconfig/20240422-001933-ladsgroup.json

2024-04-21

  • 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P61056 and previous config saved to /var/cache/conftool/dbconfig/20240421-193927-ladsgroup.json
  • 19:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 19:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P61055 and previous config saved to /var/cache/conftool/dbconfig/20240421-193904-ladsgroup.json
  • 19:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P61054 and previous config saved to /var/cache/conftool/dbconfig/20240421-192356-ladsgroup.json
  • 19:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P61053 and previous config saved to /var/cache/conftool/dbconfig/20240421-190849-ladsgroup.json
  • 18:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P61052 and previous config saved to /var/cache/conftool/dbconfig/20240421-185342-ladsgroup.json
  • 14:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T352010)', diff saved to https://phabricator.wikimedia.org/P61050 and previous config saved to /var/cache/conftool/dbconfig/20240421-142433-ladsgroup.json
  • 14:24 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 14:24 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 14:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T352010)', diff saved to https://phabricator.wikimedia.org/P61049 and previous config saved to /var/cache/conftool/dbconfig/20240421-142411-ladsgroup.json
  • 14:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P61048 and previous config saved to /var/cache/conftool/dbconfig/20240421-140904-ladsgroup.json
  • 13:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P61047 and previous config saved to /var/cache/conftool/dbconfig/20240421-135356-ladsgroup.json
  • 13:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T352010)', diff saved to https://phabricator.wikimedia.org/P61046 and previous config saved to /var/cache/conftool/dbconfig/20240421-133848-ladsgroup.json
  • 09:19 topranks: putting eqiad <-> codfw traffic back on primary 100G transport link following service restoration and stability T362486
  • 09:18 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P61045 and previous config saved to /var/cache/conftool/dbconfig/20240421-091828-ladsgroup.json
  • 09:18 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 09:18 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 01:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T352010)', diff saved to https://phabricator.wikimedia.org/P61044 and previous config saved to /var/cache/conftool/dbconfig/20240421-015952-ladsgroup.json
  • 01:59 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 01:59 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 01:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T352010)', diff saved to https://phabricator.wikimedia.org/P61043 and previous config saved to /var/cache/conftool/dbconfig/20240421-015929-ladsgroup.json
  • 01:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P61042 and previous config saved to /var/cache/conftool/dbconfig/20240421-014422-ladsgroup.json
  • 01:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P61041 and previous config saved to /var/cache/conftool/dbconfig/20240421-012913-ladsgroup.json
  • 01:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T352010)', diff saved to https://phabricator.wikimedia.org/P61040 and previous config saved to /var/cache/conftool/dbconfig/20240421-011406-ladsgroup.json

2024-04-20

  • 21:15 taavi: restart gerrit
  • 14:02 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 14:02 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T352010)', diff saved to https://phabricator.wikimedia.org/P61039 and previous config saved to /var/cache/conftool/dbconfig/20240420-131519-ladsgroup.json
  • 13:15 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 13:15 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 13:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T352010)', diff saved to https://phabricator.wikimedia.org/P61038 and previous config saved to /var/cache/conftool/dbconfig/20240420-131455-ladsgroup.json
  • 12:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P61037 and previous config saved to /var/cache/conftool/dbconfig/20240420-125948-ladsgroup.json
  • 09:27 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3 days, 0:00:00 on db1234.eqiad.wmnet with reason: Down
  • 09:26 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 3 days, 0:00:00 on db1234.eqiad.wmnet with reason: Down
  • 09:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Down', diff saved to https://phabricator.wikimedia.org/P61034 and previous config saved to /var/cache/conftool/dbconfig/20240420-092358-ladsgroup.json
  • 07:24 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 07:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 00:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T352010)', diff saved to https://phabricator.wikimedia.org/P61033 and previous config saved to /var/cache/conftool/dbconfig/20240420-003950-ladsgroup.json
  • 00:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 00:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 00:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T352010)', diff saved to https://phabricator.wikimedia.org/P61032 and previous config saved to /var/cache/conftool/dbconfig/20240420-003927-ladsgroup.json
  • 00:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P61031 and previous config saved to /var/cache/conftool/dbconfig/20240420-002420-ladsgroup.json
  • 00:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P61030 and previous config saved to /var/cache/conftool/dbconfig/20240420-000912-ladsgroup.json

2024-04-19

  • 23:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T352010)', diff saved to https://phabricator.wikimedia.org/P61029 and previous config saved to /var/cache/conftool/dbconfig/20240419-235405-ladsgroup.json
  • 22:30 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 22:30 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 21:03 taavi: taavi@mwmaint1002 ~ $ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=metawiki --logwiki=metawiki Kou.i5h 'Renamed user 8356771833137' # T362942
  • 21:02 taavi: taavi@mwmaint1002 ~ $ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=eowiki --logwiki=metawiki 'Gzsimonfbi' 'Renamed user 2409354752759' # T362941
  • 20:22 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:21 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:12 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:12 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:56 bking@cumin2002: END (PASS) - Cookbook sre.wdqs.data-transfer (exit_code=0) (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 19:51 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 19:49 jforrester@deploy1002: Finished deploy [integration/docroot@c090350]: I1c1c25 trivial CI fix (duration: 00m 06s)
  • 19:49 jforrester@deploy1002: Started deploy [integration/docroot@c090350]: I1c1c25 trivial CI fix
  • 19:41 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:16 ryankemper: [WDQS] T363004 Restarted wdqs2012 to clear out its in-application-memory ban lists (it had pybal's twisted user agent banned)
  • 18:50 ryankemper: [WDQS] T363004 Restarted wdqs2010 and wdqs2024 to clear out their in-application-memory ban lists
  • 18:34 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T362508, journal in uncertain state) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 18:33 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:33 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding more reverse v6 INCLUDES into dns for magru transport links - cmooney@cumin1002"
  • 18:32 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding more reverse v6 INCLUDES into dns for magru transport links - cmooney@cumin1002"
  • 18:24 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 18:08 cmooney@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:20:00 on mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 with reason: disabling oob link on mr1-ulsfo to stop the SSH attempts long enough to get a homer run in
  • 18:07 cmooney@cumin1002: START - Cookbook sre.hosts.downtime for 0:20:00 on mr1-ulsfo,mr1-ulsfo IPv6,mr1-ulsfo.oob,mr1-ulsfo.oob IPv6 with reason: disabling oob link on mr1-ulsfo to stop the SSH attempts long enough to get a homer run in
  • 17:58 sukhe: sudo cookbook -d sre.dns.netbox "test"
  • 17:49 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 17:48 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 16:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 16:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 16:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 16:01 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 16:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 16:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 16:00 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 15:48 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 15:44 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 15:44 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 15:44 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T352010)', diff saved to https://phabricator.wikimedia.org/P61028 and previous config saved to /var/cache/conftool/dbconfig/20240419-154430-ladsgroup.json
  • 15:35 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 15:35 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 15:35 cmooney@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 15:35 cmooney@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 15:29 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P61027 and previous config saved to /var/cache/conftool/dbconfig/20240419-152922-ladsgroup.json
  • 15:14 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P61026 and previous config saved to /var/cache/conftool/dbconfig/20240419-151415-ladsgroup.json
  • 15:11 vgutierrez: repool ncredir2001
  • 14:59 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T352010)', diff saved to https://phabricator.wikimedia.org/P61025 and previous config saved to /var/cache/conftool/dbconfig/20240419-145907-ladsgroup.json
  • 14:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T352010)', diff saved to https://phabricator.wikimedia.org/P61023 and previous config saved to /var/cache/conftool/dbconfig/20240419-142726-ladsgroup.json
  • 14:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P61022 and previous config saved to /var/cache/conftool/dbconfig/20240419-141218-ladsgroup.json
  • 13:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P61021 and previous config saved to /var/cache/conftool/dbconfig/20240419-135711-ladsgroup.json
  • 13:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T352010)', diff saved to https://phabricator.wikimedia.org/P61020 and previous config saved to /var/cache/conftool/dbconfig/20240419-134204-ladsgroup.json
  • 13:28 kevinbazira@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 13:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.copy (exit_code=0) Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 13:27 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 13:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.copy (exit_code=0) Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 13:22 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 13:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.copy (exit_code=0) Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 13:22 arnaudb@cumin1002: START - Cookbook sre.mysql.copy Will create a clone of db1178.eqiad.wmnet onto db1178.eqiad.wmnet
  • 12:19 vgutierrez: depool ncredir2001
  • 12:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T352010)', diff saved to https://phabricator.wikimedia.org/P61018 and previous config saved to /var/cache/conftool/dbconfig/20240419-120853-ladsgroup.json
  • 12:08 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 12:08 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 12:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T352010)', diff saved to https://phabricator.wikimedia.org/P61017 and previous config saved to /var/cache/conftool/dbconfig/20240419-120831-ladsgroup.json
  • 12:01 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:00 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding network interface DNS magru. - cmooney@cumin1002"
  • 12:00 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding network interface DNS magru. - cmooney@cumin1002"
  • 11:57 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 11:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P61015 and previous config saved to /var/cache/conftool/dbconfig/20240419-115323-ladsgroup.json
  • 11:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P61014 and previous config saved to /var/cache/conftool/dbconfig/20240419-113816-ladsgroup.json
  • 11:28 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:28 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts matomo1002.eqiad.wmnet
  • 11:28 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:28 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: matomo1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 11:27 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 11:27 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: matomo1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 11:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T352010)', diff saved to https://phabricator.wikimedia.org/P61013 and previous config saved to /var/cache/conftool/dbconfig/20240419-112309-ladsgroup.json
  • 11:21 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 11:15 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts matomo1002.eqiad.wmnet
  • 11:05 taavi@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cloudnet2007-dev.codfw.wmnet
  • 11:00 taavi@cumin1002: START - Cookbook sre.hosts.reboot-single for host cloudnet2007-dev.codfw.wmnet
  • 10:41 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T352010)', diff saved to https://phabricator.wikimedia.org/P61010 and previous config saved to /var/cache/conftool/dbconfig/20240419-104144-ladsgroup.json
  • 10:41 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 10:41 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P61009 and previous config saved to /var/cache/conftool/dbconfig/20240419-102438-root.json
  • 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P61008 and previous config saved to /var/cache/conftool/dbconfig/20240419-100933-root.json
  • 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P61007 and previous config saved to /var/cache/conftool/dbconfig/20240419-095427-root.json
  • 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P61006 and previous config saved to /var/cache/conftool/dbconfig/20240419-093921-root.json
  • 09:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P61005 and previous config saved to /var/cache/conftool/dbconfig/20240419-092415-root.json
  • 09:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P61004 and previous config saved to /var/cache/conftool/dbconfig/20240419-090910-root.json
  • 09:03 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
  • 09:02 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
  • 08:55 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
  • 08:54 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
  • 08:54 kevinbazira@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 08:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1194 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P61003 and previous config saved to /var/cache/conftool/dbconfig/20240419-085404-root.json
  • 08:40 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 08:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1194.eqiad.wmnet with OS bookworm
  • 07:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
  • 07:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1194.eqiad.wmnet with reason: host reimage
  • 07:38 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1194.eqiad.wmnet with OS bookworm
  • 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1194', diff saved to https://phabricator.wikimedia.org/P61001 and previous config saved to /var/cache/conftool/dbconfig/20240419-073638-root.json
  • 07:24 moritzm: installing Linux 6.1.85 on Bookworm hosts
  • 07:15 moritzm: installing PHP 7.4 security updates on cloudweb and bullseye snapshot hosts
  • 07:03 moritzm: imported PHP 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf11u2 to component/php74 (backport of latest PHP security fixes)
  • 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60999 and previous config saved to /var/cache/conftool/dbconfig/20240419-065142-root.json
  • 06:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T352010)', diff saved to https://phabricator.wikimedia.org/P60998 and previous config saved to /var/cache/conftool/dbconfig/20240419-063847-ladsgroup.json
  • 06:38 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 06:38 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 06:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T352010)', diff saved to https://phabricator.wikimedia.org/P60997 and previous config saved to /var/cache/conftool/dbconfig/20240419-063825-ladsgroup.json
  • 06:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60996 and previous config saved to /var/cache/conftool/dbconfig/20240419-063636-root.json
  • 06:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60995 and previous config saved to /var/cache/conftool/dbconfig/20240419-062317-ladsgroup.json
  • 06:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60994 and previous config saved to /var/cache/conftool/dbconfig/20240419-062130-root.json
  • 06:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60993 and previous config saved to /var/cache/conftool/dbconfig/20240419-060810-ladsgroup.json
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60992 and previous config saved to /var/cache/conftool/dbconfig/20240419-060625-root.json
  • 05:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T352010)', diff saved to https://phabricator.wikimedia.org/P60991 and previous config saved to /var/cache/conftool/dbconfig/20240419-055303-ladsgroup.json
  • 05:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60990 and previous config saved to /var/cache/conftool/dbconfig/20240419-055118-root.json
  • 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60989 and previous config saved to /var/cache/conftool/dbconfig/20240419-053612-root.json
  • 05:26 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1202.eqiad.wmnet with OS bookworm
  • 05:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1202 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60988 and previous config saved to /var/cache/conftool/dbconfig/20240419-052107-root.json
  • 05:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
  • 05:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1202.eqiad.wmnet with reason: host reimage
  • 05:02 marostegui: dbmaint Upgrade s7 eqiad to Bookworm and MariaDB 10.6 T362745
  • 05:02 marostegui: dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
  • 04:50 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1202.eqiad.wmnet with OS bookworm
  • 04:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1202', diff saved to https://phabricator.wikimedia.org/P60987 and previous config saved to /var/cache/conftool/dbconfig/20240419-044906-root.json
  • 04:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 04:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1173.eqiad.wmnet with reason: Maintenance
  • 04:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 04:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2114.codfw.wmnet with reason: Maintenance

2024-04-18

  • 23:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T352010)', diff saved to https://phabricator.wikimedia.org/P60986 and previous config saved to /var/cache/conftool/dbconfig/20240418-234247-ladsgroup.json
  • 23:42 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:42 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 23:42 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P60985 and previous config saved to /var/cache/conftool/dbconfig/20240418-234225-ladsgroup.json
  • 23:27 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P60984 and previous config saved to /var/cache/conftool/dbconfig/20240418-232717-ladsgroup.json
  • 23:12 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P60983 and previous config saved to /var/cache/conftool/dbconfig/20240418-231210-ladsgroup.json
  • 23:06 mutante: graphite - switched SSL cert provider from cergen to cfssl - restarted envoyproxy
  • 22:57 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P60982 and previous config saved to /var/cache/conftool/dbconfig/20240418-225702-ladsgroup.json
  • 22:31 bking@cumin2002: END (FAIL) - Cookbook sre.wdqs.data-transfer (exit_code=99) (T362508, excessive lag) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 21:34 damilare: civicrm upgraded from 28adb4da to e95e03d9
  • 21:11 bking@cumin2002: START - Cookbook sre.wdqs.data-transfer (T362508, excessive lag) xfer wikidata from wdqs2022.codfw.wmnet -> wdqs2023.codfw.wmnet w/ force delete existing files, repooling both afterwards
  • 21:01 cjming: end of UTC late backport window
  • 21:00 cjming@deploy1002: Finished scap: Backport for Add templateeditor right to sysops in dawiki and fix typo in group name (T361461) (duration: 16m 24s)
  • 20:48 cjming@deploy1002: cjming and nmw03: Continuing with sync
  • 20:46 cjming@deploy1002: cjming and nmw03: Backport for Add templateeditor right to sysops in dawiki and fix typo in group name (T361461) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:43 cjming@deploy1002: Started scap: Backport for Add templateeditor right to sysops in dawiki and fix typo in group name (T361461)
  • 20:42 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on wdqs2023.codfw.wmnet with reason: T362508
  • 20:42 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on wdqs2023.codfw.wmnet with reason: T362508
  • 20:42 cjming@deploy1002: Finished scap: Backport for Temporarily restore wgMinervaApplyKnownTemplateHacks for cached HTML (T362747) (duration: 17m 14s)
  • 20:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T352010)', diff saved to https://phabricator.wikimedia.org/P60980 and previous config saved to /var/cache/conftool/dbconfig/20240418-203256-ladsgroup.json
  • 20:32 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 20:32 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 20:32 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P60979 and previous config saved to /var/cache/conftool/dbconfig/20240418-203234-ladsgroup.json
  • 20:30 cjming@deploy1002: jdlrobson and cjming: Continuing with sync
  • 20:27 cjming@deploy1002: jdlrobson and cjming: Backport for Temporarily restore wgMinervaApplyKnownTemplateHacks for cached HTML (T362747) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:25 cjming@deploy1002: Started scap: Backport for Temporarily restore wgMinervaApplyKnownTemplateHacks for cached HTML (T362747)
  • 20:23 cjming@deploy1002: Finished scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt)", Revert "ext-EventLogging: Add mediawiki.ip_reputation.score" (duration: 14m 56s)
  • 20:17 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60978 and previous config saved to /var/cache/conftool/dbconfig/20240418-201727-ladsgroup.json
  • 20:11 cjming@deploy1002: cjming and phuedx: Continuing with sync
  • 20:11 cjming@deploy1002: cjming and phuedx: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt)", Revert "ext-EventLogging: Add mediawiki.ip_reputation.score" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:09 cjming@deploy1002: Started scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt)", Revert "ext-EventLogging: Add mediawiki.ip_reputation.score"
  • 20:02 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60977 and previous config saved to /var/cache/conftool/dbconfig/20240418-200218-ladsgroup.json
  • 20:00 ejegg: donorwiki upgraded from 5e39bdc5 to b005071a
  • 19:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T361627)', diff saved to https://phabricator.wikimedia.org/P60976 and previous config saved to /var/cache/conftool/dbconfig/20240418-195244-marostegui.json
  • 19:47 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P60975 and previous config saved to /var/cache/conftool/dbconfig/20240418-194711-ladsgroup.json
  • 19:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P60974 and previous config saved to /var/cache/conftool/dbconfig/20240418-193737-marostegui.json
  • 19:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P60973 and previous config saved to /var/cache/conftool/dbconfig/20240418-192229-marostegui.json
  • 19:21 Amir1: dropping wikiadmin user on 10.64.% on RW es sections
  • 19:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T361627)', diff saved to https://phabricator.wikimedia.org/P60972 and previous config saved to /var/cache/conftool/dbconfig/20240418-190722-marostegui.json
  • 19:05 bking@cumin2002: conftool action : set/pooled=true; selector: dnsdisc=wdqs,name=codfw
  • 19:03 ryankemper@puppetmaster1001: conftool action : set/weight=10:pooled=yes; selector: name=elastic2088\.codfw\.wmnet
  • 19:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2217 (T361627)', diff saved to https://phabricator.wikimedia.org/P60971 and previous config saved to /var/cache/conftool/dbconfig/20240418-190249-marostegui.json
  • 19:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 19:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 19:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T361627)', diff saved to https://phabricator.wikimedia.org/P60970 and previous config saved to /var/cache/conftool/dbconfig/20240418-190226-marostegui.json
  • 18:51 dancy@deploy1002: Installation of scap version "4.78.0" completed for 330 hosts
  • 18:50 dancy@deploy1002: Installing scap version "4.78.0" for 330 hosts
  • 18:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P60969 and previous config saved to /var/cache/conftool/dbconfig/20240418-184718-marostegui.json
  • 18:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T352010)', diff saved to https://phabricator.wikimedia.org/P60968 and previous config saved to /var/cache/conftool/dbconfig/20240418-184645-ladsgroup.json
  • 18:46 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 18:46 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 18:46 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T352010)', diff saved to https://phabricator.wikimedia.org/P60967 and previous config saved to /var/cache/conftool/dbconfig/20240418-184623-ladsgroup.json
  • 18:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P60966 and previous config saved to /var/cache/conftool/dbconfig/20240418-183211-marostegui.json
  • 18:31 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60965 and previous config saved to /var/cache/conftool/dbconfig/20240418-183116-ladsgroup.json
  • 18:27 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.43.0-wmf.1 refs T361395
  • 18:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T361627)', diff saved to https://phabricator.wikimedia.org/P60964 and previous config saved to /var/cache/conftool/dbconfig/20240418-181704-marostegui.json
  • 18:16 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60963 and previous config saved to /var/cache/conftool/dbconfig/20240418-181606-ladsgroup.json
  • 18:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T361627)', diff saved to https://phabricator.wikimedia.org/P60962 and previous config saved to /var/cache/conftool/dbconfig/20240418-181450-marostegui.json
  • 18:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 18:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 18:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 18:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60961 and previous config saved to /var/cache/conftool/dbconfig/20240418-181048-marostegui.json
  • 18:09 joal@deploy1002: Finished deploy [airflow-dags/analytics@980dc72]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@980dc725] (duration: 00m 31s)
  • 18:09 joal@deploy1002: Started deploy [airflow-dags/analytics@980dc72]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@980dc725]
  • 18:01 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T352010)', diff saved to https://phabricator.wikimedia.org/P60960 and previous config saved to /var/cache/conftool/dbconfig/20240418-180059-ladsgroup.json
  • 17:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P60959 and previous config saved to /var/cache/conftool/dbconfig/20240418-175541-marostegui.json
  • 17:41 joal@deploy1002: Finished deploy [airflow-dags/analytics@0a13b42]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@0a13b420] (duration: 00m 28s)
  • 17:41 joal@deploy1002: Started deploy [airflow-dags/analytics@0a13b42]: Deploy of Analytics airflow dags for canary-events job [airflow-dags/analytics@0a13b420]
  • 17:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P60958 and previous config saved to /var/cache/conftool/dbconfig/20240418-174033-marostegui.json
  • 17:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60957 and previous config saved to /var/cache/conftool/dbconfig/20240418-172525-marostegui.json
  • 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T361627)', diff saved to https://phabricator.wikimedia.org/P60956 and previous config saved to /var/cache/conftool/dbconfig/20240418-172412-marostegui.json
  • 17:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 17:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 17:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60955 and previous config saved to /var/cache/conftool/dbconfig/20240418-172349-marostegui.json
  • 17:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P60954 and previous config saved to /var/cache/conftool/dbconfig/20240418-170842-marostegui.json
  • 16:57 kevinbazira@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 16:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P60952 and previous config saved to /var/cache/conftool/dbconfig/20240418-165334-marostegui.json
  • 16:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on matomo1002.eqiad.wmnet with reason: Migrating to new version
  • 16:44 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on matomo1002.eqiad.wmnet with reason: Migrating to new version
  • 16:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60951 and previous config saved to /var/cache/conftool/dbconfig/20240418-163827-marostegui.json
  • 16:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60950 and previous config saved to /var/cache/conftool/dbconfig/20240418-163612-marostegui.json
  • 16:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 16:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60949 and previous config saved to /var/cache/conftool/dbconfig/20240418-163600-marostegui.json
  • 16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P60948 and previous config saved to /var/cache/conftool/dbconfig/20240418-162053-marostegui.json
  • 16:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P60947 and previous config saved to /var/cache/conftool/dbconfig/20240418-160546-marostegui.json
  • 16:03 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 16:03 vgutierrez: repool ncredir2001
  • 16:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 16:01 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 16:01 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 16:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 16:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 16:01 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 15:59 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 15:59 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 15:58 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 15:57 elukey@cumin1002: END (PASS) - Cookbook sre.kafka.roll-restart-reboot-brokers (exit_code=0) rolling restart_daemons on A:kafka-logging-eqiad
  • 15:54 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 15:53 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 15:53 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 15:50 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 15:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60946 and previous config saved to /var/cache/conftool/dbconfig/20240418-155038-marostegui.json
  • 15:50 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 15:49 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 15:45 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 15:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T361627)', diff saved to https://phabricator.wikimedia.org/P60945 and previous config saved to /var/cache/conftool/dbconfig/20240418-154547-marostegui.json
  • 15:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 15:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 15:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60944 and previous config saved to /var/cache/conftool/dbconfig/20240418-154524-marostegui.json
  • 15:44 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 15:44 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 15:43 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 15:43 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 15:42 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 15:32 moritzm: installing util-linux security updates on buster
  • 15:31 elukey@cumin1002: START - Cookbook sre.kafka.roll-restart-reboot-brokers rolling restart_daemons on A:kafka-logging-eqiad
  • 15:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P60943 and previous config saved to /var/cache/conftool/dbconfig/20240418-153017-marostegui.json
  • 15:26 volans: rolling python3-wmflib upgrade to 1.2.5 across the fleet
  • 15:19 mforns@deploy1002: Finished deploy [airflow-dags/analytics@5fb4f99]: (no justification provided) (duration: 00m 32s)
  • 15:18 mforns@deploy1002: Started deploy [airflow-dags/analytics@5fb4f99]: (no justification provided)
  • 15:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P60942 and previous config saved to /var/cache/conftool/dbconfig/20240418-151510-marostegui.json
  • 15:13 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1355.eqiad.wmnet|mw1480.eqiad.wmnet|mw1481.eqiad.wmnet|mw1487.eqiad.wmnet),cluster=kubernetes,service=kubesvc
  • 15:12 claime: Pooling and uncordoning mw1355.eqiad.wmnet,mw1480.eqiad.wmnet,mw1481.eqiad.wmnet,mw1487.eqiad.wmnet - T351074
  • 15:09 elukey@cumin2002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs2012.codfw.wmnet*: Deploy new TLS Keystore - PKI - elukey@cumin2002
  • 15:04 claime: Running homer 'cr*eqiad*' commit 'T351074'
  • 15:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1480.eqiad.wmnet with OS bullseye
  • 15:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1355.eqiad.wmnet with OS bullseye
  • 15:02 elukey@cumin2002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs2012.codfw.wmnet*: Deploy new TLS Keystore - PKI - elukey@cumin2002
  • 15:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60941 and previous config saved to /var/cache/conftool/dbconfig/20240418-150003-marostegui.json
  • 14:58 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1487.eqiad.wmnet with OS bullseye
  • 14:56 elukey@cumin2002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching aqs20[09-12]*: Deploy new TLS Keystore - PKI - elukey@cumin2002
  • 14:55 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1481.eqiad.wmnet with OS bullseye
  • 14:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T361627)', diff saved to https://phabricator.wikimedia.org/P60940 and previous config saved to /var/cache/conftool/dbconfig/20240418-145512-marostegui.json
  • 14:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 14:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60939 and previous config saved to /var/cache/conftool/dbconfig/20240418-145435-marostegui.json
  • 14:49 volans: uploaded python3-wmflib_1.2.5 to apt.wikimedia.org buster-wikimedia,bullseye-wikimedia,bookworm-wikimedia
  • 14:48 moritzm: installing PHP 7.4 security updates (as packaged in Debian, not the WMF-internal build)
  • 14:45 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1480.eqiad.wmnet with reason: host reimage
  • 14:42 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1355.eqiad.wmnet with reason: host reimage
  • 14:40 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1487.eqiad.wmnet with reason: host reimage
  • 14:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P60938 and previous config saved to /var/cache/conftool/dbconfig/20240418-143928-marostegui.json
  • 14:37 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1481.eqiad.wmnet with reason: host reimage
  • 14:36 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1487.eqiad.wmnet with reason: host reimage
  • 14:35 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1355.eqiad.wmnet with reason: host reimage
  • 14:35 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1480.eqiad.wmnet with reason: host reimage
  • 14:34 elukey@cumin2002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[09-12]*: Deploy new TLS Keystore - PKI - elukey@cumin2002
  • 14:34 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1481.eqiad.wmnet with reason: host reimage
  • 14:28 elukey@cumin1002: END (ERROR) - Cookbook sre.cassandra.roll-restart (exit_code=97) for nodes matching aqs20[9-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 14:28 moritzm: installing cryptsetup bugfix updates from bookworm point release
  • 14:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P60937 and previous config saved to /var/cache/conftool/dbconfig/20240418-142420-marostegui.json
  • 14:22 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1487.eqiad.wmnet with OS bullseye
  • 14:22 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1481.eqiad.wmnet with OS bullseye
  • 14:21 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1480.eqiad.wmnet with OS bullseye
  • 14:21 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1355.eqiad.wmnet with OS bullseye
  • 14:19 moritzm: installing usrmerge bugfix updates from bookworm point release
  • 14:12 claime: Depooling mw1355.eqiad.wmnet,mw1480.eqiad.wmnet,mw1481.eqiad.wmnet,mw1487.eqiad.wmnet - T351074
  • 14:12 stevemunene@deploy1002: helmfile [eqiad] DONE helmfile.d/services/datahub: sync on main
  • 14:11 Lucas_WMDE: UTC afternoon backport+config window done
  • 14:11 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461) (duration: 16m 51s)
  • 14:09 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[9-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 14:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60936 and previous config saved to /var/cache/conftool/dbconfig/20240418-140913-marostegui.json
  • 14:08 stevemunene@deploy1002: helmfile [eqiad] START helmfile.d/services/datahub: apply on main
  • 14:08 moritzm: installing postgresql-15 security updates
  • 14:06 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
  • 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T361627)', diff saved to https://phabricator.wikimedia.org/P60935 and previous config saved to /var/cache/conftool/dbconfig/20240418-140421-marostegui.json
  • 14:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 14:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 14:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60934 and previous config saved to /var/cache/conftool/dbconfig/20240418-140359-marostegui.json
  • 14:00 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
  • 13:59 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Continuing with sync
  • 13:58 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching aqs20[02-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 13:57 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:56 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
  • 13:54 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Added extendedconfirmed and templateeditor rights to dawiki (T361461)
  • 13:52 btullis@deploy1002: helmfile [staging] START helmfile.d/services/datahub: sync on main
  • 13:51 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171) (duration: 19m 37s)
  • 13:51 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
  • 13:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P60932 and previous config saved to /var/cache/conftool/dbconfig/20240418-134852-marostegui.json
  • 13:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1024.eqiad.wmnet
  • 13:47 jynus: add grants for dbprov1005 at dbbackups (m1) T362509
  • 13:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1024.eqiad.wmnet
  • 13:40 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1023.eqiad.wmnet
  • 13:39 moritzm: installing Linux 6.1.85 on Bookworm hosts
  • 13:38 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and jhsoby: Continuing with sync
  • 13:37 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and jhsoby: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:37 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
  • 13:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1023.eqiad.wmnet
  • 13:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2025.codfw.wmnet
  • 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129', diff saved to https://phabricator.wikimedia.org/P60930 and previous config saved to /var/cache/conftool/dbconfig/20240418-133344-marostegui.json
  • 13:32 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Add 'mainpage-title-loggedin' to $wgForceUIMsgAsContentMsg (T361171)
  • 13:28 moritzm: installing apache2 security updates
  • 13:28 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2025.codfw.wmnet
  • 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2024.codfw.wmnet
  • 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2103.codfw.wmnet
  • 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2103.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 13:19 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2103.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60928 and previous config saved to /var/cache/conftool/dbconfig/20240418-131836-marostegui.json
  • 13:17 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 13:14 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2024.codfw.wmnet
  • 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2129 (T361627)', diff saved to https://phabricator.wikimedia.org/P60927 and previous config saved to /var/cache/conftool/dbconfig/20240418-131311-marostegui.json
  • 13:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 13:12 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2103.codfw.wmnet
  • 13:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 13:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60926 and previous config saved to /var/cache/conftool/dbconfig/20240418-131248-marostegui.json
  • 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 depool', diff saved to https://phabricator.wikimedia.org/P60925 and previous config saved to /var/cache/conftool/dbconfig/20240418-131027-arnaudb.json
  • 13:07 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[02-12]*: Deploy new TLS Keystore - PKI - elukey@cumin1002
  • 13:06 elukey: aqs2001's Cassandra instances moved to PKI TLS certs
  • 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2105.codfw.wmnet
  • 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2105.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 13:00 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2105.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 12:58 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 12:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P60923 and previous config saved to /var/cache/conftool/dbconfig/20240418-125739-marostegui.json
  • 12:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2023.codfw.wmnet
  • 12:54 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2105.codfw.wmnet
  • 12:54 sukhe: sudo cumin -b1 -s600 "A:dnsbox" "systemctl restart ntp.service" to pick up magru /24: T346722
  • 12:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2105 depool', diff saved to https://phabricator.wikimedia.org/P60922 and previous config saved to /var/cache/conftool/dbconfig/20240418-125338-arnaudb.json
  • 12:49 elukey: move aqs codfw cassandra instances to PKI TLS certs - T352647
  • 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2106.codfw.wmnet
  • 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2106.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60919 and previous config saved to /var/cache/conftool/dbconfig/20240418-122721-marostegui.json
  • 12:26 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 12:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2124 (T361627)', diff saved to https://phabricator.wikimedia.org/P60918 and previous config saved to /var/cache/conftool/dbconfig/20240418-122227-marostegui.json
  • 12:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 12:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 12:21 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2107.codfw.wmnet
  • 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 depool', diff saved to https://phabricator.wikimedia.org/P60917 and previous config saved to /var/cache/conftool/dbconfig/20240418-122122-arnaudb.json
  • 12:18 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 12:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 12:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 12:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 12:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60916 and previous config saved to /var/cache/conftool/dbconfig/20240418-121559-marostegui.json
  • 12:15 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host crm2001.codfw.wmnet
  • 12:14 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host crm2001.codfw.wmnet
  • 12:14 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host matomo1003.eqiad.wmnet
  • 12:13 isaranto@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 12:10 jmm@cumin2002: START - Cookbook sre.hosts.reboot-single for host matomo1003.eqiad.wmnet
  • 12:08 vgutierrez: depool ncredir2001
  • 12:06 eoghan: Switching phab1004 to use cfssl issued ssl cert https://gerrit.wikimedia.org/r/c/operations/puppet/+/1020190
  • 12:02 moritzm: installing PHP 8.2 security updates
  • 12:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P60915 and previous config saved to /var/cache/conftool/dbconfig/20240418-120051-marostegui.json
  • 12:00 isaranto@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 11:56 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 11:54 moritzm: upgrading PHP security updates on eqiad baremetal servers T362511
  • 11:52 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw2302.codfw.wmnet|mw2303.codfw.wmnet|mw2304.codfw.wmnet|mw2332.codfw.wmnet|mw2333.codfw.wmnet|mw2334.codfw.wmnet),cluster=kubernetes,service=kubesvc
  • 11:52 claime: Pooling and uncordoning mw2302.codfw.wmnet,mw2303.codfw.wmnet,mw2304.codfw.wmnet,mw2332.codfw.wmnet,mw2333.codfw.wmnet,mw2334.codfw.wmnet - T351074
  • 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P60914 and previous config saved to /var/cache/conftool/dbconfig/20240418-114544-marostegui.json
  • 11:42 claime: Running homer 'cr*codfw*' commit 'T351074'
  • 11:35 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2333.codfw.wmnet with OS bullseye
  • 11:33 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:32 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:32 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:31 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60913 and previous config saved to /var/cache/conftool/dbconfig/20240418-113037-marostegui.json
  • 11:30 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2302.codfw.wmnet with OS bullseye
  • 11:29 cgoubert@deploy1002: Finished scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308 (duration: 37m 04s)
  • 11:28 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T361627)', diff saved to https://phabricator.wikimedia.org/P60912 and previous config saved to /var/cache/conftool/dbconfig/20240418-112827-marostegui.json
  • 11:28 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T352010)', diff saved to https://phabricator.wikimedia.org/P60911 and previous config saved to /var/cache/conftool/dbconfig/20240418-112816-ladsgroup.json
  • 11:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 11:28 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 11:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 11:27 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 11:26 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1021.eqiad.wmnet
  • 11:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 11:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2304.codfw.wmnet with OS bullseye
  • 11:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60910 and previous config saved to /var/cache/conftool/dbconfig/20240418-112459-marostegui.json
  • 11:23 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2334.codfw.wmnet with OS bullseye
  • 11:20 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2332.codfw.wmnet with OS bullseye
  • 11:18 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1021.eqiad.wmnet
  • 11:16 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2303.codfw.wmnet with OS bullseye
  • 11:13 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1020.eqiad.wmnet
  • 11:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60909 and previous config saved to /var/cache/conftool/dbconfig/20240418-111132-marostegui.json
  • 11:10 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage
  • 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60908 and previous config saved to /var/cache/conftool/dbconfig/20240418-110950-marostegui.json
  • 11:08 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage
  • 11:05 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage
  • 11:03 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1020.eqiad.wmnet
  • 11:02 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage
  • 11:01 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2022.codfw.wmnet
  • 11:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage
  • 10:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage
  • 10:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60907 and previous config saved to /var/cache/conftool/dbconfig/20240418-105624-marostegui.json
  • 10:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2334.codfw.wmnet with reason: host reimage
  • 10:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2333.codfw.wmnet with reason: host reimage
  • 10:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2304.codfw.wmnet with reason: host reimage
  • 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2332.codfw.wmnet with reason: host reimage
  • 10:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P60906 and previous config saved to /var/cache/conftool/dbconfig/20240418-105441-marostegui.json
  • 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2302.codfw.wmnet with reason: host reimage
  • 10:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2303.codfw.wmnet with reason: host reimage
  • 10:52 cgoubert@deploy1002: Started scap: Redeploy mw-on-k8s with full rebuild - Fix setting php.timeout - T358308
  • 10:52 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2022.codfw.wmnet
  • 10:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2021.codfw.wmnet
  • 10:45 claime: Rebuild php7.4-fpm production images - T358308
  • 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60905 and previous config saved to /var/cache/conftool/dbconfig/20240418-104117-marostegui.json
  • 10:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2021.codfw.wmnet
  • 10:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2334.codfw.wmnet with OS bullseye
  • 10:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60904 and previous config saved to /var/cache/conftool/dbconfig/20240418-103933-marostegui.json
  • 10:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2020.codfw.wmnet
  • 10:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2333.codfw.wmnet with OS bullseye
  • 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2332.codfw.wmnet with OS bullseye
  • 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2304.codfw.wmnet with OS bullseye
  • 10:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2303.codfw.wmnet with OS bullseye
  • 10:37 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2302.codfw.wmnet with OS bullseye
  • 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T361627)', diff saved to https://phabricator.wikimedia.org/P60903 and previous config saved to /var/cache/conftool/dbconfig/20240418-103422-marostegui.json
  • 10:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60902 and previous config saved to /var/cache/conftool/dbconfig/20240418-103359-marostegui.json
  • 10:30 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2020.codfw.wmnet
  • 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60901 and previous config saved to /var/cache/conftool/dbconfig/20240418-102609-marostegui.json
  • 10:25 claime: Depooling mw2302.codfw.wmnet,mw2303.codfw.wmnet,mw2304.codfw.wmnet,mw2332.codfw.wmnet,mw2333.codfw.wmnet,mw2334.codfw.wmnet for reimage - T351074
  • 10:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P60900 and previous config saved to /var/cache/conftool/dbconfig/20240418-101852-marostegui.json
  • 10:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T355609)', diff saved to https://phabricator.wikimedia.org/P60899 and previous config saved to /var/cache/conftool/dbconfig/20240418-101841-marostegui.json
  • 10:18 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 10:18 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 10:08 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host contint1002.wikimedia.org
  • 10:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P60898 and previous config saved to /var/cache/conftool/dbconfig/20240418-100338-marostegui.json
  • 09:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host contint1002.wikimedia.org
  • 09:54 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host an-worker1087.eqiad.wmnet
  • 09:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T352010)', diff saved to https://phabricator.wikimedia.org/P60897 and previous config saved to /var/cache/conftool/dbconfig/20240418-095331-ladsgroup.json
  • 09:53 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 09:53 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 09:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60896 and previous config saved to /var/cache/conftool/dbconfig/20240418-095308-ladsgroup.json
  • 09:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60895 and previous config saved to /var/cache/conftool/dbconfig/20240418-094830-marostegui.json
  • 09:46 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host an-worker1087.eqiad.wmnet
  • 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T361627)', diff saved to https://phabricator.wikimedia.org/P60894 and previous config saved to /var/cache/conftool/dbconfig/20240418-094619-marostegui.json
  • 09:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 09:46 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60893 and previous config saved to /var/cache/conftool/dbconfig/20240418-094556-marostegui.json
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60892 and previous config saved to /var/cache/conftool/dbconfig/20240418-094235-marostegui.json
  • 09:39 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1034.eqiad.wmnet
  • 09:38 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60891 and previous config saved to /var/cache/conftool/dbconfig/20240418-093759-ladsgroup.json
  • 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2108.codfw.wmnet
  • 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2108.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:34 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2108.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:32 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P60890 and previous config saved to /var/cache/conftool/dbconfig/20240418-093049-marostegui.json
  • 09:27 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2108.codfw.wmnet
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60889 and previous config saved to /var/cache/conftool/dbconfig/20240418-092728-marostegui.json
  • 09:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60888 and previous config saved to /var/cache/conftool/dbconfig/20240418-092718-arnaudb.json
  • 09:25 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 00m 20s)
  • 09:25 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60887 and previous config saved to /var/cache/conftool/dbconfig/20240418-092504-arnaudb.json
  • 09:24 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1034.eqiad.wmnet
  • 09:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60886 and previous config saved to /var/cache/conftool/dbconfig/20240418-092252-ladsgroup.json
  • 09:22 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1028.eqiad.wmnet
  • 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2109.codfw.wmnet
  • 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2109.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:19 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2109.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:17 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P60885 and previous config saved to /var/cache/conftool/dbconfig/20240418-091541-marostegui.json
  • 09:13 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2109.codfw.wmnet
  • 09:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P60884 and previous config saved to /var/cache/conftool/dbconfig/20240418-091251-arnaudb.json
  • 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183', diff saved to https://phabricator.wikimedia.org/P60883 and previous config saved to /var/cache/conftool/dbconfig/20240418-091126-marostegui.json
  • 09:09 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@be07da9e] (duration: 02m 46s)
  • 09:08 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1028.eqiad.wmnet
  • 09:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2034.codfw.wmnet
  • 09:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60882 and previous config saved to /var/cache/conftool/dbconfig/20240418-090744-ladsgroup.json
  • 09:06 mforns@deploy1002: Started deploy [analytics/refinery@be07da9] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@be07da9e]
  • 09:06 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9] (thin): Regular analytics weekly train THIN [analytics/refinery@be07da9e] (duration: 03m 45s)
  • 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2110.codfw.wmnet
  • 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2110.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:03 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2110.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 09:02 mforns@deploy1002: Started deploy [analytics/refinery@be07da9] (thin): Regular analytics weekly train THIN [analytics/refinery@be07da9e]
  • 09:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60881 and previous config saved to /var/cache/conftool/dbconfig/20240418-090032-marostegui.json
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T361627)', diff saved to https://phabricator.wikimedia.org/P60880 and previous config saved to /var/cache/conftool/dbconfig/20240418-085922-marostegui.json
  • 08:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 08:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60879 and previous config saved to /var/cache/conftool/dbconfig/20240418-085900-marostegui.json
  • 08:57 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2110.codfw.wmnet
  • 08:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2034.codfw.wmnet
  • 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60878 and previous config saved to /var/cache/conftool/dbconfig/20240418-085619-marostegui.json
  • 08:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2110', diff saved to https://phabricator.wikimedia.org/P60877 and previous config saved to /var/cache/conftool/dbconfig/20240418-085608-arnaudb.json
  • 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 depool', diff saved to https://phabricator.wikimedia.org/P60876 and previous config saved to /var/cache/conftool/dbconfig/20240418-085235-arnaudb.json
  • 08:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1183 (T356166)', diff saved to https://phabricator.wikimedia.org/P60875 and previous config saved to /var/cache/conftool/dbconfig/20240418-084510-marostegui.json
  • 08:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 08:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P60874 and previous config saved to /var/cache/conftool/dbconfig/20240418-084353-marostegui.json
  • 08:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2029.codfw.wmnet
  • 08:42 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60873 and previous config saved to /var/cache/conftool/dbconfig/20240418-084223-arnaudb.json
  • 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2111.codfw.wmnet
  • 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2111.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:41 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 00m 15s)
  • 08:41 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
  • 08:40 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2111.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:38 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 08:34 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2111.codfw.wmnet
  • 08:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2029.codfw.wmnet
  • 08:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60872 and previous config saved to /var/cache/conftool/dbconfig/20240418-083422-arnaudb.json
  • 08:34 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2027.codfw.wmnet
  • 08:32 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60871 and previous config saved to /var/cache/conftool/dbconfig/20240418-083245-arnaudb.json
  • 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P60870 and previous config saved to /var/cache/conftool/dbconfig/20240418-082845-marostegui.json
  • 08:27 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60869 and previous config saved to /var/cache/conftool/dbconfig/20240418-082717-arnaudb.json
  • 08:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2112.codfw.wmnet
  • 08:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2112.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:25 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2112.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:25 mforns@deploy1002: Finished deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e] (duration: 14m 07s)
  • 08:24 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2027.codfw.wmnet
  • 08:23 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 08:15 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2112.codfw.wmnet
  • 08:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 depool', diff saved to https://phabricator.wikimedia.org/P60867 and previous config saved to /var/cache/conftool/dbconfig/20240418-081439-arnaudb.json
  • 08:13 kostajh: UTC morning deploys done
  • 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60866 and previous config saved to /var/cache/conftool/dbconfig/20240418-081338-marostegui.json
  • 08:12 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60865 and previous config saved to /var/cache/conftool/dbconfig/20240418-081210-arnaudb.json
  • 08:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T361627)', diff saved to https://phabricator.wikimedia.org/P60864 and previous config saved to /var/cache/conftool/dbconfig/20240418-081127-marostegui.json
  • 08:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 08:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 08:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60863 and previous config saved to /var/cache/conftool/dbconfig/20240418-081104-marostegui.json
  • 08:11 mforns@deploy1002: Started deploy [analytics/refinery@be07da9]: Regular analytics weekly train [analytics/refinery@be07da9e]
  • 08:10 kharlan@deploy1002: Finished scap: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597) (duration: 19m 36s)
  • 08:07 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2113.codfw.wmnet
  • 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2113.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:00 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2113.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 07:58 kharlan@deploy1002: urbanecm and kharlan: Continuing with sync
  • 07:57 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 07:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 25%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60862 and previous config saved to /var/cache/conftool/dbconfig/20240418-075704-arnaudb.json
  • 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P60861 and previous config saved to /var/cache/conftool/dbconfig/20240418-075557-marostegui.json
  • 07:54 kharlan@deploy1002: urbanecm and kharlan: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:52 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2113.codfw.wmnet
  • 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 depool', diff saved to https://phabricator.wikimedia.org/P60860 and previous config saved to /var/cache/conftool/dbconfig/20240418-075154-arnaudb.json
  • 07:51 kharlan@deploy1002: Started scap: Backport for EventStreamConfig: Fix stream title for mediawiki.ip_reputation.score (T354597)
  • 07:47 urbanecm@deploy1002: Finished scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597) (duration: 22m 27s)
  • 07:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 15%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60859 and previous config saved to /var/cache/conftool/dbconfig/20240418-074158-arnaudb.json
  • 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P60858 and previous config saved to /var/cache/conftool/dbconfig/20240418-074049-marostegui.json
  • 07:34 urbanecm@deploy1002: kharlan and urbanecm: Continuing with sync
  • 07:31 moritzm: upgrading PHP security updates on codfw baremetal servers T362511
  • 07:28 urbanecm@deploy1002: kharlan and urbanecm: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 10%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60857 and previous config saved to /var/cache/conftool/dbconfig/20240418-072653-arnaudb.json
  • 07:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60856 and previous config saved to /var/cache/conftool/dbconfig/20240418-072542-marostegui.json
  • 07:25 urbanecm@deploy1002: Started scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (2nd attempt) (T354597), ext-EventLogging: Add mediawiki.ip_reputation.score (T354597)
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60855 and previous config saved to /var/cache/conftool/dbconfig/20240418-072410-root.json
  • 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T361627)', diff saved to https://phabricator.wikimedia.org/P60854 and previous config saved to /var/cache/conftool/dbconfig/20240418-072331-marostegui.json
  • 07:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 07:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60853 and previous config saved to /var/cache/conftool/dbconfig/20240418-072309-marostegui.json
  • 07:21 urbanecm@deploy1002: Finished scap: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756) (duration: 17m 15s)
  • 07:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 5%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60852 and previous config saved to /var/cache/conftool/dbconfig/20240418-071147-arnaudb.json
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60851 and previous config saved to /var/cache/conftool/dbconfig/20240418-070904-root.json
  • 07:08 urbanecm@deploy1002: msz2001 and urbanecm: Continuing with sync
  • 07:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P60850 and previous config saved to /var/cache/conftool/dbconfig/20240418-070801-marostegui.json
  • 07:07 urbanecm@deploy1002: msz2001 and urbanecm: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:04 urbanecm@deploy1002: Started scap: Backport for [plwiki] Limit Content Translation publishing to mainspace for non-editors (T362756)
  • 06:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 2%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60849 and previous config saved to /var/cache/conftool/dbconfig/20240418-065641-arnaudb.json
  • 06:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60848 and previous config saved to /var/cache/conftool/dbconfig/20240418-065358-root.json
  • 06:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P60847 and previous config saved to /var/cache/conftool/dbconfig/20240418-065254-marostegui.json
  • 06:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db1183 (re)pooling @ 1%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60846 and previous config saved to /var/cache/conftool/dbconfig/20240418-064135-arnaudb.json
  • 06:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 06:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1183.eqiad.wmnet with reason: Maintenance
  • 06:38 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60845 and previous config saved to /var/cache/conftool/dbconfig/20240418-063852-root.json
  • 06:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60844 and previous config saved to /var/cache/conftool/dbconfig/20240418-063746-marostegui.json
  • 06:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1183.eqiad.wmnet with OS bookworm
  • 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T361627)', diff saved to https://phabricator.wikimedia.org/P60843 and previous config saved to /var/cache/conftool/dbconfig/20240418-063536-marostegui.json
  • 06:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 06:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 06:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 06:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 06:30 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1157.eqiad.wmnet with reason: Maintenance
  • 06:23 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60842 and previous config saved to /var/cache/conftool/dbconfig/20240418-062346-root.json
  • 06:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1183.eqiad.wmnet with reason: host reimage
  • 06:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1183.eqiad.wmnet with reason: host reimage
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60841 and previous config saved to /var/cache/conftool/dbconfig/20240418-060841-root.json
  • 06:02 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1183.eqiad.wmnet with OS bookworm
  • 06:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 06:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 05:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1183.eqiad.wmnet with reason: upgrade db1183 T360116
  • 05:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1183.eqiad.wmnet with reason: upgrade db1183 T360116
  • 05:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2108.codfw.wmnet with OS bookworm
  • 05:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2108 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60840 and previous config saved to /var/cache/conftool/dbconfig/20240418-055335-root.json
  • 05:50 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 05:50 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 05:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db1183 T362668', diff saved to https://phabricator.wikimedia.org/P60838 and previous config saved to /var/cache/conftool/dbconfig/20240418-054247-arnaudb.json
  • 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db1230 to s5 primary and set section read-write T362668', diff saved to https://phabricator.wikimedia.org/P60837 and previous config saved to /var/cache/conftool/dbconfig/20240418-053852-arnaudb.json
  • 05:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Set s5 eqiad as read-only for maintenance - T362668', diff saved to https://phabricator.wikimedia.org/P60836 and previous config saved to /var/cache/conftool/dbconfig/20240418-053657-arnaudb.json
  • 05:35 arnaudb: Starting s5 eqiad failover from db1183 to db1230 - T362668
  • 05:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2108.codfw.wmnet with reason: host reimage
  • 05:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2108.codfw.wmnet with reason: host reimage
  • 05:20 marostegui: dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
  • 05:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db1230 with weight 0 T362668', diff saved to https://phabricator.wikimedia.org/P60835 and previous config saved to /var/cache/conftool/dbconfig/20240418-051639-arnaudb.json
  • 05:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362668
  • 05:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362668
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2108.codfw.wmnet with OS bookworm
  • 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2108', diff saved to https://phabricator.wikimedia.org/P60834 and previous config saved to /var/cache/conftool/dbconfig/20240418-051129-root.json
  • 00:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T352010)', diff saved to https://phabricator.wikimedia.org/P60833 and previous config saved to /var/cache/conftool/dbconfig/20240418-000639-ladsgroup.json
  • 00:06 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 00:06 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 00:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60832 and previous config saved to /var/cache/conftool/dbconfig/20240418-000616-ladsgroup.json

2024-04-17

  • 23:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60831 and previous config saved to /var/cache/conftool/dbconfig/20240417-235105-ladsgroup.json
  • 23:48 amastilovic@deploy1002: Finished deploy [airflow-dags/analytics@c9d6969]: (no justification provided) (duration: 00m 37s)
  • 23:47 amastilovic@deploy1002: Started deploy [airflow-dags/analytics@c9d6969]: (no justification provided)
  • 23:37 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 23:37 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 23:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60830 and previous config saved to /var/cache/conftool/dbconfig/20240417-233731-ladsgroup.json
  • 23:35 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60829 and previous config saved to /var/cache/conftool/dbconfig/20240417-233557-ladsgroup.json
  • 23:22 sukhe: sukhe@cp1114:~$ sudo -i haproxy-restart
  • 23:22 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60828 and previous config saved to /var/cache/conftool/dbconfig/20240417-232221-ladsgroup.json
  • 23:20 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60827 and previous config saved to /var/cache/conftool/dbconfig/20240417-232050-ladsgroup.json
  • 23:14 mutante: rsyncing jenkins data from contint2002 to contint1002, pre-sync in preparation for migration next week - /srv/jenkins (291G) and much smaller zuul and jenkins data dirs T334517
  • 23:07 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P60826 and previous config saved to /var/cache/conftool/dbconfig/20240417-230714-ladsgroup.json
  • 22:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60825 and previous config saved to /var/cache/conftool/dbconfig/20240417-225206-ladsgroup.json
  • 22:42 zabe@deploy1002: Finished scap: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817) (duration: 17m 14s)
  • 22:29 zabe@deploy1002: jforrester and zabe: Continuing with sync
  • 22:28 zabe@deploy1002: jforrester and zabe: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 22:24 zabe@deploy1002: Started scap: Backport for Revert "REST: Deprecate using "post" as the parameter source" (T362817)
  • 22:11 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on 19 hosts with reason: T362508
  • 22:10 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on 19 hosts with reason: T362508
  • 21:50 mutante: deploying scap config change (gerrit:1020321) - [cumin2002:~] $ sudo cumin -b 4 -s 40 'C:scap AND mw*' 'run-puppet-agent' T359643
  • 21:09 mutante: DNS - created ae.wikimedia.org for United Arab Emirates User Group wiki - T362529
  • 21:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60824 and previous config saved to /var/cache/conftool/dbconfig/20240417-210256-marostegui.json
  • 20:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60823 and previous config saved to /var/cache/conftool/dbconfig/20240417-204748-marostegui.json
  • 20:44 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/echostore: apply
  • 20:44 eevans@deploy1002: helmfile [staging] START helmfile.d/services/echostore: apply
  • 20:44 cjming: end of UTC late backport window
  • 20:44 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
  • 20:43 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
  • 20:43 cjming@deploy1002: Finished scap: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861) (duration: 17m 30s)
  • 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60822 and previous config saved to /var/cache/conftool/dbconfig/20240417-203241-marostegui.json
  • 20:30 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
  • 20:29 cjming@deploy1002: cjming and jdlrobson: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:26 cjming@deploy1002: Started scap: Backport for Upstream tablet infobox styles (T3603861), Upstream tablet infobox styles (T3603861)
  • 20:25 cjming@deploy1002: Finished scap: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555) (duration: 18m 13s)
  • 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60821 and previous config saved to /var/cache/conftool/dbconfig/20240417-201733-marostegui.json
  • 20:11 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
  • 20:09 cjming@deploy1002: cjming and jdlrobson: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:06 cjming@deploy1002: Started scap: Backport for Enable WikimediaSkinStyles on English Wikipedia Vector 2022 skin (T362726), Enable night mode in AMC for all projects (T361555)
  • 19:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T361627)', diff saved to https://phabricator.wikimedia.org/P60820 and previous config saved to /var/cache/conftool/dbconfig/20240417-195628-marostegui.json
  • 19:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 19:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 19:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60819 and previous config saved to /var/cache/conftool/dbconfig/20240417-195605-marostegui.json
  • 19:46 eileen: civicrm upgraded from fdd12ed1 to 28adb4da
  • 19:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60818 and previous config saved to /var/cache/conftool/dbconfig/20240417-194058-marostegui.json
  • 19:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60817 and previous config saved to /var/cache/conftool/dbconfig/20240417-192551-marostegui.json
  • 19:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60816 and previous config saved to /var/cache/conftool/dbconfig/20240417-191043-marostegui.json
  • 18:56 ejegg: payments-wiki upgraded from 72e3bf19 to fb0367a4
  • 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T361627)', diff saved to https://phabricator.wikimedia.org/P60815 and previous config saved to /var/cache/conftool/dbconfig/20240417-184931-marostegui.json
  • 18:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 18:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60814 and previous config saved to /var/cache/conftool/dbconfig/20240417-184908-marostegui.json
  • 18:35 dancy@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.43.0-wmf.1 refs T361395
  • 18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60813 and previous config saved to /var/cache/conftool/dbconfig/20240417-183401-marostegui.json
  • 18:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60812 and previous config saved to /var/cache/conftool/dbconfig/20240417-181854-marostegui.json
  • 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60810 and previous config saved to /var/cache/conftool/dbconfig/20240417-180346-marostegui.json
  • 17:59 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:59 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:57 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:57 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T361627)', diff saved to https://phabricator.wikimedia.org/P60809 and previous config saved to /var/cache/conftool/dbconfig/20240417-174233-marostegui.json
  • 17:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 17:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 17:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60808 and previous config saved to /var/cache/conftool/dbconfig/20240417-174210-marostegui.json
  • 17:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60807 and previous config saved to /var/cache/conftool/dbconfig/20240417-172702-marostegui.json
  • 17:14 sukhe: running authdns-update for adding magru geo-resources/IPs: T346722
  • 17:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60805 and previous config saved to /var/cache/conftool/dbconfig/20240417-171154-marostegui.json
  • 16:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60804 and previous config saved to /var/cache/conftool/dbconfig/20240417-165647-marostegui.json
  • 16:56 topranks: running authdns-update to make magru dns records live T362421
  • 16:47 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/page-analytics: apply
  • 16:46 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/page-analytics: apply
  • 16:45 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/page-analytics: apply
  • 16:45 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/page-analytics: apply
  • 16:45 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/page-analytics: apply
  • 16:44 btullis@deploy1002: helmfile [staging] START helmfile.d/services/page-analytics: apply
  • 16:39 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/media-analytics: apply
  • 16:39 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/media-analytics: apply
  • 16:39 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/media-analytics: apply
  • 16:39 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/media-analytics: apply
  • 16:38 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/media-analytics: apply
  • 16:38 btullis@deploy1002: helmfile [staging] START helmfile.d/services/media-analytics: apply
  • 16:36 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:36 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:35 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T361627)', diff saved to https://phabricator.wikimedia.org/P60803 and previous config saved to /var/cache/conftool/dbconfig/20240417-163532-marostegui.json
  • 16:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 16:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 16:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60802 and previous config saved to /var/cache/conftool/dbconfig/20240417-163518-marostegui.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60801 and previous config saved to /var/cache/conftool/dbconfig/20240417-163506-arnaudb.json
  • 16:30 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 16:29 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:29 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:29 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:27 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 16:25 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:24 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 16:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60800 and previous config saved to /var/cache/conftool/dbconfig/20240417-162008-marostegui.json
  • 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60799 and previous config saved to /var/cache/conftool/dbconfig/20240417-161958-arnaudb.json
  • 16:18 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:18 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:17 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:14 claime: restarted rsyslog on mw2412 - T357616
  • 16:13 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 16:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2119.codfw.wmnet
  • 16:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:09 cdanis: above conftool actions had no impact on production, no dbctl config commit was performed.
  • 16:09 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 16:08 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:08 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:07 cdanis@cumin1002: conftool action : set/host_ip=10.64.16.8; selector: name=db1211
  • 16:07 cdanis@cumin1002: conftool action : set/host_ip=1.1.1.1; selector: name=db1211
  • 16:06 cdanis@cumin1002: conftool action : set/host_ip=10.64.16.8; selector: name=db1211
  • 16:06 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 16:05 cdanis@cumin1002: conftool action : set/host_ip=69.69.69.69; selector: name=db1211
  • 16:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60798 and previous config saved to /var/cache/conftool/dbconfig/20240417-160501-marostegui.json
  • 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60797 and previous config saved to /var/cache/conftool/dbconfig/20240417-160451-arnaudb.json
  • 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'db2119 depool T358741', diff saved to https://phabricator.wikimedia.org/P60796 and previous config saved to /var/cache/conftool/dbconfig/20240417-160443-arnaudb.json
  • 16:04 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/geo-analytics: apply
  • 16:04 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1114.eqiad.wmnet,service=(cdn|ats-be)
  • 16:04 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/geo-analytics: apply
  • 16:04 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/geo-analytics: apply
  • 16:03 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 16:03 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/geo-analytics: apply
  • 16:03 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/geo-analytics: apply
  • 16:03 btullis@deploy1002: helmfile [staging] START helmfile.d/services/geo-analytics: apply
  • 16:02 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2119.codfw.wmnet
  • 16:00 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:00 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:59 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:57 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 15:53 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:53 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:52 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:51 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1114.eqiad.wmnet with OS bullseye
  • 15:50 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2120.codfw.wmnet
  • 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2120.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 15:44 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2120.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 15:42 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 15:40 topranks: merging patch and updating dns servers with new magru ranges T362421
  • 15:35 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2120.codfw.wmnet
  • 15:34 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:34 cmooney@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:33 cmooney@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Adding first entries for magru IPs - cmooney@cumin1002"
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60795 and previous config saved to /var/cache/conftool/dbconfig/20240417-153238-arnaudb.json
  • 15:31 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1114.eqiad.wmnet with reason: host reimage
  • 15:31 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 15:30 topranks: making magru IPs live in netbox and generating DNS records with cookbook T362421
  • 15:27 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1114.eqiad.wmnet with reason: host reimage
  • 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60794 and previous config saved to /var/cache/conftool/dbconfig/20240417-152023-marostegui.json
  • 15:18 arnaudb@cumin1002: dbctl commit (dc=all): 'db2120 depool T358741', diff saved to https://phabricator.wikimedia.org/P60793 and previous config saved to /var/cache/conftool/dbconfig/20240417-151811-arnaudb.json
  • 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2127 (T360332)', diff saved to https://phabricator.wikimedia.org/P60792 and previous config saved to /var/cache/conftool/dbconfig/20240417-151653-arnaudb.json
  • 15:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 15:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 15:13 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:12 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 15:09 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet
  • 15:07 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1114.eqiad.wmnet with OS bullseye
  • 15:06 vgutierrez: repool ncredir2001
  • 15:05 Lucas_WMDE: UTC afternoon backport+config window (belatedly) done
  • 15:04 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript namespaceDupes mlwiki --fix # T362653: 0 pages to fix, 0 were resolvable; 82 links to fix, 82 were resolvable, 0 were deleted.
  • 15:03 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for mlwiki: create draft namespace (T362653) (duration: 32m 43s)
  • 14:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60790 and previous config saved to /var/cache/conftool/dbconfig/20240417-145916-marostegui.json
  • 14:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 14:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60789 and previous config saved to /var/cache/conftool/dbconfig/20240417-145838-marostegui.json
  • 14:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T352010)', diff saved to https://phabricator.wikimedia.org/P60788 and previous config saved to /var/cache/conftool/dbconfig/20240417-145136-ladsgroup.json
  • 14:51 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:51 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60787 and previous config saved to /var/cache/conftool/dbconfig/20240417-145113-ladsgroup.json
  • 14:50 logmsgbot: lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Continuing with sync
  • 14:44 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 14:44 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60786 and previous config saved to /var/cache/conftool/dbconfig/20240417-144330-marostegui.json
  • 14:36 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60785 and previous config saved to /var/cache/conftool/dbconfig/20240417-143606-ladsgroup.json
  • 14:34 logmsgbot: lucaswerkmeister-wmde@deploy1002 anzx and lucaswerkmeister-wmde: Backport for mlwiki: create draft namespace (T362653) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: insetup::data_persistence
  • 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60784 and previous config saved to /var/cache/conftool/dbconfig/20240417-143103-root.json
  • 14:31 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for mlwiki: create draft namespace (T362653)
  • 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60783 and previous config saved to /var/cache/conftool/dbconfig/20240417-142823-marostegui.json
  • 14:22 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:22 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp1114.eqiad.wmnet
  • 14:21 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1114.eqiad.wmnet
  • 14:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60782 and previous config saved to /var/cache/conftool/dbconfig/20240417-142057-ladsgroup.json
  • 14:20 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:20 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:20 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1114.eqiad.wmnet,service=(cdn|ats-be)
  • 14:20 sukhe: depool cp1114.eqiad.wmnet for PXE boot testing issues and downgrade NIC firmware: T350179
  • 14:19 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:19 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:18 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60781 and previous config saved to /var/cache/conftool/dbconfig/20240417-141557-root.json
  • 14:15 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 14:13 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 14:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60780 and previous config saved to /var/cache/conftool/dbconfig/20240417-141314-marostegui.json
  • 14:13 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 14:10 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 14:10 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:09 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 14:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 14:08 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: insetup::data_persistence
  • 14:08 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score" (duration: 16m 49s)
  • 14:06 vgutierrez: depool ncredir2001
  • 14:05 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60779 and previous config saved to /var/cache/conftool/dbconfig/20240417-140549-ladsgroup.json
  • 14:02 sukhe: running authdns-update for adding magru to geo-maps: T346722
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60778 and previous config saved to /var/cache/conftool/dbconfig/20240417-140051-root.json
  • 13:55 logmsgbot: lucaswerkmeister-wmde@deploy1002 trainbranchbot and lucaswerkmeister-wmde: Continuing with sync
  • 13:55 logmsgbot: lucaswerkmeister-wmde@deploy1002 trainbranchbot and lucaswerkmeister-wmde: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score" synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T361627)', diff saved to https://phabricator.wikimedia.org/P60777 and previous config saved to /var/cache/conftool/dbconfig/20240417-135253-marostegui.json
  • 13:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 13:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 13:51 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Revert "WikimediaEvents: Set IPoid URL and enable ip_reputation/score"
  • 13:49 logmsgbot: lucaswerkmeister-wmde@deploy1002 Sync cancelled.
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60776 and previous config saved to /var/cache/conftool/dbconfig/20240417-134545-root.json
  • 13:40 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1033.eqiad.wmnet
  • 13:36 sukhe: running authdns-update for CR 1020823
  • 13:36 logmsgbot: lucaswerkmeister-wmde@deploy1002 kharlan and lucaswerkmeister-wmde: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (T354597) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:33 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for WikimediaEvents: Set IPoid URL and enable ip_reputation/score (T354597)
  • 13:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 13:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60775 and previous config saved to /var/cache/conftool/dbconfig/20240417-133318-marostegui.json
  • 13:32 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60774 and previous config saved to /var/cache/conftool/dbconfig/20240417-133040-root.json
  • 13:29 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1033.eqiad.wmnet
  • 13:29 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1026.eqiad.wmnet
  • 13:23 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for cp1115.eqiad.wmnet
  • 13:23 sukhe@cumin1002: START - Cookbook sre.hosts.remove-downtime for cp1115.eqiad.wmnet
  • 13:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60773 and previous config saved to /var/cache/conftool/dbconfig/20240417-131811-marostegui.json
  • 13:18 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1026.eqiad.wmnet
  • 13:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2033.codfw.wmnet
  • 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60772 and previous config saved to /var/cache/conftool/dbconfig/20240417-131533-root.json
  • 13:11 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2033.codfw.wmnet
  • 13:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2031.codfw.wmnet
  • 13:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2120.codfw.wmnet with OS bookworm
  • 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127', diff saved to https://phabricator.wikimedia.org/P60771 and previous config saved to /var/cache/conftool/dbconfig/20240417-130303-marostegui.json
  • 13:01 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2031.codfw.wmnet
  • 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2120 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60770 and previous config saved to /var/cache/conftool/dbconfig/20240417-130027-root.json
  • 12:54 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2026.codfw.wmnet
  • 12:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60769 and previous config saved to /var/cache/conftool/dbconfig/20240417-124756-marostegui.json
  • 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2127 (T361627)', diff saved to https://phabricator.wikimedia.org/P60768 and previous config saved to /var/cache/conftool/dbconfig/20240417-122748-marostegui.json
  • 12:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 12:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2127.codfw.wmnet with reason: Maintenance
  • 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60767 and previous config saved to /var/cache/conftool/dbconfig/20240417-122725-marostegui.json
  • 12:25 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2120.codfw.wmnet with OS bookworm
  • 12:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2120', diff saved to https://phabricator.wikimedia.org/P60766 and previous config saved to /var/cache/conftool/dbconfig/20240417-122150-root.json
  • 12:12 vgutierrez: repool ncredir2001
  • 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P60765 and previous config saved to /var/cache/conftool/dbconfig/20240417-121218-marostegui.json
  • 12:06 moritzm: upgrading PHP on mediawiki baremetal canaries servers T362511
  • 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109', diff saved to https://phabricator.wikimedia.org/P60763 and previous config saved to /var/cache/conftool/dbconfig/20240417-115709-marostegui.json
  • 11:57 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
  • 11:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2032.codfw.wmnet
  • 11:44 vgutierrez: depool ncredir2001
  • 11:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60762 and previous config saved to /var/cache/conftool/dbconfig/20240417-114201-marostegui.json
  • 11:36 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
  • 11:33 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2032.codfw.wmnet
  • 11:30 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 11:30 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 11:29 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1032.eqiad.wmnet
  • 11:29 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 11:24 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 11:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T352010)', diff saved to https://phabricator.wikimedia.org/P60761 and previous config saved to /var/cache/conftool/dbconfig/20240417-112418-ladsgroup.json
  • 11:24 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 11:23 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 11:23 jiji@deploy1002: Finished scap: NoOp (duration: 09m 38s)
  • 11:22 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1032.eqiad.wmnet
  • 11:20 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2109 (T361627)', diff saved to https://phabricator.wikimedia.org/P60760 and previous config saved to /var/cache/conftool/dbconfig/20240417-112040-marostegui.json
  • 11:20 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 11:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60759 and previous config saved to /var/cache/conftool/dbconfig/20240417-112017-marostegui.json
  • 11:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2030.codfw.wmnet
  • 11:13 jiji@deploy1002: Started scap: NoOp
  • 11:06 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2030.codfw.wmnet
  • 11:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60758 and previous config saved to /var/cache/conftool/dbconfig/20240417-110510-marostegui.json
  • 11:04 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:54 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:53 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:53 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
  • 10:53 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60757 and previous config saved to /var/cache/conftool/dbconfig/20240417-105002-marostegui.json
  • 10:46 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-ext-ro,name=eqiad
  • 10:45 effie: pool eqiad back for mw-web-ro, mw-api-int-ro and mw-api-ext-ro
  • 10:44 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-api-int-ro,name=eqiad
  • 10:42 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-wikifunctions: apply
  • 10:42 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1027.eqiad.wmnet
  • 10:42 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-wikifunctions: apply
  • 10:41 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 10:41 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 10:40 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-parsoid: apply
  • 10:38 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-parsoid: apply
  • 10:37 jiji@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 10:36 jiji@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 10:36 jiji@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 10:36 jiji@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 10:35 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 10:35 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 10:35 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:35 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60756 and previous config saved to /var/cache/conftool/dbconfig/20240417-103455-marostegui.json
  • 10:34 akosiaris: apply the coredns patches for bumping instances from 4 to 6. They are noop, I am applying them to update helm's state.
  • 10:34 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 10:34 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:34 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:34 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1027.eqiad.wmnet
  • 10:33 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 10:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es2028.codfw.wmnet
  • 10:22 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es2028.codfw.wmnet
  • 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2105 (T361627)', diff saved to https://phabricator.wikimedia.org/P60755 and previous config saved to /var/cache/conftool/dbconfig/20240417-101446-marostegui.json
  • 10:14 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 10:14 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 10:08 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync
  • 10:08 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: sync
  • 10:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 10:00 akosiaris: manually bump coredns in eqiad to 6
  • 09:59 akosiaris: manually bump coredns in codfw to 6
  • 09:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60753 and previous config saved to /var/cache/conftool/dbconfig/20240417-095731-marostegui.json
  • 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-ext-ro,name=eqiad
  • 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-api-int-ro,name=eqiad
  • 09:44 cgoubert@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60750 and previous config saved to /var/cache/conftool/dbconfig/20240417-094223-marostegui.json
  • 09:31 jiji@deploy1002: scap failed: KeyError 'production' (duration: 22m 21s)
  • 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60749 and previous config saved to /var/cache/conftool/dbconfig/20240417-092923-root.json
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60748 and previous config saved to /var/cache/conftool/dbconfig/20240417-092714-marostegui.json
  • 09:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60747 and previous config saved to /var/cache/conftool/dbconfig/20240417-091418-root.json
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60746 and previous config saved to /var/cache/conftool/dbconfig/20240417-091203-marostegui.json
  • 09:08 jiji@deploy1002: Started scap: Switch mediawiki in eqiad to use node-local mcrouter ds - T346690
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T361627)', diff saved to https://phabricator.wikimedia.org/P60745 and previous config saved to /var/cache/conftool/dbconfig/20240417-090539-marostegui.json
  • 09:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 09:05 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 09:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60744 and previous config saved to /var/cache/conftool/dbconfig/20240417-090516-marostegui.json
  • 09:03 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60743 and previous config saved to /var/cache/conftool/dbconfig/20240417-085912-root.json
  • 08:57 hashar@deploy1002: Finished scap: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838) (duration: 16m 37s)
  • 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60742 and previous config saved to /var/cache/conftool/dbconfig/20240417-085009-marostegui.json
  • 08:44 hashar@deploy1002: hashar: Continuing with sync
  • 08:44 hashar@deploy1002: hashar: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 08:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60741 and previous config saved to /var/cache/conftool/dbconfig/20240417-084407-root.json
  • 08:41 hashar@deploy1002: Started scap: Backport for logging: pluralize $wmgDefaultMonologHandler (T238838)
  • 08:40 aqu: Deployed refinery using scap, then deployed onto hdfs
  • 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60739 and previous config saved to /var/cache/conftool/dbconfig/20240417-083501-marostegui.json
  • 08:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60738 and previous config saved to /var/cache/conftool/dbconfig/20240417-082901-root.json
  • 08:26 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa] (duration: 02m 23s)
  • 08:24 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@c4e197fa]
  • 08:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60737 and previous config saved to /var/cache/conftool/dbconfig/20240417-081953-marostegui.json
  • 08:16 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa] (duration: 03m 39s)
  • 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60736 and previous config saved to /var/cache/conftool/dbconfig/20240417-081356-root.json
  • 08:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T361627)', diff saved to https://phabricator.wikimedia.org/P60735 and previous config saved to /var/cache/conftool/dbconfig/20240417-081326-marostegui.json
  • 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 08:13 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f] (thin): Regular analytics weekly train THIN [analytics/refinery@c4e197fa]
  • 08:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 08:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60734 and previous config saved to /var/cache/conftool/dbconfig/20240417-081256-marostegui.json
  • 08:10 jayme@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host kubestage2002.codfw.wmnet
  • 08:07 aqu@deploy1002: Finished deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa] (duration: 27m 57s)
  • 08:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2150.codfw.wmnet with OS bookworm
  • 08:00 jayme@cumin1002: START - Cookbook sre.hosts.reboot-single for host kubestage2002.codfw.wmnet
  • 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2150 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60733 and previous config saved to /var/cache/conftool/dbconfig/20240417-075850-root.json
  • 07:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60732 and previous config saved to /var/cache/conftool/dbconfig/20240417-075748-marostegui.json
  • 07:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1173.eqiad.wmnet
  • 07:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2150.codfw.wmnet with reason: host reimage
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60731 and previous config saved to /var/cache/conftool/dbconfig/20240417-074241-marostegui.json
  • 07:40 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1173.eqiad.wmnet
  • 07:39 aqu@deploy1002: Started deploy [analytics/refinery@c4e197f]: Regular analytics weekly train [analytics/refinery@c4e197fa]
  • 07:39 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2150.codfw.wmnet with reason: host reimage
  • 07:39 aqu: analytics/refinery deploy begin (added source jars 0.2.35)
  • 07:38 jynus: restart db1216 database for mariadb upgrade
  • 07:37 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2214.codfw.wmnet
  • 07:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60730 and previous config saved to /var/cache/conftool/dbconfig/20240417-072733-marostegui.json
  • 07:27 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2214.codfw.wmnet
  • 07:26 jynus: restart db1240 database for mariadb upgrade
  • 07:22 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2150.codfw.wmnet with OS bookworm
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T361627)', diff saved to https://phabricator.wikimedia.org/P60729 and previous config saved to /var/cache/conftool/dbconfig/20240417-072122-marostegui.json
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2150', diff saved to https://phabricator.wikimedia.org/P60728 and previous config saved to /var/cache/conftool/dbconfig/20240417-072115-root.json
  • 07:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60727 and previous config saved to /var/cache/conftool/dbconfig/20240417-072059-marostegui.json
  • 07:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60726 and previous config saved to /var/cache/conftool/dbconfig/20240417-070552-marostegui.json
  • 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60725 and previous config saved to /var/cache/conftool/dbconfig/20240417-070206-root.json
  • 06:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60724 and previous config saved to /var/cache/conftool/dbconfig/20240417-065044-marostegui.json
  • 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60723 and previous config saved to /var/cache/conftool/dbconfig/20240417-064700-root.json
  • 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60722 and previous config saved to /var/cache/conftool/dbconfig/20240417-063537-marostegui.json
  • 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60721 and previous config saved to /var/cache/conftool/dbconfig/20240417-063155-root.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60720 and previous config saved to /var/cache/conftool/dbconfig/20240417-062918-marostegui.json
  • 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 06:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60719 and previous config saved to /var/cache/conftool/dbconfig/20240417-062856-marostegui.json
  • 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60718 and previous config saved to /var/cache/conftool/dbconfig/20240417-061649-root.json
  • 06:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60717 and previous config saved to /var/cache/conftool/dbconfig/20240417-061349-marostegui.json
  • 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60716 and previous config saved to /var/cache/conftool/dbconfig/20240417-060143-root.json
  • 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60715 and previous config saved to /var/cache/conftool/dbconfig/20240417-055841-marostegui.json
  • 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60714 and previous config saved to /var/cache/conftool/dbconfig/20240417-054637-root.json
  • 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60713 and previous config saved to /var/cache/conftool/dbconfig/20240417-054333-marostegui.json
  • 05:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60712 and previous config saved to /var/cache/conftool/dbconfig/20240417-053716-marostegui.json
  • 05:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 05:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 05:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60711 and previous config saved to /var/cache/conftool/dbconfig/20240417-053653-marostegui.json
  • 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2182.codfw.wmnet with OS bookworm
  • 05:31 marostegui@cumin1002: dbctl commit (dc=all): 'db2182 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60710 and previous config saved to /var/cache/conftool/dbconfig/20240417-053131-root.json
  • 05:26 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T352010)', diff saved to https://phabricator.wikimedia.org/P60709 and previous config saved to /var/cache/conftool/dbconfig/20240417-052600-ladsgroup.json
  • 05:25 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 05:25 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 05:25 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60708 and previous config saved to /var/cache/conftool/dbconfig/20240417-052537-ladsgroup.json
  • 05:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60707 and previous config saved to /var/cache/conftool/dbconfig/20240417-052145-marostegui.json
  • 05:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
  • 05:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2182.codfw.wmnet with reason: host reimage
  • 05:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60706 and previous config saved to /var/cache/conftool/dbconfig/20240417-051029-ladsgroup.json
  • 05:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60705 and previous config saved to /var/cache/conftool/dbconfig/20240417-050638-marostegui.json
  • 05:05 marostegui: Rename machine_vision tables on db1249 eqiad dbmaint s4 T362229
  • 05:00 marostegui: dbmaint Upgrade s7 codfw to Bookworm and MariaDB 10.6 T362745
  • 04:55 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60704 and previous config saved to /var/cache/conftool/dbconfig/20240417-045522-ladsgroup.json
  • 04:55 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2182.codfw.wmnet with OS bookworm
  • 04:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2182', diff saved to https://phabricator.wikimedia.org/P60703 and previous config saved to /var/cache/conftool/dbconfig/20240417-045353-root.json
  • 04:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60702 and previous config saved to /var/cache/conftool/dbconfig/20240417-045130-marostegui.json
  • 04:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T361627)', diff saved to https://phabricator.wikimedia.org/P60701 and previous config saved to /var/cache/conftool/dbconfig/20240417-044517-marostegui.json
  • 04:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:40 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60700 and previous config saved to /var/cache/conftool/dbconfig/20240417-044015-ladsgroup.json
  • 04:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance
  • 04:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1222.eqiad.wmnet with reason: Maintenance
  • 03:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T352010)', diff saved to https://phabricator.wikimedia.org/P60699 and previous config saved to /var/cache/conftool/dbconfig/20240417-033948-ladsgroup.json
  • 03:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 03:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 03:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60698 and previous config saved to /var/cache/conftool/dbconfig/20240417-033926-ladsgroup.json
  • 03:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60697 and previous config saved to /var/cache/conftool/dbconfig/20240417-032418-ladsgroup.json
  • 03:09 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60696 and previous config saved to /var/cache/conftool/dbconfig/20240417-030911-ladsgroup.json
  • 02:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60695 and previous config saved to /var/cache/conftool/dbconfig/20240417-025403-ladsgroup.json
  • 02:48 ryankemper: T361525 Trying to powercycle `elastic2088` thru mgmt port (host not responding to ssh)
  • 02:43 dani@deploy1002: helmfile [codfw] DONE helmfile.d/services/miscweb: apply
  • 02:43 dani@deploy1002: helmfile [codfw] START helmfile.d/services/miscweb: apply
  • 02:43 dani@deploy1002: helmfile [eqiad] DONE helmfile.d/services/miscweb: apply
  • 02:43 dani@deploy1002: helmfile [eqiad] START helmfile.d/services/miscweb: apply
  • 02:43 dani@deploy1002: helmfile [staging] DONE helmfile.d/services/miscweb: apply
  • 02:42 dani@deploy1002: helmfile [staging] START helmfile.d/services/miscweb: apply

2024-04-16

  • 23:25 hmonroy@deploy1002: Finished scap: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795) (duration: 17m 29s)
  • 23:12 hmonroy@deploy1002: musikanimal and hmonroy: Continuing with sync
  • 23:11 hmonroy@deploy1002: musikanimal and hmonroy: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 23:08 hmonroy@deploy1002: Started scap: Backport for [mediawikiwiki] enable CodeMirror V6 (T357795)
  • 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
  • 23:06 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002"
  • 23:03 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - pt1979@cumin2002"
  • 22:46 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage
  • 22:43 pt1979@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudcontrol2009-dev.codfw.wmnet with reason: host reimage
  • 22:25 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
  • 21:54 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
  • 21:48 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:47 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:45 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:42 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 cjming: end of UTC late backport window
  • 21:38 cjming@deploy1002: Finished scap: Backport for Use WikimediaMessages for template overrides (T361589) (duration: 19m 30s)
  • 21:25 cjming@deploy1002: jdlrobson and cjming: Continuing with sync
  • 21:21 cjming@deploy1002: jdlrobson and cjming: Backport for Use WikimediaMessages for template overrides (T361589) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 21:18 cjming@deploy1002: Started scap: Backport for Use WikimediaMessages for template overrides (T361589)
  • 21:16 cjming@deploy1002: Finished scap: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212) (duration: 18m 26s)
  • 21:02 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
  • 21:01 cjming@deploy1002: cjming and jdlrobson: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:57 cjming@deploy1002: Started scap: Backport for [phase 4] Vector-2022.js should no longer load legacy Vector site and user scripts/styles (T301212)
  • 20:56 cjming@deploy1002: Finished scap: Backport for Thumbnail styles generalized and moved to core (T360388) (duration: 22m 48s)
  • 20:42 cjming@deploy1002: cjming and jdlrobson: Continuing with sync
  • 20:36 cjming@deploy1002: cjming and jdlrobson: Backport for Thumbnail styles generalized and moved to core (T360388) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:33 cjming@deploy1002: Started scap: Backport for Thumbnail styles generalized and moved to core (T360388)
  • 20:30 mutante: CI - jenkins and zuul-merger are re-enabled on contint1002 after distro upgrade to bullseye - T334517
  • 20:26 pt1979@cumin2002: START - Cookbook sre.hosts.reimage for host cloudcontrol2009-dev.codfw.wmnet with OS bookworm
  • 20:22 mutante: CI - re-enabled jenkins and zuul-merged on contint1002 after distro upgrade - T360964
  • 20:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:22 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 20:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60693 and previous config saved to /var/cache/conftool/dbconfig/20240416-202206-marostegui.json
  • 20:08 aqu: Weekly deploy of refinery using scap, then deployed onto hdfs
  • 20:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P60691 and previous config saved to /var/cache/conftool/dbconfig/20240416-200659-marostegui.json
  • 19:55 pt1979@cumin2002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts ['cloudcontrol2009-dev']
  • 19:53 pt1979@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['cloudcontrol2009-dev']
  • 19:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246', diff saved to https://phabricator.wikimedia.org/P60690 and previous config saved to /var/cache/conftool/dbconfig/20240416-195151-marostegui.json
  • 19:47 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 08s)
  • 19:47 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
  • 19:42 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@59f7d091] (duration: 02m 24s)
  • 19:40 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09] (hadoop-test): Regular analytics weekly train TEST [analytics/refinery@59f7d091]
  • 19:38 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09] (thin): Regular analytics weekly train THIN [analytics/refinery@59f7d091] (duration: 04m 10s)
  • 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60689 and previous config saved to /var/cache/conftool/dbconfig/20240416-193643-marostegui.json
  • 19:33 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09] (thin): Regular analytics weekly train THIN [analytics/refinery@59f7d091]
  • 19:31 aqu@deploy1002: Finished deploy [analytics/refinery@59f7d09]: Regular analytics weekly train [analytics/refinery@59f7d091] (duration: 13m 08s)
  • 19:24 aqu@deploy1002: Finished deploy [airflow-dags/analytics_test@9208108]: Regular analytics weekly train [airflow-dags/analytics_test@9208108e] (duration: 00m 10s)
  • 19:23 aqu@deploy1002: Started deploy [airflow-dags/analytics_test@9208108]: Regular analytics weekly train [airflow-dags/analytics_test@9208108e]
  • 19:18 aqu@deploy1002: Started deploy [analytics/refinery@59f7d09]: Regular analytics weekly train [analytics/refinery@59f7d091]
  • 19:17 aqu: Deployment train for analytics/refinery
  • 19:15 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1246 (T361627)', diff saved to https://phabricator.wikimedia.org/P60687 and previous config saved to /var/cache/conftool/dbconfig/20240416-191522-marostegui.json
  • 19:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 19:15 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 19:14 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 13s)
  • 19:14 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
  • 19:12 hashar@deploy1002: Finished deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517 (duration: 00m 03s)
  • 19:12 hashar@deploy1002: Started deploy [zuul/deploy@efce3ee]: Redeploy Zuul following host reimaging - T334517
  • 19:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60686 and previous config saved to /var/cache/conftool/dbconfig/20240416-191128-marostegui.json
  • 19:08 dancy@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.43.0-wmf.1 refs T361395
  • 18:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60685 and previous config saved to /var/cache/conftool/dbconfig/20240416-185621-marostegui.json
  • 18:52 aqu@deploy1002: Finished deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e] (duration: 00m 26s)
  • 18:52 aqu@deploy1002: Started deploy [airflow-dags/analytics@9208108]: Regular analytics weekly train [airflow-dags/analytics@9208108e]
  • 18:50 dancy@deploy1002: Installation of scap version "4.77.0" completed for 340 hosts
  • 18:49 dancy@deploy1002: Installing scap version "4.77.0" for 340 hosts
  • 18:48 dancy@deploy1002: Finished scap: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692) (duration: 22m 56s)
  • 18:44 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host contint1002.wikimedia.org with OS bullseye
  • 18:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60684 and previous config saved to /var/cache/conftool/dbconfig/20240416-184113-marostegui.json
  • 18:40 mutante: contint1002 - sudo a2dismod mpm_event to work around known race condition and fix failed initial puppet run - T334517
  • 18:35 dancy@deploy1002: cscott and dancy: Continuing with sync
  • 18:29 bearloga@deploy1002: Finished deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided) (duration: 00m 07s)
  • 18:29 bearloga@deploy1002: Started deploy [airflow-dags/analytics_product@77af7cb]: (no justification provided)
  • 18:29 dancy@deploy1002: cscott and dancy: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 18:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60683 and previous config saved to /var/cache/conftool/dbconfig/20240416-182606-marostegui.json
  • 18:26 dancy@deploy1002: Started scap: Backport for [Parser] Temporarily disable deprecation warnings for dynamic properties (T362692)
  • 18:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T361627)', diff saved to https://phabricator.wikimedia.org/P60682 and previous config saved to /var/cache/conftool/dbconfig/20240416-181001-marostegui.json
  • 18:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 18:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60681 and previous config saved to /var/cache/conftool/dbconfig/20240416-180938-marostegui.json
  • 18:07 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: host reimage
  • 18:04 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: host reimage
  • 17:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60680 and previous config saved to /var/cache/conftool/dbconfig/20240416-175431-marostegui.json
  • 17:52 dzahn@cumin2002: START - Cookbook sre.hosts.reimage for host contint1002.wikimedia.org with OS bullseye
  • 17:51 mutante: CI - jenkins on contint1002 disabled - reimaging in progress
  • 17:50 bearloga@deploy1002: Finished deploy [airflow-dags/analytics_product@bb33843]: (no justification provided) (duration: 00m 06s)
  • 17:50 bearloga@deploy1002: Started deploy [airflow-dags/analytics_product@bb33843]: (no justification provided)
  • 17:49 dzahn@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517
  • 17:48 dzahn@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on contint1002.wikimedia.org with reason: reimage https://phabricator.wikmedia.org/T334517
  • 17:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60679 and previous config saved to /var/cache/conftool/dbconfig/20240416-173923-marostegui.json
  • 17:37 mutante: CI - disabling zuul-merger on contint1002 - there is another on contint2002
  • 17:35 pt1979@cumin2002: END (PASS) - Cookbook sre.hosts.provision (exit_code=0) for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60678 and previous config saved to /var/cache/conftool/dbconfig/20240416-172515-root.json
  • 17:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60677 and previous config saved to /var/cache/conftool/dbconfig/20240416-172415-marostegui.json
  • 17:22 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T361627)', diff saved to https://phabricator.wikimedia.org/P60676 and previous config saved to /var/cache/conftool/dbconfig/20240416-172201-marostegui.json
  • 17:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 17:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 17:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 17:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 17:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60675 and previous config saved to /var/cache/conftool/dbconfig/20240416-171738-marostegui.json
  • 17:16 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T352010)', diff saved to https://phabricator.wikimedia.org/P60674 and previous config saved to /var/cache/conftool/dbconfig/20240416-171047-ladsgroup.json
  • 17:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 17:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 17:10 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 17:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60673 and previous config saved to /var/cache/conftool/dbconfig/20240416-171010-root.json
  • 17:10 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 17:10 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60672 and previous config saved to /var/cache/conftool/dbconfig/20240416-171006-ladsgroup.json
  • 17:04 pt1979@cumin2002: START - Cookbook sre.hosts.provision for host cloudcontrol2009-dev.mgmt.codfw.wmnet with reboot policy FORCED
  • 17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:02 pt1979@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002"
  • 17:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60671 and previous config saved to /var/cache/conftool/dbconfig/20240416-170231-marostegui.json
  • 17:01 pt1979@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudcontrol2009 DNS add - pt1979@cumin2002"
  • 16:59 pt1979@cumin2002: START - Cookbook sre.dns.netbox
  • 16:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60670 and previous config saved to /var/cache/conftool/dbconfig/20240416-165504-root.json
  • 16:54 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60669 and previous config saved to /var/cache/conftool/dbconfig/20240416-165458-ladsgroup.json
  • 16:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60668 and previous config saved to /var/cache/conftool/dbconfig/20240416-164722-marostegui.json
  • 16:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60666 and previous config saved to /var/cache/conftool/dbconfig/20240416-163958-root.json
  • 16:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60665 and previous config saved to /var/cache/conftool/dbconfig/20240416-163951-ladsgroup.json
  • 16:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60664 and previous config saved to /var/cache/conftool/dbconfig/20240416-163215-marostegui.json
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 100%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60663 and previous config saved to /var/cache/conftool/dbconfig/20240416-162926-arnaudb.json
  • 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T361627)', diff saved to https://phabricator.wikimedia.org/P60662 and previous config saved to /var/cache/conftool/dbconfig/20240416-162900-marostegui.json
  • 16:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60661 and previous config saved to /var/cache/conftool/dbconfig/20240416-162838-marostegui.json
  • 16:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60660 and previous config saved to /var/cache/conftool/dbconfig/20240416-162452-root.json
  • 16:24 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60659 and previous config saved to /var/cache/conftool/dbconfig/20240416-162443-ladsgroup.json
  • 16:16 brennen: finished phabricator deploy for T362689 - believe things are currently stable
  • 16:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 75%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60657 and previous config saved to /var/cache/conftool/dbconfig/20240416-161420-arnaudb.json
  • 16:14 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 16:13 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 16:13 brennen@deploy1002: Finished deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689 (duration: 00m 42s)
  • 16:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60656 and previous config saved to /var/cache/conftool/dbconfig/20240416-161330-marostegui.json
  • 16:13 brennen@deploy1002: Started deploy [phabricator/deployment@098b9c2]: deploy phab1004 for T362689
  • 16:13 rzl: rzl@mwmaint1002:~$ sudo systemctl start mediawiki_job_globalblocking-fixGlobalBlockWhitelist.service # T360516
  • 16:12 brennen@deploy1002: Finished deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689 (duration: 00m 32s)
  • 16:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2217.codfw.wmnet
  • 16:12 brennen@deploy1002: Started deploy [phabricator/deployment@098b9c2]: test deploy phab2002 for T362689
  • 16:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60655 and previous config saved to /var/cache/conftool/dbconfig/20240416-160946-root.json
  • 16:07 brennen: starting phabricator deploy for T362689
  • 16:06 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 16:05 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 16:01 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 16:00 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 15:59 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 50%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60654 and previous config saved to /var/cache/conftool/dbconfig/20240416-155914-arnaudb.json
  • 15:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2217.codfw.wmnet
  • 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60653 and previous config saved to /var/cache/conftool/dbconfig/20240416-155823-marostegui.json
  • 15:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2127.codfw.wmnet with OS bookworm
  • 15:55 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1231.eqiad.wmnet
  • 15:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2127 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60652 and previous config saved to /var/cache/conftool/dbconfig/20240416-155440-root.json
  • 15:49 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: Post clone', diff saved to https://phabricator.wikimedia.org/P60651 and previous config saved to /var/cache/conftool/dbconfig/20240416-154915-arnaudb.json
  • 15:48 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:47 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:46 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1231.eqiad.wmnet
  • 15:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1168.eqiad.wmnet
  • 15:44 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 25%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60650 and previous config saved to /var/cache/conftool/dbconfig/20240416-154408-arnaudb.json
  • 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60649 and previous config saved to /var/cache/conftool/dbconfig/20240416-154316-marostegui.json
  • 15:39 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T361627)', diff saved to https://phabricator.wikimedia.org/P60648 and previous config saved to /var/cache/conftool/dbconfig/20240416-153902-marostegui.json
  • 15:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 15:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60647 and previous config saved to /var/cache/conftool/dbconfig/20240416-153839-marostegui.json
  • 15:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2127.codfw.wmnet with reason: host reimage
  • 15:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: Post clone', diff saved to https://phabricator.wikimedia.org/P60646 and previous config saved to /var/cache/conftool/dbconfig/20240416-153408-arnaudb.json
  • 15:32 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1168.eqiad.wmnet
  • 15:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2127.codfw.wmnet with reason: host reimage
  • 15:31 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2129.codfw.wmnet
  • 15:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 15%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60645 and previous config saved to /var/cache/conftool/dbconfig/20240416-152902-arnaudb.json
  • 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60644 and previous config saved to /var/cache/conftool/dbconfig/20240416-152331-marostegui.json
  • 15:19 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: Post clone', diff saved to https://phabricator.wikimedia.org/P60643 and previous config saved to /var/cache/conftool/dbconfig/20240416-151902-arnaudb.json
  • 15:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2129.codfw.wmnet
  • 15:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance
  • 15:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2199.codfw.wmnet with reason: Maintenance
  • 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60642 and previous config saved to /var/cache/conftool/dbconfig/20240416-151649-arnaudb.json
  • 15:15 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1224.eqiad.wmnet
  • 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 10%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60641 and previous config saved to /var/cache/conftool/dbconfig/20240416-151357-arnaudb.json
  • 15:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2127.codfw.wmnet with OS bookworm
  • 15:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2127 T362616', diff saved to https://phabricator.wikimedia.org/P60640 and previous config saved to /var/cache/conftool/dbconfig/20240416-151032-root.json
  • 15:09 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db2205 to s3 primary T362616', diff saved to https://phabricator.wikimedia.org/P60639 and previous config saved to /var/cache/conftool/dbconfig/20240416-150933-root.json
  • 15:08 marostegui: Starting s3 codfw failover from db2127 to db2205 - T362616
  • 15:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60638 and previous config saved to /var/cache/conftool/dbconfig/20240416-150824-marostegui.json
  • 15:07 brennen@deploy1002: Finished deploy [phabricator/deployment@7773191]: deploy phab1004 for T362666 (duration: 00m 30s)
  • 15:06 brennen@deploy1002: Started deploy [phabricator/deployment@7773191]: deploy phab1004 for T362666
  • 15:06 brennen@deploy1002: Finished deploy [phabricator/deployment@7773191]: test deploy phab2002 for T362666 (duration: 00m 32s)
  • 15:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1224.eqiad.wmnet
  • 15:05 brennen@deploy1002: Started deploy [phabricator/deployment@7773191]: test deploy phab2002 for T362666
  • 15:03 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 20%: Post clone', diff saved to https://phabricator.wikimedia.org/P60637 and previous config saved to /var/cache/conftool/dbconfig/20240416-150356-arnaudb.json
  • 15:03 samtar@deploy1002: Finished scap: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki (duration: 17m 17s)
  • 15:03 jelto@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
  • 15:03 jelto@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on phab1004.eqiad.wmnet with reason: Phabricator/Phorge update
  • 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60636 and previous config saved to /var/cache/conftool/dbconfig/20240416-150141-arnaudb.json
  • 15:00 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1201.eqiad.wmnet
  • 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 5%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60635 and previous config saved to /var/cache/conftool/dbconfig/20240416-145851-arnaudb.json
  • 14:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60634 and previous config saved to /var/cache/conftool/dbconfig/20240416-145316-marostegui.json
  • 14:50 samtar@deploy1002: samtar: Continuing with sync
  • 14:50 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2205 with weight 0 T362616', diff saved to https://phabricator.wikimedia.org/P60633 and previous config saved to /var/cache/conftool/dbconfig/20240416-144957-root.json
  • 14:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s3 T362616
  • 14:49 samtar@deploy1002: samtar: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s3 T362616
  • 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 10%: Post clone', diff saved to https://phabricator.wikimedia.org/P60632 and previous config saved to /var/cache/conftool/dbconfig/20240416-144850-arnaudb.json
  • 14:48 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop analytics cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 14:47 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1201.eqiad.wmnet
  • 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T361627)', diff saved to https://phabricator.wikimedia.org/P60631 and previous config saved to /var/cache/conftool/dbconfig/20240416-144727-marostegui.json
  • 14:47 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 14:47 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60630 and previous config saved to /var/cache/conftool/dbconfig/20240416-144704-marostegui.json
  • 14:46 samtar@deploy1002: Started scap: Backport for IS: Set Phonos to Inline Audio Player mode on test.wiki
  • 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60629 and previous config saved to /var/cache/conftool/dbconfig/20240416-144634-arnaudb.json
  • 14:45 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2193.codfw.wmnet
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 2%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60628 and previous config saved to /var/cache/conftool/dbconfig/20240416-144346-arnaudb.json
  • 14:43 taavi@deploy1002: Finished scap: Backport for Disallow changing email on Wikitech directly (T360883) (duration: 16m 24s)
  • 14:36 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2193.codfw.wmnet
  • 14:36 vgutierrez: pool ncredir2002
  • 14:33 vgutierrez: depool ncredir2002
  • 14:32 vgutierrez: pool ncredir2001
  • 14:32 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1187.eqiad.wmnet
  • 14:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P60627 and previous config saved to /var/cache/conftool/dbconfig/20240416-143157-marostegui.json
  • 14:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60626 and previous config saved to /var/cache/conftool/dbconfig/20240416-143126-arnaudb.json
  • 14:30 taavi@deploy1002: taavi: Continuing with sync
  • 14:29 taavi@deploy1002: taavi: Backport for Disallow changing email on Wikitech directly (T360883) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2123 (re)pooling @ 1%: post maintenance repool', diff saved to https://phabricator.wikimedia.org/P60625 and previous config saved to /var/cache/conftool/dbconfig/20240416-142840-arnaudb.json
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P60624 and previous config saved to /var/cache/conftool/dbconfig/20240416-142808-arnaudb.json
  • 14:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 14:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 14:26 taavi@deploy1002: Started scap: Backport for Disallow changing email on Wikitech directly (T360883)
  • 14:26 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 14:26 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 14:23 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1187.eqiad.wmnet
  • 14:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:22 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2123.codfw.wmnet with reason: Maintenance
  • 14:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2114.codfw.wmnet
  • 14:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2123.codfw.wmnet with OS bookworm
  • 14:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162', diff saved to https://phabricator.wikimedia.org/P60623 and previous config saved to /var/cache/conftool/dbconfig/20240416-141649-marostegui.json
  • 14:06 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Restrict local uploads to uploader user group in azwiki (T360847) (duration: 35m 04s)
  • 14:02 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2114.codfw.wmnet
  • 14:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60622 and previous config saved to /var/cache/conftool/dbconfig/20240416-140142-marostegui.json
  • 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1162 (T361627)', diff saved to https://phabricator.wikimedia.org/P60621 and previous config saved to /var/cache/conftool/dbconfig/20240416-135928-marostegui.json
  • 13:59 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 13:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60620 and previous config saved to /var/cache/conftool/dbconfig/20240416-135906-marostegui.json
  • 13:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2123.codfw.wmnet with reason: host reimage
  • 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2123.codfw.wmnet with reason: host reimage
  • 13:53 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Continuing with sync
  • 13:44 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 13:44 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 13:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60619 and previous config saved to /var/cache/conftool/dbconfig/20240416-134358-marostegui.json
  • 13:43 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 13:43 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 13:38 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2123.codfw.wmnet with OS bookworm
  • 13:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2123.codfw.wmnet with reason: T360116
  • 13:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2123.codfw.wmnet with reason: T360116
  • 13:34 logmsgbot: lucaswerkmeister-wmde@deploy1002 nmw03 and lucaswerkmeister-wmde: Backport for Restrict local uploads to uploader user group in azwiki (T360847) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:33 vgutierrez: depool ncredir2001
  • 13:31 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Restrict local uploads to uploader user group in azwiki (T360847)
  • 13:29 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414) (duration: 18m 39s)
  • 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60618 and previous config saved to /var/cache/conftool/dbconfig/20240416-132851-marostegui.json
  • 13:23 vgutierrez: pool ncredir2001
  • 13:20 vgutierrez: depool ncredir2001
  • 13:20 vgutierrez: pool ncredir1001
  • 13:18 vgutierrez: depool ncredir1001
  • 13:17 vgutierrez: pool ncredir2001
  • 13:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 msz2001 and lucaswerkmeister-wmde: Continuing with sync
  • 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 msz2001 and lucaswerkmeister-wmde: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60617 and previous config saved to /var/cache/conftool/dbconfig/20240416-131344-marostegui.json
  • 13:11 vgutierrez: depool ncredir2001
  • 13:11 vgutierrez: pool ncredir1001
  • 13:11 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Remove 'obsolete-tag' from $wgSignatureAllowedLintErrors on Polish Wikipedia (T362414)
  • 13:07 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T361627)', diff saved to https://phabricator.wikimedia.org/P60616 and previous config saved to /var/cache/conftool/dbconfig/20240416-130710-marostegui.json
  • 13:07 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:07 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 13:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 13:06 vgutierrez: depool ncredir1001
  • 13:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
  • 13:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2204.codfw.wmnet with reason: Maintenance
  • 13:01 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-mcrouter: apply
  • 13:01 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-mcrouter: apply
  • 12:57 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 12:55 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 12:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2124.codfw.wmnet
  • 12:13 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2124.codfw.wmnet
  • 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60615 and previous config saved to /var/cache/conftool/dbconfig/20240416-121211-marostegui.json
  • 12:09 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2180.codfw.wmnet
  • 11:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2180.codfw.wmnet
  • 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2151.codfw.wmnet
  • 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60613 and previous config saved to /var/cache/conftool/dbconfig/20240416-115703-marostegui.json
  • 11:50 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2151.codfw.wmnet
  • 11:48 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2169.codfw.wmnet
  • 11:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60612 and previous config saved to /var/cache/conftool/dbconfig/20240416-114155-marostegui.json
  • 11:30 stevemunene@deploy1002: helmfile [codfw] DONE helmfile.d/services/datahub: sync on main
  • 11:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60611 and previous config saved to /var/cache/conftool/dbconfig/20240416-112648-marostegui.json
  • 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T361627)', diff saved to https://phabricator.wikimedia.org/P60610 and previous config saved to /var/cache/conftool/dbconfig/20240416-112134-marostegui.json
  • 11:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 11:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 11:19 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2169.codfw.wmnet
  • 11:16 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:16 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 11:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60609 and previous config saved to /var/cache/conftool/dbconfig/20240416-111602-marostegui.json
  • 11:08 stevemunene@deploy1002: helmfile [codfw] START helmfile.d/services/datahub: apply on main
  • 11:07 hnowlan@puppetmaster1001: conftool action : set/pooled=yes; selector: name=restbase1042.eqiad.wmnet
  • 11:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60608 and previous config saved to /var/cache/conftool/dbconfig/20240416-110055-marostegui.json
  • 10:58 hnowlan@puppetmaster1001: conftool action : set/pooled=no; selector: name=restbase1042.eqiad.wmnet
  • 10:56 hnowlan: disabling puppet on A:restbase before switching to cfssl
  • 10:55 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:55 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60607 and previous config saved to /var/cache/conftool/dbconfig/20240416-104547-marostegui.json
  • 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60605 and previous config saved to /var/cache/conftool/dbconfig/20240416-103040-marostegui.json
  • 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60604 and previous config saved to /var/cache/conftool/dbconfig/20240416-102540-root.json
  • 10:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T361627)', diff saved to https://phabricator.wikimedia.org/P60603 and previous config saved to /var/cache/conftool/dbconfig/20240416-102510-marostegui.json
  • 10:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 10:24 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 10:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60602 and previous config saved to /var/cache/conftool/dbconfig/20240416-102447-marostegui.json
  • 10:20 moritzm: upgrading PHP on remaining mwdebug servers T362511
  • 10:19 cmooney@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 10:17 cmooney@cumin1002: START - Cookbook sre.dns.netbox
  • 10:15 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on all k8s nodes - T357616
  • 10:13 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:13 moritzm: uploaded PHP 7.4 1:7.4.33-1+0~20221108.73+debian10~1.gbpa00350a+wmf10u2+icu67u2 to buster-wikimedia/component/icu67 T362511
  • 10:12 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:12 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:10 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:10 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60601 and previous config saved to /var/cache/conftool/dbconfig/20240416-101034-root.json
  • 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60600 and previous config saved to /var/cache/conftool/dbconfig/20240416-100939-marostegui.json
  • 10:09 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 10:09 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 10:08 jayme@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 10:07 jayme@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 09:56 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 09:55 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 09:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60599 and previous config saved to /var/cache/conftool/dbconfig/20240416-095528-root.json
  • 09:54 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 09:54 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 09:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60598 and previous config saved to /var/cache/conftool/dbconfig/20240416-095432-marostegui.json
  • 09:49 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 09:48 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 09:48 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 09:48 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 09:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60597 and previous config saved to /var/cache/conftool/dbconfig/20240416-094023-root.json
  • 09:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60596 and previous config saved to /var/cache/conftool/dbconfig/20240416-093924-marostegui.json
  • 09:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T361627)', diff saved to https://phabricator.wikimedia.org/P60595 and previous config saved to /var/cache/conftool/dbconfig/20240416-093318-marostegui.json
  • 09:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 09:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 09:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60594 and previous config saved to /var/cache/conftool/dbconfig/20240416-093255-marostegui.json
  • 09:31 arnaudb: Starting s5 codfw failover from db2123 to db2213 - T362614 (forgot to send it)
  • 09:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool db2123 T362614', diff saved to https://phabricator.wikimedia.org/P60593 and previous config saved to /var/cache/conftool/dbconfig/20240416-093041-arnaudb.json
  • 09:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2213 to s5 primary T362614', diff saved to https://phabricator.wikimedia.org/P60592 and previous config saved to /var/cache/conftool/dbconfig/20240416-092800-arnaudb.json
  • 09:25 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60591 and previous config saved to /var/cache/conftool/dbconfig/20240416-092517-root.json
  • 09:21 slyngshede@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host cloudidm2001-dev.codfw.wmnet
  • 09:20 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudidm2001-dev.codfw.wmnet with OS bookworm
  • 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60590 and previous config saved to /var/cache/conftool/dbconfig/20240416-091747-marostegui.json
  • 09:10 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60588 and previous config saved to /var/cache/conftool/dbconfig/20240416-091009-root.json
  • 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2213 with weight 0 T362614', diff saved to https://phabricator.wikimedia.org/P60587 and previous config saved to /var/cache/conftool/dbconfig/20240416-090755-arnaudb.json
  • 09:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362614
  • 09:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 26 hosts with reason: Primary switchover s5 T362614
  • 09:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60586 and previous config saved to /var/cache/conftool/dbconfig/20240416-090625-arnaudb.json
  • 09:05 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudidm2001-dev.codfw.wmnet with reason: host reimage
  • 09:02 slyngshede@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudidm2001-dev.codfw.wmnet with reason: host reimage
  • 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60585 and previous config saved to /var/cache/conftool/dbconfig/20240416-090240-marostegui.json
  • 08:59 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on wikikube eqiad - T357616
  • 08:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2105 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60584 and previous config saved to /var/cache/conftool/dbconfig/20240416-085503-root.json
  • 08:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60583 and previous config saved to /var/cache/conftool/dbconfig/20240416-085120-arnaudb.json
  • 08:48 slyngshede@cumin1002: START - Cookbook sre.hosts.reimage for host cloudidm2001-dev.codfw.wmnet with OS bookworm
  • 08:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60582 and previous config saved to /var/cache/conftool/dbconfig/20240416-084733-marostegui.json
  • 08:47 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on wikikube codfw - T357616
  • 08:46 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
  • 08:45 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
  • 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) cloudidm2001-dev.codfw.wmnet on all recursors
  • 08:45 slyngshede@cumin1002: START - Cookbook sre.dns.wipe-cache cloudidm2001-dev.codfw.wmnet on all recursors
  • 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:45 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
  • 08:44 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM cloudidm2001-dev.codfw.wmnet - slyngshede@cumin1002"
  • 08:42 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 08:42 slyngshede@cumin1002: START - Cookbook sre.ganeti.makevm for new host cloudidm2001-dev.codfw.wmnet
  • 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T361627)', diff saved to https://phabricator.wikimedia.org/P60581 and previous config saved to /var/cache/conftool/dbconfig/20240416-084118-marostegui.json
  • 08:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 08:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 08:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60580 and previous config saved to /var/cache/conftool/dbconfig/20240416-084055-marostegui.json
  • 08:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60579 and previous config saved to /var/cache/conftool/dbconfig/20240416-083614-arnaudb.json
  • 08:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2105.codfw.wmnet with OS bookworm
  • 08:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60578 and previous config saved to /var/cache/conftool/dbconfig/20240416-082548-marostegui.json
  • 08:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60577 and previous config saved to /var/cache/conftool/dbconfig/20240416-082108-arnaudb.json
  • 08:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1161.eqiad.wmnet with OS bookworm
  • 08:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2105.codfw.wmnet with reason: host reimage
  • 08:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60576 and previous config saved to /var/cache/conftool/dbconfig/20240416-081040-marostegui.json
  • 08:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2105.codfw.wmnet with reason: host reimage
  • 07:56 volans@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox-canary
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage
  • 07:56 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60575 and previous config saved to /var/cache/conftool/dbconfig/20240416-075533-marostegui.json
  • 07:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1161.eqiad.wmnet with reason: host reimage
  • 07:52 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2105.codfw.wmnet with OS bookworm
  • 07:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2105', diff saved to https://phabricator.wikimedia.org/P60574 and previous config saved to /var/cache/conftool/dbconfig/20240416-075056-root.json
  • 07:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T361627)', diff saved to https://phabricator.wikimedia.org/P60573 and previous config saved to /var/cache/conftool/dbconfig/20240416-074952-marostegui.json
  • 07:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 07:49 volans@cumin1002: END (PASS) - Cookbook sre.netbox.update-extras (exit_code=0) rolling restart_daemons on A:netbox
  • 07:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 07:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60572 and previous config saved to /var/cache/conftool/dbconfig/20240416-074928-marostegui.json
  • 07:43 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox
  • 07:43 volans@cumin1002: END (FAIL) - Cookbook sre.netbox.update-extras (exit_code=1) rolling restart_daemons on A:netbox-canary
  • 07:40 volans@cumin1002: START - Cookbook sre.netbox.update-extras rolling restart_daemons on A:netbox-canary
  • 07:40 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1161.eqiad.wmnet with OS bookworm
  • 07:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: T360116
  • 07:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db[1154,1161].eqiad.wmnet with reason: T360116
  • 07:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 3:00:00 on db1161.eqiad.wmnet with reason: T360116
  • 07:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 3:00:00 on db1161.eqiad.wmnet with reason: T360116
  • 07:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db1161 depool T360116', diff saved to https://phabricator.wikimedia.org/P60571 and previous config saved to /var/cache/conftool/dbconfig/20240416-073521-arnaudb.json
  • 07:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60570 and previous config saved to /var/cache/conftool/dbconfig/20240416-073420-marostegui.json
  • 07:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60569 and previous config saved to /var/cache/conftool/dbconfig/20240416-071913-marostegui.json
  • 07:16 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60568 and previous config saved to /var/cache/conftool/dbconfig/20240416-071611-root.json
  • 07:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60567 and previous config saved to /var/cache/conftool/dbconfig/20240416-070405-marostegui.json
  • 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T361627)', diff saved to https://phabricator.wikimedia.org/P60566 and previous config saved to /var/cache/conftool/dbconfig/20240416-070139-marostegui.json
  • 07:02 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 07:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 07:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60565 and previous config saved to /var/cache/conftool/dbconfig/20240416-070105-root.json
  • 07:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 07:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60564 and previous config saved to /var/cache/conftool/dbconfig/20240416-070100-marostegui.json
  • 06:46 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60563 and previous config saved to /var/cache/conftool/dbconfig/20240416-064559-root.json
  • 06:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60562 and previous config saved to /var/cache/conftool/dbconfig/20240416-064552-marostegui.json
  • 06:37 volans: upgraed spicerack to v8.5.0 on cumin1002
  • 06:36 volans@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:05:00 on cumin2002.codfw.wmnet with reason: test spicerack v8.5.0
  • 06:36 volans@cumin2002: START - Cookbook sre.hosts.downtime for 0:05:00 on cumin2002.codfw.wmnet with reason: test spicerack v8.5.0
  • 06:30 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60561 and previous config saved to /var/cache/conftool/dbconfig/20240416-063053-root.json
  • 06:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60560 and previous config saved to /var/cache/conftool/dbconfig/20240416-063045-marostegui.json
  • 06:15 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60559 and previous config saved to /var/cache/conftool/dbconfig/20240416-061546-root.json
  • 06:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60558 and previous config saved to /var/cache/conftool/dbconfig/20240416-061536-marostegui.json
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T361627)', diff saved to https://phabricator.wikimedia.org/P60557 and previous config saved to /var/cache/conftool/dbconfig/20240416-060826-marostegui.json
  • 06:08 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 06:08 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 06:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T361627)', diff saved to https://phabricator.wikimedia.org/P60556 and previous config saved to /var/cache/conftool/dbconfig/20240416-060803-marostegui.json
  • 06:00 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60555 and previous config saved to /var/cache/conftool/dbconfig/20240416-060034-root.json
  • 05:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60554 and previous config saved to /var/cache/conftool/dbconfig/20240416-055256-marostegui.json
  • 05:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T352010)', diff saved to https://phabricator.wikimedia.org/P60553 and previous config saved to /var/cache/conftool/dbconfig/20240416-055237-ladsgroup.json
  • 05:52 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 05:52 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 05:52 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60552 and previous config saved to /var/cache/conftool/dbconfig/20240416-055215-ladsgroup.json
  • 05:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2156.codfw.wmnet with OS bookworm
  • 05:45 marostegui@cumin1002: dbctl commit (dc=all): 'db2156 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60551 and previous config saved to /var/cache/conftool/dbconfig/20240416-054528-root.json
  • 05:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60550 and previous config saved to /var/cache/conftool/dbconfig/20240416-053749-marostegui.json
  • 05:37 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60549 and previous config saved to /var/cache/conftool/dbconfig/20240416-053706-ladsgroup.json
  • 05:26 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2156.codfw.wmnet with reason: host reimage
  • 05:24 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2156.codfw.wmnet with reason: host reimage
  • 05:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T361627)', diff saved to https://phabricator.wikimedia.org/P60548 and previous config saved to /var/cache/conftool/dbconfig/20240416-052241-marostegui.json
  • 05:21 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60547 and previous config saved to /var/cache/conftool/dbconfig/20240416-052158-ladsgroup.json
  • 05:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db2107 (T361627)', diff saved to Unable to send diff to phaste and previous config saved to /var/cache/conftool/dbconfig/20240416-051623-marostegui.json
  • 05:16 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 05:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 05:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 05:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 6:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 05:06 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60546 and previous config saved to /var/cache/conftool/dbconfig/20240416-050651-ladsgroup.json
  • 05:04 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2156.codfw.wmnet with OS bookworm
  • 05:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2156', diff saved to https://phabricator.wikimedia.org/P60545 and previous config saved to /var/cache/conftool/dbconfig/20240416-050315-root.json
  • 04:03 mwpresync@deploy1002: Finished scap: testwikis wikis to 1.43.0-wmf.1 refs T361395 (duration: 57m 31s)
  • 03:05 mwpresync@deploy1002: Started scap: testwikis wikis to 1.43.0-wmf.1 refs T361395
  • 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.24 (duration: 03m 11s)
  • 02:51 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 02:51 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 02:42 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 02:42 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 02:38 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 02:38 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply

2024-04-15

  • 23:35 eileen: civicrm upgraded from 0445bfaa to fdd12ed1
  • 23:17 eileen: civicrm upgraded from 4d5a4fc3 to 0445bfaa
  • 22:57 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:57 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:44 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:44 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2 days, 0:00:00 on 19 hosts with reason: T362508
  • 22:09 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2 days, 0:00:00 on 19 hosts with reason: T362508
  • 21:48 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:48 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:45 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:45 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:37 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:30 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:30 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:14 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:14 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 kindrobot: closing the UTC late backport window
  • 21:00 kindrobot@deploy1002: Finished scap: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427) (duration: 18m 30s)
  • 20:48 kindrobot@deploy1002: s8321414 and kindrobot: Continuing with sync
  • 20:44 kindrobot@deploy1002: s8321414 and kindrobot: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:42 kindrobot@deploy1002: Started scap: Backport for zhwikivoyage: Make RelatedArticles extension usable on zhwikivoyage (T361427)
  • 20:37 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:37 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:36 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:36 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:36 kindrobot@deploy1002: Finished scap: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace (duration: 17m 06s)
  • 20:35 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:34 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:34 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:34 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 20:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 20:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60539 and previous config saved to /var/cache/conftool/dbconfig/20240415-202943-marostegui.json
  • 20:24 kindrobot@deploy1002: jdlrobson and kindrobot: Continuing with sync
  • 20:21 kindrobot@deploy1002: jdlrobson and kindrobot: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:19 kindrobot@deploy1002: Started scap: Backport for Enable desktop watchlist on beta cluster, clean up old references (T109277), Enable night mode on template namespace
  • 20:19 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:19 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P60538 and previous config saved to /var/cache/conftool/dbconfig/20240415-201436-marostegui.json
  • 20:13 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:13 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:12 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:12 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:06 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:06 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:05 kindrobot: staring UTC late backport window
  • 19:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P60537 and previous config saved to /var/cache/conftool/dbconfig/20240415-195928-marostegui.json
  • 19:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60536 and previous config saved to /var/cache/conftool/dbconfig/20240415-194420-marostegui.json
  • 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T352010)', diff saved to https://phabricator.wikimedia.org/P60535 and previous config saved to /var/cache/conftool/dbconfig/20240415-193921-ladsgroup.json
  • 19:39 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 19:39 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 19:39 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60534 and previous config saved to /var/cache/conftool/dbconfig/20240415-193858-ladsgroup.json
  • 19:23 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60533 and previous config saved to /var/cache/conftool/dbconfig/20240415-192350-ladsgroup.json
  • 19:12 mutante: deleting unused kibana-next.svc records from DNS - T234854
  • 19:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60532 and previous config saved to /var/cache/conftool/dbconfig/20240415-190842-ladsgroup.json
  • 19:01 mutante: deleting unused cas-logstash.wikimedia.org from DNS
  • 18:53 ladsgroup@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60531 and previous config saved to /var/cache/conftool/dbconfig/20240415-185334-ladsgroup.json
  • 18:51 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=ncredir1001.eqiad.wmnet,service=nginx
  • 18:51 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1001.eqiad.wmnet,service=nginx
  • 18:50 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T356166)', diff saved to https://phabricator.wikimedia.org/P60530 and previous config saved to /var/cache/conftool/dbconfig/20240415-185008-marostegui.json
  • 18:50 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 18:49 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60529 and previous config saved to /var/cache/conftool/dbconfig/20240415-184945-marostegui.json
  • 18:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1002.eqiad.wmnet,service=nginx
  • 18:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=ncredir1001.eqiad.wmnet,service=nginx
  • 18:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 18:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P60528 and previous config saved to /var/cache/conftool/dbconfig/20240415-183437-marostegui.json
  • 18:34 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
  • 18:24 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
  • 18:22 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1002.eqiad.wmnet with OS bullseye
  • 18:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P60527 and previous config saved to /var/cache/conftool/dbconfig/20240415-181930-marostegui.json
  • 18:15 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 18:12 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 18:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60526 and previous config saved to /var/cache/conftool/dbconfig/20240415-180422-marostegui.json
  • 18:02 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 18:00 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 18:00 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:59 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 17:58 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1002.eqiad.wmnet with reason: host reimage
  • 17:43 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1002.eqiad.wmnet with OS bullseye
  • 17:42 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T361647 - bking@cumin2002
  • 17:38 brett@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host ncredir1001.eqiad.wmnet with OS bullseye
  • 17:23 taavi@deploy1002: Finished scap: Backport for wmf-config: add private subnets for magru (T346722) (duration: 17m 21s)
  • 17:13 jynus: stop db2139 dbs for upgrade T360751
  • 17:10 taavi@deploy1002: taavi and sukhe: Continuing with sync
  • 17:08 taavi@deploy1002: taavi and sukhe: Backport for wmf-config: add private subnets for magru (T346722) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2101.codfw.wmnet
  • 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 17:06 jynus@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2101.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002"
  • 17:06 taavi@deploy1002: Started scap: Backport for wmf-config: add private subnets for magru (T346722)
  • 17:05 jynus@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2101.codfw.wmnet decommissioned, removing all IPs except the asset tag one - jynus@cumin2002"
  • 17:03 jynus@cumin2002: START - Cookbook sre.dns.netbox
  • 16:57 jynus@cumin2002: START - Cookbook sre.hosts.decommission for hosts db2101.codfw.wmnet
  • 16:32 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T356166)', diff saved to https://phabricator.wikimedia.org/P60524 and previous config saved to /var/cache/conftool/dbconfig/20240415-163011-marostegui.json
  • 16:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60523 and previous config saved to /var/cache/conftool/dbconfig/20240415-162949-marostegui.json
  • 16:28 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:23 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster search_codfw: T361647 - bking@cumin2002
  • 16:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:19 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:14 brett@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 16:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P60522 and previous config saved to /var/cache/conftool/dbconfig/20240415-161441-marostegui.json
  • 16:14 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:14 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:11 brett@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on ncredir1001.eqiad.wmnet with reason: host reimage
  • 16:10 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:10 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:08 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:08 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:05 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:05 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:02 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bullseye
  • 16:01 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:01 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P60521 and previous config saved to /var/cache/conftool/dbconfig/20240415-155932-marostegui.json
  • 15:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:56 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:56 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:55 brett@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host ncredir1001.eqiad.wmnet with OS bullseye
  • 15:55 brett@cumin2002: START - Cookbook sre.hosts.reimage for host ncredir1001.eqiad.wmnet with OS bullseye
  • 15:53 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:53 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:51 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:50 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
  • 15:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:45 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:45 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60520 and previous config saved to /var/cache/conftool/dbconfig/20240415-154422-marostegui.json
  • 15:40 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
  • 15:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60519 and previous config saved to /var/cache/conftool/dbconfig/20240415-152132-arnaudb.json
  • 15:19 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:18 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:15 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 15:11 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.rolling-operation (exit_code=0) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002
  • 15:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60518 and previous config saved to /var/cache/conftool/dbconfig/20240415-150626-arnaudb.json
  • 15:04 dancy@deploy1002: Installation of scap version "4.76.0" completed for 340 hosts
  • 15:04 Daimona: Running query for T362365#9710047
  • 15:03 dancy@deploy1002: Installing scap version "4.76.0" for 340 hosts
  • 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T356166)', diff saved to https://phabricator.wikimedia.org/P60517 and previous config saved to /var/cache/conftool/dbconfig/20240415-150257-marostegui.json
  • 15:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 15:02 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 15:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60516 and previous config saved to /var/cache/conftool/dbconfig/20240415-150235-marostegui.json
  • 14:52 Dreamy_Jazz: Afternoon backport window done
  • 14:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60515 and previous config saved to /var/cache/conftool/dbconfig/20240415-145120-arnaudb.json
  • 14:50 dreamyjazz@deploy1002: Finished scap: Backport for Define 'useYear' as true for temp user serial mapping config (T349506) (duration: 16m 16s)
  • 14:48 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster cloudelastic: T361647 - bking@cumin2002
  • 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60514 and previous config saved to /var/cache/conftool/dbconfig/20240415-144725-marostegui.json
  • 14:41 jynus: fixed grants for db2098
  • 14:37 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
  • 14:36 dreamyjazz@deploy1002: dreamyjazz: Backport for Define 'useYear' as true for temp user serial mapping config (T349506) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60513 and previous config saved to /var/cache/conftool/dbconfig/20240415-143614-arnaudb.json
  • 14:34 dreamyjazz@deploy1002: Started scap: Backport for Define 'useYear' as true for temp user serial mapping config (T349506)
  • 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P60512 and previous config saved to /var/cache/conftool/dbconfig/20240415-143217-marostegui.json
  • 14:31 urbanecm@deploy1002: Finished scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) (duration: 30m 12s)
  • 14:31 stevemunene@deploy1002: helmfile [staging] DONE helmfile.d/services/datahub: sync on main
  • 14:30 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp
  • 14:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2111.codfw.wmnet with OS bookworm
  • 14:23 stevemunene@deploy1002: helmfile [staging] START helmfile.d/services/datahub: apply on main
  • 14:21 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp4051.ulsfo.wmnet,cp5030.eqsin.wmnet,cp5032.eqsin.wmnet} and A:cp
  • 14:18 urbanecm@deploy1002: urbanecm and dreamyjazz and arthurtaylor: Continuing with sync
  • 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60511 and previous config saved to /var/cache/conftool/dbconfig/20240415-141710-marostegui.json
  • 14:16 elukey: move cassandra instances on cassandra-dev to pki - T352647
  • 14:14 urbanecm@deploy1002: urbanecm and dreamyjazz and arthurtaylor: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:13 vgutierrez: uploaded tcp-mss-clamper 0.4+deb11u2 to bullseye-wikimedia (apt.wm.o)
  • 14:09 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
  • 14:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2111.codfw.wmnet with reason: host reimage
  • 14:04 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (1 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
  • 14:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2111.codfw.wmnet with reason: host reimage
  • 14:04 bking@cumin2002: END (ERROR) - Cookbook sre.elasticsearch.rolling-operation (exit_code=97) Operation.RESTART (3 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
  • 14:04 bking@cumin2002: START - Cookbook sre.elasticsearch.rolling-operation Operation.RESTART (3 nodes at a time) for ElasticSearch cluster relforge: T361647 - bking@cumin2002
  • 14:01 urbanecm@deploy1002: Started scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169)
  • 13:59 jynus: update dbprov2005 dbbackups password T362509
  • 13:58 urbanecm@deploy1002: sync-world aborted: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) (duration: 52m 11s)
  • 13:54 TheresNoTime: `[samtar@mwmaint1002 ~]$ mwscript extensions/Flow/maintenance/FlowFixInconsistentBoards.php --wiki=zhwiki --namespaceName User_talk` T362530
  • 13:54 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 13:48 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2111.codfw.wmnet with OS bookworm
  • 13:47 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 depool', diff saved to https://phabricator.wikimedia.org/P60510 and previous config saved to /var/cache/conftool/dbconfig/20240415-134710-arnaudb.json
  • 13:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db2111.codfw.wmnet with reason: reboot multiinstance replica
  • 13:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db2111.codfw.wmnet with reason: reboot multiinstance replica
  • 13:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 100%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60509 and previous config saved to /var/cache/conftool/dbconfig/20240415-134522-arnaudb.json
  • 13:45 vgutierrez: update thirdparty/haproxy28 to 2.8.9 for bullseye-wikimedia (apt.wm.o)
  • 13:37 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-esams and not P{cp3066.esams.wmnet,cp3069.esams.wmnet,cp3070.esams.wmnet,cp3071.esams.wmnet,cp3072.esams.wmnet,cp3073.esams.wmnet} and A:cp
  • 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T356166)', diff saved to https://phabricator.wikimedia.org/P60508 and previous config saved to /var/cache/conftool/dbconfig/20240415-133433-marostegui.json
  • 13:34 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:34 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 13:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60507 and previous config saved to /var/cache/conftool/dbconfig/20240415-133410-marostegui.json
  • 13:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 75%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60506 and previous config saved to /var/cache/conftool/dbconfig/20240415-133016-arnaudb.json
  • 13:19 volans: upgraed spicerack to v8.5.0 on cumin2002
  • 13:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P60505 and previous config saved to /var/cache/conftool/dbconfig/20240415-131902-marostegui.json
  • 13:16 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp1115.eqiad.wmnet
  • 13:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 50%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60504 and previous config saved to /var/cache/conftool/dbconfig/20240415-131510-arnaudb.json
  • 13:08 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-esams and not P{cp3066.esams.wmnet,cp3069.esams.wmnet,cp3070.esams.wmnet,cp3071.esams.wmnet,cp3072.esams.wmnet,cp3073.esams.wmnet} and A:cp
  • 13:08 urbanecm@deploy1002: urbanecm and arthurtaylor and dreamyjazz: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:06 urbanecm@deploy1002: Started scap: Backport for Add wgAutoCreateTempUser configuration for production (T349506 T337090), Change mul deployment on beta to limited version (T356169)
  • 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P60503 and previous config saved to /var/cache/conftool/dbconfig/20240415-130355-marostegui.json
  • 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 (re)pooling @ 25%: post reimage repool', diff saved to https://phabricator.wikimedia.org/P60502 and previous config saved to /var/cache/conftool/dbconfig/20240415-130005-arnaudb.json
  • 12:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2128.codfw.wmnet with OS bookworm
  • 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60501 and previous config saved to /var/cache/conftool/dbconfig/20240415-124848-marostegui.json
  • 12:12 jynus: deploy new database grants for m1 <- dbbprov1005
  • 12:09 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2128.codfw.wmnet with OS bookworm
  • 12:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T356166)', diff saved to https://phabricator.wikimedia.org/P60500 and previous config saved to /var/cache/conftool/dbconfig/20240415-120650-marostegui.json
  • 12:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 12:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 12:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60499 and previous config saved to /var/cache/conftool/dbconfig/20240415-120627-marostegui.json
  • 12:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2128.codfw.wmnet
  • 12:04 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-eqiad and not P{cp1112.eqiad.wmnet,cp1113.eqiad.wmnet,cp1115.eqiad.wmnet} and A:cp
  • 12:00 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2128.codfw.wmnet
  • 11:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db[2128,2186].codfw.wmnet with reason: upgrade db2128 T360116
  • 11:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db[2128,2186].codfw.wmnet with reason: upgrade db2128 T360116
  • 11:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2128 depool T360116', diff saved to https://phabricator.wikimedia.org/P60498 and previous config saved to /var/cache/conftool/dbconfig/20240415-115708-arnaudb.json
  • 11:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P60497 and previous config saved to /var/cache/conftool/dbconfig/20240415-115118-marostegui.json
  • 11:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P60496 and previous config saved to /var/cache/conftool/dbconfig/20240415-113610-marostegui.json
  • 11:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60495 and previous config saved to /var/cache/conftool/dbconfig/20240415-112102-marostegui.json
  • 11:13 volans: uploaded spicerack_8.5.0 to apt.wikimedia.org bullseye-wikimedia
  • 11:07 moritzm: imported shellcheck 0.7.1-1~bpo10+1 to component/shellcheck T362518
  • 11:03 btullis@cumin1002: END (FAIL) - Cookbook sre.hadoop.roll-restart-masters (exit_code=99) restart masters for Hadoop analytics cluster: Restart of jvm daemons.
  • 10:46 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop analytics cluster: Restart of jvm daemons.
  • 10:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T356166)', diff saved to https://phabricator.wikimedia.org/P60494 and previous config saved to /var/cache/conftool/dbconfig/20240415-103853-marostegui.json
  • 10:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 10:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 10:33 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 10:31 godog: bounce prometheus@k8s-staging in eqiad - T343529
  • 10:31 moritzm: imported lilypond/lilypond-data 2.22.0-10~bpo10+1 to component/lilypond T362518
  • 10:22 claime: Launching build-base-images on build2001 - T362518
  • 10:10 hashar@deploy1002: Finished deploy [gerrit/gerrit@47eacb9]: Update Javascript plugins for Gerrit 3.8 - T354886 (duration: 00m 07s)
  • 10:10 hashar@deploy1002: Started deploy [gerrit/gerrit@47eacb9]: Update Javascript plugins for Gerrit 3.8 - T354886
  • 09:57 hashar@deploy1002: Finished deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit1003 - T354886 (duration: 00m 06s)
  • 09:56 hashar@deploy1002: Started deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit1003 - T354886
  • 09:53 hashar@deploy1002: Finished deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit2002 - T354886 (duration: 00m 08s)
  • 09:52 hashar@deploy1002: Started deploy [gerrit/gerrit@2f3d3d4]: Gerrit to 3.8.5 on gerrit2002 - T354886
  • 09:50 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-masters (exit_code=0) restart masters for Hadoop test cluster: Restart of jvm daemons.
  • 09:26 cgoubert@deploy1002: Finished scap: T351237 (duration: 11m 43s)
  • 09:14 cgoubert@deploy1002: Started scap: T351237
  • 09:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60493 and previous config saved to /var/cache/conftool/dbconfig/20240415-091145-marostegui.json
  • 09:09 ladsgroup@deploy1002: Finished scap: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237) (duration: 08m 51s)
  • 09:08 ladsgroup@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T352010)', diff saved to https://phabricator.wikimedia.org/P60492 and previous config saved to /var/cache/conftool/dbconfig/20240415-090834-ladsgroup.json
  • 09:08 ladsgroup@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:08 ladsgroup@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:03 ladsgroup@deploy1002: ladsgroup: Continuing with sync
  • 09:02 ladsgroup@deploy1002: ladsgroup: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 09:00 ladsgroup@deploy1002: Started scap: Backport for Set all wikis to read new for pagelinks migration except trwiki, zhwiki (T351237)
  • 08:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P60491 and previous config saved to /var/cache/conftool/dbconfig/20240415-085638-marostegui.json
  • 08:54 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-masters restart masters for Hadoop test cluster: Restart of jvm daemons.
  • 08:53 jynus: restart dbprov2005
  • 08:46 godog: logstash.w.o now uses sso - T246998
  • 08:45 btullis@cumin1002: END (PASS) - Cookbook sre.druid.roll-restart-workers (exit_code=0) for Druid test cluster: Roll restart of Druid jvm daemons.
  • 08:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P60490 and previous config saved to /var/cache/conftool/dbconfig/20240415-084130-marostegui.json
  • 08:35 btullis@cumin1002: START - Cookbook sre.druid.roll-restart-workers for Druid test cluster: Roll restart of Druid jvm daemons.
  • 08:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60488 and previous config saved to /var/cache/conftool/dbconfig/20240415-082623-marostegui.json
  • 08:01 Emperor: depool wdqs in codfw T362508
  • 08:01 mvernon@cumin2002: conftool action : set/pooled=false; selector: dnsdisc=wdqs,name=codfw
  • 07:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica
  • 07:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on db1217.eqiad.wmnet with reason: reboot multiinstance replica
  • 07:48 jayme: restarting k8s-mlstaging and k8s-staging prometheus instances - T343529
  • 07:11 dcausse: restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T356166)', diff saved to https://phabricator.wikimedia.org/P60487 and previous config saved to /var/cache/conftool/dbconfig/20240415-065659-marostegui.json
  • 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60486 and previous config saved to /var/cache/conftool/dbconfig/20240415-065636-marostegui.json
  • 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60485 and previous config saved to /var/cache/conftool/dbconfig/20240415-064129-marostegui.json
  • 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P60484 and previous config saved to /var/cache/conftool/dbconfig/20240415-062621-marostegui.json
  • 06:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60483 and previous config saved to /var/cache/conftool/dbconfig/20240415-061114-marostegui.json
  • 05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T356166)', diff saved to https://phabricator.wikimedia.org/P60482 and previous config saved to /var/cache/conftool/dbconfig/20240415-053001-marostegui.json
  • 05:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 05:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance

2024-04-14

  • 16:00 marostegui@cumin1002: dbctl commit (dc=all): 'Set db2142 as x2 codfw master', diff saved to https://phabricator.wikimedia.org/P60481 and previous config saved to /var/cache/conftool/dbconfig/20240414-160016-marostegui.json
  • 11:22 marostegui: Restart x2 codfw master
  • 11:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on 6 hosts with reason: Investigating
  • 11:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on 6 hosts with reason: Investigating

2024-04-13

  • 23:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60479 and previous config saved to /var/cache/conftool/dbconfig/20240413-233953-marostegui.json
  • 23:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60478 and previous config saved to /var/cache/conftool/dbconfig/20240413-232443-marostegui.json
  • 23:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60477 and previous config saved to /var/cache/conftool/dbconfig/20240413-230935-marostegui.json
  • 22:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60476 and previous config saved to /var/cache/conftool/dbconfig/20240413-225428-marostegui.json
  • 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60475 and previous config saved to /var/cache/conftool/dbconfig/20240413-154240-marostegui.json
  • 15:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 15:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 15:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60474 and previous config saved to /var/cache/conftool/dbconfig/20240413-154217-marostegui.json
  • 15:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60473 and previous config saved to /var/cache/conftool/dbconfig/20240413-152709-marostegui.json
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60472 and previous config saved to /var/cache/conftool/dbconfig/20240413-151201-marostegui.json
  • 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60471 and previous config saved to /var/cache/conftool/dbconfig/20240413-145653-marostegui.json
  • 06:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60470 and previous config saved to /var/cache/conftool/dbconfig/20240413-060646-marostegui.json
  • 06:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 06:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 00:52 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host cp1115.eqiad.wmnet

2024-04-12

  • 21:03 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 21:03 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 20:43 cdanis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 20:43 cdanis@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 19:36 bking@cumin2002: END (PASS) - Cookbook sre.elasticsearch.ban (exit_code=0) Unbanning all hosts in search_codfw
  • 19:36 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Unbanning all hosts in search_codfw
  • 18:56 andrew@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts cloudbackup2002.codfw.wmnet
  • 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:56 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:55 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2002.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:52 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 18:47 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2002.codfw.wmnet
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts cloudbackup2001.codfw.wmnet
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:46 andrew@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:44 andrew@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: cloudbackup2001.codfw.wmnet decommissioned, removing all IPs except the asset tag one - andrew@cumin1002"
  • 18:40 andrew@cumin1002: START - Cookbook sre.dns.netbox
  • 18:35 andrew@cumin1002: START - Cookbook sre.hosts.decommission for hosts cloudbackup2001.codfw.wmnet
  • 17:00 mutante: crm2001 - on initial puppet run adding envoy build-envoy-config failed building config and service failed due to dependency issue. manual run of "sudo /usr/local/sbin/build-envoy-config -c /etc/envoy/" and restarted envoyproxy.service
  • 16:19 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 16:16 elukey: move cassandra instances on cassandra-dev to the new truststore (allowing PKI certs) - T352647
  • 15:59 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 15:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 15:56 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 4 days, 0:00:00 on cp1115.eqiad.wmnet with reason: testing PXE boot issues
  • 15:55 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'readability' for release 'main' .
  • 15:53 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:52 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
  • 15:51 bking@cumin2002: START - Cookbook sre.hosts.downtime for 1:00:00 on elastic2090.codfw.wmnet with reason: T353878
  • 15:51 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-reverted' for release 'main' .
  • 15:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-goodfaith' for release 'main' .
  • 15:50 bking@cumin2002: END (FAIL) - Cookbook sre.elasticsearch.ban (exit_code=99) Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
  • 15:50 bking@cumin2002: START - Cookbook sre.elasticsearch.ban Banning hosts: elastic2090 for reboot to get rid of broken systemd units - bking@cumin2002 - T353878
  • 15:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-drafttopic' for release 'main' .
  • 15:49 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-draftquality' for release 'main' .
  • 15:48 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host cp1115.eqiad.wmnet
  • 15:47 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articletopic' for release 'main' .
  • 15:46 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:46 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-articlequality' for release 'main' .
  • 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:23 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:22 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'articletopic-outlink' for release 'main' .
  • 15:21 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:07 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'article-descriptions' for release 'main' .
  • 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 15:03 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 15:03 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 15:02 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:01 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "magru - ayounsi@cumin1002"
  • 14:59 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:22 hashar@deploy1002: Finished scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) (duration: 16m 29s)
  • 14:19 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revscoring-editquality-damaging' for release 'main' .
  • 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'sync'.
  • 14:18 elukey@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'sync'.
  • 14:17 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:09 hashar@deploy1002: hashar and jforrester: Continuing with sync
  • 14:08 hashar@deploy1002: hashar and jforrester: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:08 sukhe: depool cp1115 for PXE boot issue testing: T350179
  • 14:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 14:05 hashar@deploy1002: Started scap: Backport for Parser::statelessFetchTemplate: don't add interwiki redirects to dependencies (T362221)
  • 12:53 jayme: updated rsyslog to 8.2404.0-1~bpo11+1 on staging-codfw and staging-eqiad k8s clusters - T357616
  • 12:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60466 and previous config saved to /var/cache/conftool/dbconfig/20240412-122045-marostegui.json
  • 12:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P60464 and previous config saved to /var/cache/conftool/dbconfig/20240412-120537-marostegui.json
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 11:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60463 and previous config saved to /var/cache/conftool/dbconfig/20240412-115029-marostegui.json
  • 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 11:06 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 10:55 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=frwiki --search-index (T362367)
  • 09:58 urbanecm: mwmaint1002: mwscript extensions/GrowthExperiments/maintenance/fixLinkRecommendationData.php --wiki=eswiki --search-index (T362367)
  • 09:36 moritzm: installing postgresql-common bugfix updates from Bullseye point release
  • 09:26 moritzm: installing debootstrap bugfix updates from Bullseye point release
  • 09:25 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
  • 09:25 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on matomo1003.eqiad.wmnet with reason: Still in setup
  • 08:56 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab2002.wikimedia.org with reason: Upgrade GitLab to new version
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60461 and previous config saved to /var/cache/conftool/dbconfig/20240412-072435-root.json
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60460 and previous config saved to /var/cache/conftool/dbconfig/20240412-070930-root.json
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60459 and previous config saved to /var/cache/conftool/dbconfig/20240412-065424-root.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60458 and previous config saved to /var/cache/conftool/dbconfig/20240412-063918-root.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60457 and previous config saved to /var/cache/conftool/dbconfig/20240412-062412-root.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60456 and previous config saved to /var/cache/conftool/dbconfig/20240412-060907-root.json
  • 05:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2109.codfw.wmnet with OS bookworm
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60455 and previous config saved to /var/cache/conftool/dbconfig/20240412-055401-root.json
  • 05:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
  • 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2109.codfw.wmnet with reason: host reimage
  • 05:23 moritzm: prune obsolete nginx debs on apt-staging after switch to new nginx provider scheme T329529
  • 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2109.codfw.wmnet with OS bookworm
  • 05:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2109', diff saved to https://phabricator.wikimedia.org/P60454 and previous config saved to /var/cache/conftool/dbconfig/20240412-051606-root.json
  • 03:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T356166)', diff saved to https://phabricator.wikimedia.org/P60453 and previous config saved to /var/cache/conftool/dbconfig/20240412-033317-marostegui.json
  • 03:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 03:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 03:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60452 and previous config saved to /var/cache/conftool/dbconfig/20240412-033254-marostegui.json
  • 03:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60451 and previous config saved to /var/cache/conftool/dbconfig/20240412-031744-marostegui.json
  • 03:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P60450 and previous config saved to /var/cache/conftool/dbconfig/20240412-030237-marostegui.json
  • 02:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60449 and previous config saved to /var/cache/conftool/dbconfig/20240412-024729-marostegui.json
  • 01:05 denisse: Manually deleting /srv/syslog/.linux.dhcp.DictModel/syslog.log from November 30 on centrallog1002 and centrallog2002 after the prune_old_srv_syslog_directories.service failed to delete the non-empty directory - T362376

2024-04-11

  • 23:04 cstone: civicrm upgraded from c2569254 to 4d5a4fc3
  • 20:20 urbanecm@deploy1002: Finished scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames (duration: 17m 38s)
  • 20:08 urbanecm@deploy1002: urbanecm and phuedx: Continuing with sync
  • 20:05 urbanecm@deploy1002: urbanecm and phuedx: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:03 urbanecm@deploy1002: Started scap: Backport for ext-EventLogging: Add mediawiki.product_metrics.wikifunctions_ui to $wgEventLoggingStreamNames
  • 19:41 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 19:40 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T356166)', diff saved to https://phabricator.wikimedia.org/P60448 and previous config saved to /var/cache/conftool/dbconfig/20240411-193537-marostegui.json
  • 19:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 19:35 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 19:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60447 and previous config saved to /var/cache/conftool/dbconfig/20240411-193514-marostegui.json
  • 19:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60446 and previous config saved to /var/cache/conftool/dbconfig/20240411-192006-marostegui.json
  • 19:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P60445 and previous config saved to /var/cache/conftool/dbconfig/20240411-190459-marostegui.json
  • 18:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60443 and previous config saved to /var/cache/conftool/dbconfig/20240411-184951-marostegui.json
  • 17:50 dancy@deploy1002: rebuilt and synchronized wikiversions files: group2 wikis to 1.42.0-wmf.26 refs T360158
  • 17:27 swfrench@deploy1002: Finished scap: (no justification provided) (duration: 07m 57s)
  • 17:20 swfrench@deploy1002: Started scap: (no justification provided)
  • 17:12 bd808@deploy1002: helmfile [eqiad] DONE helmfile.d/services/developer-portal: apply
  • 17:12 bd808@deploy1002: helmfile [eqiad] START helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [codfw] DONE helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [codfw] START helmfile.d/services/developer-portal: apply
  • 17:11 bd808@deploy1002: helmfile [staging] DONE helmfile.d/services/developer-portal: apply
  • 17:10 bd808@deploy1002: helmfile [staging] START helmfile.d/services/developer-portal: apply
  • 16:45 hashar@deploy1002: Finished scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) (duration: 16m 47s)
  • 16:33 hashar@deploy1002: hashar: Continuing with sync
  • 16:31 hashar@deploy1002: hashar: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 16:29 hashar@deploy1002: Started scap: Backport for Revert "Update mobile search for dark mode, remove unused functions in MobilePage.php" (T362297)
  • 16:27 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60442 and previous config saved to /var/cache/conftool/dbconfig/20240411-161536-arnaudb.json
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60441 and previous config saved to /var/cache/conftool/dbconfig/20240411-161522-arnaudb.json
  • 16:03 herron: beginning rolling hardware upgrades for titan100[12] T361251
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60440 and previous config saved to /var/cache/conftool/dbconfig/20240411-160030-arnaudb.json
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60439 and previous config saved to /var/cache/conftool/dbconfig/20240411-160016-arnaudb.json
  • 15:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60438 and previous config saved to /var/cache/conftool/dbconfig/20240411-155836-root.json
  • 15:56 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe2002.codfw.wmnet with OS bookworm
  • 15:51 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host moss-fe1002.eqiad.wmnet with OS bookworm
  • 15:47 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-drmrs and A:cp
  • 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hadoop.roll-restart-workers (exit_code=0) restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60437 and previous config saved to /var/cache/conftool/dbconfig/20240411-154524-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60436 and previous config saved to /var/cache/conftool/dbconfig/20240411-154510-arnaudb.json
  • 15:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60435 and previous config saved to /var/cache/conftool/dbconfig/20240411-154330-root.json
  • 15:39 mvernon@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
  • 15:36 mvernon@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe2002.codfw.wmnet with reason: host reimage
  • 15:35 mvernon@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
  • 15:33 btullis@cumin1002: START - Cookbook sre.hadoop.roll-restart-workers restart workers for Hadoop test cluster: Roll restart of jvm daemons for openjdk upgrade.
  • 15:31 mvernon@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on moss-fe1002.eqiad.wmnet with reason: host reimage
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2111 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60434 and previous config saved to /var/cache/conftool/dbconfig/20240411-153019-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2110 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60433 and previous config saved to /var/cache/conftool/dbconfig/20240411-153003-arnaudb.json
  • 15:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60432 and previous config saved to /var/cache/conftool/dbconfig/20240411-152825-root.json
  • 15:24 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:24 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:24 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:23 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:20 mvernon@cumin2002: START - Cookbook sre.hosts.reimage for host moss-fe2002.codfw.wmnet with OS bookworm
  • 15:18 mvernon@cumin1002: START - Cookbook sre.hosts.reimage for host moss-fe1002.eqiad.wmnet with OS bookworm
  • 15:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60431 and previous config saved to /var/cache/conftool/dbconfig/20240411-151319-root.json
  • 15:12 dancy@deploy1002: Finished scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) (duration: 17m 41s)
  • 15:11 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:00 dancy@deploy1002: dancy: Continuing with sync
  • 14:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60430 and previous config saved to /var/cache/conftool/dbconfig/20240411-145841-arnaudb.json
  • 14:58 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60429 and previous config saved to /var/cache/conftool/dbconfig/20240411-145813-root.json
  • 14:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-drmrs and A:cp
  • 14:57 dancy@deploy1002: dancy: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repool', diff saved to https://phabricator.wikimedia.org/P60428 and previous config saved to /var/cache/conftool/dbconfig/20240411-145658-arnaudb.json
  • 14:54 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
  • 14:54 dancy@deploy1002: Started scap: Backport for static.php: Handle mediawiki.org/ontology/ontology.owl (T171807 T359643)
  • 14:52 sukhe: sudo cumin "A:cp and A:esams" "run-puppet-agent --enable 'merging CR 1014571'"
  • 14:52 dreamyjazz@deploy1002: Finished scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) (duration: 24m 11s)
  • 14:47 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:45 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:44 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 100%: reool', diff saved to https://phabricator.wikimedia.org/P60427 and previous config saved to /var/cache/conftool/dbconfig/20240411-144416-arnaudb.json
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60426 and previous config saved to /var/cache/conftool/dbconfig/20240411-144336-arnaudb.json
  • 14:43 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 100%: repool', diff saved to https://phabricator.wikimedia.org/P60425 and previous config saved to /var/cache/conftool/dbconfig/20240411-144311-arnaudb.json
  • 14:43 sukhe: sudo cumin "A:cp and A:esams" "disable-puppet 'merging CR 1014571'"
  • 14:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60424 and previous config saved to /var/cache/conftool/dbconfig/20240411-144307-root.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repool', diff saved to https://phabricator.wikimedia.org/P60423 and previous config saved to /var/cache/conftool/dbconfig/20240411-144152-arnaudb.json
  • 14:39 dreamyjazz@deploy1002: dreamyjazz and esanders: Continuing with sync
  • 14:36 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 14:34 moritzm: installing distro-info-data updates from Bullseye point release
  • 14:31 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 14:30 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2149.codfw.wmnet with OS bookworm
  • 14:30 dreamyjazz@deploy1002: dreamyjazz and esanders: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 75%: reool', diff saved to https://phabricator.wikimedia.org/P60422 and previous config saved to /var/cache/conftool/dbconfig/20240411-142910-arnaudb.json
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60421 and previous config saved to /var/cache/conftool/dbconfig/20240411-142830-arnaudb.json
  • 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
  • 14:28 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 75%: repool', diff saved to https://phabricator.wikimedia.org/P60420 and previous config saved to /var/cache/conftool/dbconfig/20240411-142806-arnaudb.json
  • 14:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2149 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60419 and previous config saved to /var/cache/conftool/dbconfig/20240411-142801-root.json
  • 14:27 dreamyjazz@deploy1002: Started scap: Backport for Set wgMFFallbackEditor to visual for most VE wikis (T361134)
  • 14:27 Dreamy_Jazz: Extending UTC Afternoon backport window
  • 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repool', diff saved to https://phabricator.wikimedia.org/P60418 and previous config saved to /var/cache/conftool/dbconfig/20240411-142645-arnaudb.json
  • 14:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp2042.codfw.wmnet with OS bullseye
  • 14:24 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3073.esams.wmnet with OS bullseye
  • 14:19 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 14:18 elukey: drain and restart cassandra-b on aqs2007 - didn't pick up the new truststore during the past roll restart - T352647
  • 14:15 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
  • 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 50%: reool', diff saved to https://phabricator.wikimedia.org/P60417 and previous config saved to /var/cache/conftool/dbconfig/20240411-141404-arnaudb.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60416 and previous config saved to /var/cache/conftool/dbconfig/20240411-141324-arnaudb.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 50%: repool', diff saved to https://phabricator.wikimedia.org/P60415 and previous config saved to /var/cache/conftool/dbconfig/20240411-141300-arnaudb.json
  • 14:12 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repool', diff saved to https://phabricator.wikimedia.org/P60414 and previous config saved to /var/cache/conftool/dbconfig/20240411-141139-arnaudb.json
  • 14:11 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:10 elukey: move cassandra instances on aqs1010 to PKI TLS certs - T352647
  • 14:10 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
  • 14:09 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:09 Dreamy_Jazz: Afternoon UTC backport window finished
  • 14:09 moritzm: installing NSS security updates
  • 14:08 dreamyjazz@deploy1002: Finished scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) (duration: 17m 42s)
  • 14:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2149.codfw.wmnet with reason: host reimage
  • 14:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
  • 14:06 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:03 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp2042.codfw.wmnet with reason: host reimage
  • 14:01 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
  • 13:59 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
  • 13:59 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Upgrade to PKI
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'db2103 (re)pooling @ 25%: reool', diff saved to https://phabricator.wikimedia.org/P60413 and previous config saved to /var/cache/conftool/dbconfig/20240411-135858-arnaudb.json
  • 13:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60412 and previous config saved to /var/cache/conftool/dbconfig/20240411-135846-root.json
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 20%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60411 and previous config saved to /var/cache/conftool/dbconfig/20240411-135819-arnaudb.json
  • 13:58 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3073.esams.wmnet with reason: host reimage
  • 13:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2109 (re)pooling @ 25%: repool', diff saved to https://phabricator.wikimedia.org/P60410 and previous config saved to /var/cache/conftool/dbconfig/20240411-135754-arnaudb.json
  • 13:57 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-codfw and not P{cp2042.codfw.wmnet} and A:cp
  • 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repool', diff saved to https://phabricator.wikimedia.org/P60409 and previous config saved to /var/cache/conftool/dbconfig/20240411-135634-arnaudb.json
  • 13:55 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
  • 13:55 dreamyjazz@deploy1002: dreamyjazz: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:54 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 13:53 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2149.codfw.wmnet with OS bookworm
  • 13:50 dreamyjazz@deploy1002: Started scap: Backport for Ignore missing title/page in CheckUserLookupUtils::getManualLogEntryFromRow (T362284)
  • 13:49 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2149', diff saved to https://phabricator.wikimedia.org/P60408 and previous config saved to /var/cache/conftool/dbconfig/20240411-134932-root.json
  • 13:49 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:46 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:46 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp2042.codfw.wmnet with OS bullseye
  • 13:45 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp2042.codfw.wmnet,service=(cdn|ats-be)
  • 13:43 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60407 and previous config saved to /var/cache/conftool/dbconfig/20240411-134341-root.json
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 10%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60406 and previous config saved to /var/cache/conftool/dbconfig/20240411-134312-arnaudb.json
  • 13:41 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 13:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 13:34 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3073.esams.wmnet with OS bullseye
  • 13:32 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3073.esams.wmnet,service=(cdn|ats-be)
  • 13:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
  • 13:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db2160.codfw.wmnet with reason: reboot multiinstance replica
  • 13:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:31 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2133.codfw.wmnet
  • 13:28 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60405 and previous config saved to /var/cache/conftool/dbconfig/20240411-132834-root.json
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 5%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60404 and previous config saved to /var/cache/conftool/dbconfig/20240411-132807-arnaudb.json
  • 13:27 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-eqsin and not P{cp[5030,5032].eqsin.wmnet} and A:cp
  • 13:26 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2133.codfw.wmnet
  • 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
  • 13:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2133,2160].codfw.wmnet with reason: reboot
  • 13:23 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2135.codfw.wmnet
  • 13:18 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2135.codfw.wmnet
  • 13:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
  • 13:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2135,2160].codfw.wmnet with reason: reboot
  • 13:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2134.codfw.wmnet
  • 13:13 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60403 and previous config saved to /var/cache/conftool/dbconfig/20240411-131327-root.json
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 4%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60402 and previous config saved to /var/cache/conftool/dbconfig/20240411-131301-arnaudb.json
  • 13:12 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:12 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2134.codfw.wmnet
  • 13:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
  • 13:11 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 0:30:00 on db[2134,2160].codfw.wmnet with reason: reboot
  • 13:00 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 12:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.upgrade (exit_code=0) for db2132.codfw.wmnet
  • 12:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2177 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60401 and previous config saved to /var/cache/conftool/dbconfig/20240411-125821-root.json
  • 12:57 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 2%: Post upgrade', diff saved to https://phabricator.wikimedia.org/P60400 and previous config saved to /var/cache/conftool/dbconfig/20240411-125755-arnaudb.json
  • 12:54 akosiaris: lower weight of mw1437 back to 10 from the 30 I had upped it to yesterday. The backlog of videoscaling is apparently now served and CPU usage has reached "normal" levels
  • 12:54 arnaudb@cumin1002: START - Cookbook sre.mysql.upgrade for db2132.codfw.wmnet
  • 12:54 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:53 akosiaris@cumin1002: conftool action : set/weight=10; selector: name=mw1437.*.wmnet,dc=eqiad
  • 12:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
  • 12:53 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db[2132,2160].codfw.wmnet with reason: reboot
  • 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:52 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 12:51 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 12:50 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:49 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 12:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:24 btullis@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
  • 12:24 btullis@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
  • 12:23 btullis@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
  • 12:22 btullis@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
  • 12:21 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
  • 12:21 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:20 btullis@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
  • 12:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:20 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
  • 12:20 btullis@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
  • 12:19 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
  • 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:19 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:18 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:16 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 12:16 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host testvm2008.wikimedia.org
  • 12:16 ayounsi@cumin1002: END (FAIL) - Cookbook sre.dns.netbox (exit_code=97)
  • 12:16 ayounsi@cumin1002: END (ERROR) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=97) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:16 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 12:15 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 12:15 moritzm: installing gnutls28 security updates
  • 12:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 12:13 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:13 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 12:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts testvm2008.wikimedia.org
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:13 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 12:13 btullis@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host matomo1003.eqiad.wmnet with OS bullseye
  • 12:12 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2008.wikimedia.org decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 12:10 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 12:06 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2008.wikimedia.org
  • 12:06 ayounsi@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host testvm2008.wikimedia.org
  • 12:06 ayounsi@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host testvm2008.wikimedia.org with OS bookworm
  • 12:02 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-ext: apply
  • 12:01 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-ext: apply
  • 12:01 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-ext: apply
  • 11:59 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-ext: apply
  • 11:59 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 11:58 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 11:58 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2008.wikimedia.org with OS bookworm
  • 11:57 cgoubert@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-web: apply
  • 11:57 cgoubert@deploy1002: helmfile [codfw] START helmfile.d/services/mw-web: apply
  • 11:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2177.codfw.wmnet with OS bookworm
  • 11:50 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:50 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2008.wikimedia.org on all recursors
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2008.wikimedia.org on all recursors
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:47 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2008.wikimedia.org - ayounsi@cumin1002"
  • 11:47 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 11:45 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 11:45 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2008.wikimedia.org
  • 11:33 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bullseye
  • 11:31 moritzm: installing postgresql-15 security updates
  • 11:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
  • 11:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2177.codfw.wmnet with reason: host reimage
  • 11:24 effie: upload prometheus-memcached-exporter 0.14.2-1~wmf1 to bookworm-wikimedia main - T350807
  • 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main - T362160
  • 11:22 effie: upload memkeys 20181031-2-s1 to bookworm-wikimedia main
  • 11:10 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2177.codfw.wmnet with OS bookworm
  • 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2177', diff saved to https://phabricator.wikimedia.org/P60394 and previous config saved to /var/cache/conftool/dbconfig/20240411-110938-root.json
  • 10:53 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw2412.codfw.wmnet|mw2413.codfw.wmnet|mw2414.codfw.wmnet|mw2415.codfw.wmnet|mw2416.codfw.wmnet|mw2417.codfw.wmnet|mw2418.codfw.wmnet),cluster=kubernetes,service=kubesvc
  • 10:52 claime: Pooling and uncordoning mw2412.codfw.wmnet,mw2413.codfw.wmnet,mw2414.codfw.wmnet,mw2415.codfw.wmnet,mw2416.codfw.wmnet,mw2417.codfw.wmnet,mw2418.codfw.wmnet - T351074
  • 10:43 moritzm: installing modsecurity-apache security updates
  • 10:37 claime: Running homer 'cr*codfw*' commit 'T351074'
  • 10:36 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2413.codfw.wmnet with OS bullseye
  • 10:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2418.codfw.wmnet with OS bullseye
  • 10:30 moritzm: installing xerces-c security updates
  • 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60393 and previous config saved to /var/cache/conftool/dbconfig/20240411-103005-root.json
  • 10:28 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2412.codfw.wmnet with OS bullseye
  • 10:25 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2415.codfw.wmnet with OS bullseye
  • 10:22 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2416.codfw.wmnet with OS bullseye
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T356166)', diff saved to https://phabricator.wikimedia.org/P60392 and previous config saved to /var/cache/conftool/dbconfig/20240411-102153-marostegui.json
  • 10:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 10:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 10:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 100%: post schema update', diff saved to https://phabricator.wikimedia.org/P60391 and previous config saved to /var/cache/conftool/dbconfig/20240411-102031-arnaudb.json
  • 10:19 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2417.codfw.wmnet with OS bullseye
  • 10:17 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
  • 10:15 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw2414.codfw.wmnet with OS bullseye
  • 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60390 and previous config saved to /var/cache/conftool/dbconfig/20240411-101500-root.json
  • 10:13 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
  • 10:09 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
  • 10:06 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
  • 10:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 75%: post schema update', diff saved to https://phabricator.wikimedia.org/P60389 and previous config saved to /var/cache/conftool/dbconfig/20240411-100525-arnaudb.json
  • 10:03 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
  • 10:00 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
  • 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60388 and previous config saved to /var/cache/conftool/dbconfig/20240411-095954-root.json
  • 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2418.codfw.wmnet with reason: host reimage
  • 09:57 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
  • 09:57 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
  • 09:57 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2417.codfw.wmnet with reason: host reimage
  • 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2416.codfw.wmnet with reason: host reimage
  • 09:56 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "testvm2007 - ayounsi@cumin1002"
  • 09:56 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2415.codfw.wmnet with reason: host reimage
  • 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2413.codfw.wmnet with reason: host reimage
  • 09:55 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2412.codfw.wmnet with reason: host reimage
  • 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2007.codfw.wmnet
  • 09:55 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2007.codfw.wmnet with OS bookworm
  • 09:54 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw2414.codfw.wmnet with reason: host reimage
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 50%: post schema update', diff saved to https://phabricator.wikimedia.org/P60387 and previous config saved to /var/cache/conftool/dbconfig/20240411-095019-arnaudb.json
  • 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60386 and previous config saved to /var/cache/conftool/dbconfig/20240411-094448-root.json
  • 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2418.codfw.wmnet with OS bullseye
  • 09:40 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
  • 09:40 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2417.codfw.wmnet with OS bullseye
  • 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2416.codfw.wmnet with OS bullseye
  • 09:39 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2415.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2414.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2413.codfw.wmnet with OS bullseye
  • 09:38 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw2412.codfw.wmnet with OS bullseye
  • 09:38 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2007.codfw.wmnet with reason: host reimage
  • 09:37 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3072.esams.wmnet
  • 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 (re)pooling @ 25%: post schema update', diff saved to https://phabricator.wikimedia.org/P60384 and previous config saved to /var/cache/conftool/dbconfig/20240411-093513-arnaudb.json
  • 09:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3072.esams.wmnet with OS bullseye
  • 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2129.codfw.wmnet with reason: Maintenance
  • 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60383 and previous config saved to /var/cache/conftool/dbconfig/20240411-092942-root.json
  • 09:27 arnaudb@cumin1002: dbctl restore of MediaWiki config (dc=all) from /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
  • 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P60382 and previous config saved to /var/cache/conftool/dbconfig/20240411-092622-arnaudb.json
  • 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 09:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 09:25 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2007.codfw.wmnet with OS bookworm
  • 09:25 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 depool', diff saved to https://phabricator.wikimedia.org/P60381 and previous config saved to /var/cache/conftool/dbconfig/20240411-092501-arnaudb.json
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2007.codfw.wmnet on all recursors
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2007.codfw.wmnet on all recursors
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:24 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2129 weight bump T362302', diff saved to https://phabricator.wikimedia.org/P60380 and previous config saved to /var/cache/conftool/dbconfig/20240411-092318-arnaudb.json
  • 09:20 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2007.codfw.wmnet - ayounsi@cumin1002"
  • 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2114 to s6 primary T362302', diff saved to https://phabricator.wikimedia.org/P60379 and previous config saved to /var/cache/conftool/dbconfig/20240411-092012-arnaudb.json
  • 09:19 arnaudb: Starting s6 codfw failover from db2129 to db2114 - T362302
  • 09:16 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 09:16 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2007.codfw.wmnet
  • 09:13 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60378 and previous config saved to /var/cache/conftool/dbconfig/20240411-091255-root.json
  • 09:12 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 09:06 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 09:03 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2114 with weight 0 T362302', diff saved to https://phabricator.wikimedia.org/P60377 and previous config saved to /var/cache/conftool/dbconfig/20240411-085926-arnaudb.json
  • 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
  • 08:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s6 T362302
  • 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.ganeti.makevm (exit_code=0) for new host testvm2006.codfw.wmnet
  • 08:58 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host testvm2006.codfw.wmnet with OS bookworm
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1198 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60376 and previous config saved to /var/cache/conftool/dbconfig/20240411-085749-root.json
  • 08:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1198.eqiad.wmnet with OS bookworm
  • 08:50 jelto@cumin1002: END (FAIL) - Cookbook sre.gitlab.upgrade (exit_code=99) on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
  • 08:45 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1004.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
  • 08:45 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on matomo1003.eqiad.wmnet with reason: Adding disk
  • 08:42 jelto@cumin1002: END (PASS) - Cookbook sre.gitlab.upgrade (exit_code=0) on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:42 ayounsi@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on testvm2006.codfw.wmnet with reason: host reimage
  • 08:40 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
  • 08:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:36 jelto@cumin1002: START - Cookbook sre.gitlab.upgrade on GitLab host gitlab1003.wikimedia.org with reason: Upgrade GitLab Replica to new version
  • 08:36 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
  • 08:31 ayounsi@cumin1002: START - Cookbook sre.hosts.reimage for host testvm2006.codfw.wmnet with OS bookworm
  • 08:29 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:29 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) testvm2006.codfw.wmnet on all recursors
  • 08:28 ayounsi@cumin1002: START - Cookbook sre.dns.wipe-cache testvm2006.codfw.wmnet on all recursors
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:28 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:27 marostegui@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=93) for host db1198.eqiad.wmnet with OS bookworm
  • 08:27 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM testvm2006.codfw.wmnet - ayounsi@cumin1002"
  • 08:26 fabfur@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host cp3072.esams.wmnet with OS bullseye
  • 08:25 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 08:25 ayounsi@cumin1002: START - Cookbook sre.ganeti.makevm for new host testvm2006.codfw.wmnet
  • 08:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:20 hashar: MediaWiki train is blocked
  • 08:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1198.eqiad.wmnet with reason: host reimage
  • 08:13 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:10 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3072.esams.wmnet with reason: host reimage
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp2002.wikimedia.org
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:06 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 08:06 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1198.eqiad.wmnet with OS bookworm
  • 08:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1198', diff saved to https://phabricator.wikimedia.org/P60374 and previous config saved to /var/cache/conftool/dbconfig/20240411-080502-root.json
  • 08:03 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp2002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 08:01 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 07:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp2002.wikimedia.org
  • 07:47 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3072.esams.wmnet with OS bullseye
  • 07:44 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3072.esams.wmnet
  • 07:39 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 07:39 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 07:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60373 and previous config saved to /var/cache/conftool/dbconfig/20240411-072503-root.json
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp1002.wikimedia.org
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 07:10 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60372 and previous config saved to /var/cache/conftool/dbconfig/20240411-070958-root.json
  • 07:08 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp1002.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 07:05 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 07:00 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
  • 06:57 slyngshede@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=99) for hosts idp1002.wikimedia.org
  • 06:56 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp1002.wikimedia.org
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60371 and previous config saved to /var/cache/conftool/dbconfig/20240411-065452-root.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60370 and previous config saved to /var/cache/conftool/dbconfig/20240411-063946-root.json
  • 06:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60369 and previous config saved to /var/cache/conftool/dbconfig/20240411-062728-arnaudb.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60368 and previous config saved to /var/cache/conftool/dbconfig/20240411-062440-root.json
  • 06:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60367 and previous config saved to /var/cache/conftool/dbconfig/20240411-061220-arnaudb.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60366 and previous config saved to /var/cache/conftool/dbconfig/20240411-060934-root.json
  • 05:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216', diff saved to https://phabricator.wikimedia.org/P60365 and previous config saved to /var/cache/conftool/dbconfig/20240411-055712-arnaudb.json
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1189 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60364 and previous config saved to /var/cache/conftool/dbconfig/20240411-055428-root.json
  • 05:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1189.eqiad.wmnet with OS bookworm
  • 05:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60363 and previous config saved to /var/cache/conftool/dbconfig/20240411-054205-arnaudb.json
  • 05:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2216 (T360332)', diff saved to https://phabricator.wikimedia.org/P60362 and previous config saved to /var/cache/conftool/dbconfig/20240411-053903-arnaudb.json
  • 05:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 05:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2216.codfw.wmnet with reason: Maintenance
  • 05:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60361 and previous config saved to /var/cache/conftool/dbconfig/20240411-053840-arnaudb.json
  • 05:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
  • 05:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1189.eqiad.wmnet with reason: host reimage
  • 05:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60360 and previous config saved to /var/cache/conftool/dbconfig/20240411-052333-arnaudb.json
  • 05:15 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1189.eqiad.wmnet with OS bookworm
  • 05:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1189', diff saved to https://phabricator.wikimedia.org/P60359 and previous config saved to /var/cache/conftool/dbconfig/20240411-051341-root.json
  • 05:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212', diff saved to https://phabricator.wikimedia.org/P60358 and previous config saved to /var/cache/conftool/dbconfig/20240411-050825-arnaudb.json
  • 04:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60357 and previous config saved to /var/cache/conftool/dbconfig/20240411-045317-arnaudb.json
  • 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60356 and previous config saved to /var/cache/conftool/dbconfig/20240411-045024-arnaudb.json
  • 04:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 04:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2212.codfw.wmnet with reason: Maintenance
  • 04:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60355 and previous config saved to /var/cache/conftool/dbconfig/20240411-045011-arnaudb.json
  • 04:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60354 and previous config saved to /var/cache/conftool/dbconfig/20240411-043502-arnaudb.json
  • 04:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188', diff saved to https://phabricator.wikimedia.org/P60353 and previous config saved to /var/cache/conftool/dbconfig/20240411-041954-arnaudb.json
  • 04:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60352 and previous config saved to /var/cache/conftool/dbconfig/20240411-040447-arnaudb.json
  • 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60351 and previous config saved to /var/cache/conftool/dbconfig/20240411-040147-arnaudb.json
  • 04:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 04:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2188.codfw.wmnet with reason: Maintenance
  • 04:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60350 and previous config saved to /var/cache/conftool/dbconfig/20240411-040124-arnaudb.json
  • 03:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60349 and previous config saved to /var/cache/conftool/dbconfig/20240411-034617-arnaudb.json
  • 03:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176', diff saved to https://phabricator.wikimedia.org/P60348 and previous config saved to /var/cache/conftool/dbconfig/20240411-033109-arnaudb.json
  • 03:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60347 and previous config saved to /var/cache/conftool/dbconfig/20240411-031602-arnaudb.json
  • 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2176 (T360332)', diff saved to https://phabricator.wikimedia.org/P60346 and previous config saved to /var/cache/conftool/dbconfig/20240411-031310-arnaudb.json
  • 03:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 03:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2176.codfw.wmnet with reason: Maintenance
  • 03:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60345 and previous config saved to /var/cache/conftool/dbconfig/20240411-031247-arnaudb.json
  • 02:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60344 and previous config saved to /var/cache/conftool/dbconfig/20240411-025740-arnaudb.json
  • 02:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174', diff saved to https://phabricator.wikimedia.org/P60343 and previous config saved to /var/cache/conftool/dbconfig/20240411-024232-arnaudb.json
  • 02:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 02:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 02:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60342 and previous config saved to /var/cache/conftool/dbconfig/20240411-023125-marostegui.json
  • 02:30 cstone: civicrm upgraded from a382a7b0 to c2569254
  • 02:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60341 and previous config saved to /var/cache/conftool/dbconfig/20240411-022725-arnaudb.json
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2174 (T360332)', diff saved to https://phabricator.wikimedia.org/P60340 and previous config saved to /var/cache/conftool/dbconfig/20240411-022433-arnaudb.json
  • 02:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 02:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2174.codfw.wmnet with reason: Maintenance
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60339 and previous config saved to /var/cache/conftool/dbconfig/20240411-022410-arnaudb.json
  • 02:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60338 and previous config saved to /var/cache/conftool/dbconfig/20240411-021617-marostegui.json
  • 02:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60337 and previous config saved to /var/cache/conftool/dbconfig/20240411-020903-arnaudb.json
  • 02:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P60336 and previous config saved to /var/cache/conftool/dbconfig/20240411-020110-marostegui.json
  • 01:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173', diff saved to https://phabricator.wikimedia.org/P60335 and previous config saved to /var/cache/conftool/dbconfig/20240411-015355-arnaudb.json
  • 01:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60334 and previous config saved to /var/cache/conftool/dbconfig/20240411-014602-marostegui.json
  • 01:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60333 and previous config saved to /var/cache/conftool/dbconfig/20240411-013848-arnaudb.json
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2173 (T360332)', diff saved to https://phabricator.wikimedia.org/P60332 and previous config saved to /var/cache/conftool/dbconfig/20240411-013657-arnaudb.json
  • 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2173.codfw.wmnet with reason: Maintenance
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60331 and previous config saved to /var/cache/conftool/dbconfig/20240411-013618-arnaudb.json
  • 01:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60330 and previous config saved to /var/cache/conftool/dbconfig/20240411-012110-arnaudb.json
  • 01:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170', diff saved to https://phabricator.wikimedia.org/P60329 and previous config saved to /var/cache/conftool/dbconfig/20240411-010601-arnaudb.json
  • 00:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60328 and previous config saved to /var/cache/conftool/dbconfig/20240411-005054-arnaudb.json
  • 00:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2170 (T360332)', diff saved to https://phabricator.wikimedia.org/P60327 and previous config saved to /var/cache/conftool/dbconfig/20240411-004758-arnaudb.json
  • 00:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 00:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2170.codfw.wmnet with reason: Maintenance
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60326 and previous config saved to /var/cache/conftool/dbconfig/20240411-004735-arnaudb.json
  • 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T356166)', diff saved to https://phabricator.wikimedia.org/P60325 and previous config saved to /var/cache/conftool/dbconfig/20240411-004536-marostegui.json
  • 00:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 00:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 00:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60324 and previous config saved to /var/cache/conftool/dbconfig/20240411-004514-marostegui.json
  • 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60323 and previous config saved to /var/cache/conftool/dbconfig/20240411-003226-arnaudb.json
  • 00:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60322 and previous config saved to /var/cache/conftool/dbconfig/20240411-003005-marostegui.json
  • 00:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153', diff saved to https://phabricator.wikimedia.org/P60321 and previous config saved to /var/cache/conftool/dbconfig/20240411-001718-arnaudb.json
  • 00:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P60320 and previous config saved to /var/cache/conftool/dbconfig/20240411-001458-marostegui.json
  • 00:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60319 and previous config saved to /var/cache/conftool/dbconfig/20240411-000211-arnaudb.json

2024-04-10

  • 23:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60318 and previous config saved to /var/cache/conftool/dbconfig/20240410-235950-marostegui.json
  • 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2153 (T360332)', diff saved to https://phabricator.wikimedia.org/P60317 and previous config saved to /var/cache/conftool/dbconfig/20240410-235920-arnaudb.json
  • 23:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 23:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2153.codfw.wmnet with reason: Maintenance
  • 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60316 and previous config saved to /var/cache/conftool/dbconfig/20240410-235857-arnaudb.json
  • 23:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60315 and previous config saved to /var/cache/conftool/dbconfig/20240410-234350-arnaudb.json
  • 23:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146', diff saved to https://phabricator.wikimedia.org/P60314 and previous config saved to /var/cache/conftool/dbconfig/20240410-232842-arnaudb.json
  • 23:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60313 and previous config saved to /var/cache/conftool/dbconfig/20240410-231335-arnaudb.json
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2146 (T360332)', diff saved to https://phabricator.wikimedia.org/P60312 and previous config saved to /var/cache/conftool/dbconfig/20240410-231032-arnaudb.json
  • 23:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 23:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2146.codfw.wmnet with reason: Maintenance
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60311 and previous config saved to /var/cache/conftool/dbconfig/20240410-231008-arnaudb.json
  • 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60310 and previous config saved to /var/cache/conftool/dbconfig/20240410-225500-arnaudb.json
  • 22:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145', diff saved to https://phabricator.wikimedia.org/P60309 and previous config saved to /var/cache/conftool/dbconfig/20240410-223953-arnaudb.json
  • 22:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60308 and previous config saved to /var/cache/conftool/dbconfig/20240410-222445-arnaudb.json
  • 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2145 (T360332)', diff saved to https://phabricator.wikimedia.org/P60307 and previous config saved to /var/cache/conftool/dbconfig/20240410-222150-arnaudb.json
  • 22:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 22:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2145.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2141.codfw.wmnet with reason: Maintenance
  • 22:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60306 and previous config saved to /var/cache/conftool/dbconfig/20240410-222028-arnaudb.json
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60305 and previous config saved to /var/cache/conftool/dbconfig/20240410-220521-arnaudb.json
  • 21:56 mutante: prometheus - recreating deleted TLS certs/keys in private repo
  • 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130', diff saved to https://phabricator.wikimedia.org/P60304 and previous config saved to /var/cache/conftool/dbconfig/20240410-215014-arnaudb.json
  • 21:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60303 and previous config saved to /var/cache/conftool/dbconfig/20240410-213506-arnaudb.json
  • 21:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2130 (T360332)', diff saved to https://phabricator.wikimedia.org/P60302 and previous config saved to /var/cache/conftool/dbconfig/20240410-213203-arnaudb.json
  • 21:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2130.codfw.wmnet with reason: Maintenance
  • 21:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60301 and previous config saved to /var/cache/conftool/dbconfig/20240410-213140-arnaudb.json
  • 21:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60300 and previous config saved to /var/cache/conftool/dbconfig/20240410-211632-arnaudb.json
  • 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116', diff saved to https://phabricator.wikimedia.org/P60298 and previous config saved to /var/cache/conftool/dbconfig/20240410-210125-arnaudb.json
  • 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60297 and previous config saved to /var/cache/conftool/dbconfig/20240410-204617-arnaudb.json
  • 20:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2116 (T360332)', diff saved to https://phabricator.wikimedia.org/P60296 and previous config saved to /var/cache/conftool/dbconfig/20240410-204316-arnaudb.json
  • 20:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 20:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2116.codfw.wmnet with reason: Maintenance
  • 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60295 and previous config saved to /var/cache/conftool/dbconfig/20240410-204253-arnaudb.json
  • 20:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60294 and previous config saved to /var/cache/conftool/dbconfig/20240410-202745-arnaudb.json
  • 20:17 cjming: end of UTC late backport window
  • 20:15 cjming@deploy1002: Finished scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) (duration: 13m 51s)
  • 20:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112', diff saved to https://phabricator.wikimedia.org/P60293 and previous config saved to /var/cache/conftool/dbconfig/20240410-201237-arnaudb.json
  • 20:04 cjming@deploy1002: cjming and daniel: Continuing with sync
  • 20:04 cjming@deploy1002: cjming and daniel: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:01 cjming@deploy1002: Started scap: Backport for LogStash: log HtmlOutputRendererHelper channel (T356157)
  • 19:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60292 and previous config saved to /var/cache/conftool/dbconfig/20240410-195730-arnaudb.json
  • 19:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2112 (T360332)', diff saved to https://phabricator.wikimedia.org/P60291 and previous config saved to /var/cache/conftool/dbconfig/20240410-195430-arnaudb.json
  • 19:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 19:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2112.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2103.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2102.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 19:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60290 and previous config saved to /var/cache/conftool/dbconfig/20240410-194909-arnaudb.json
  • 19:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60289 and previous config saved to /var/cache/conftool/dbconfig/20240410-193402-arnaudb.json
  • 19:24 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
  • 19:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3071.esams.wmnet with OS bullseye
  • 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P60288 and previous config saved to /var/cache/conftool/dbconfig/20240410-191854-arnaudb.json
  • 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60287 and previous config saved to /var/cache/conftool/dbconfig/20240410-190347-arnaudb.json
  • 18:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
  • 18:51 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3071.esams.wmnet with reason: host reimage
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1235 (T360332)', diff saved to https://phabricator.wikimedia.org/P60285 and previous config saved to /var/cache/conftool/dbconfig/20240410-184656-arnaudb.json
  • 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1235.eqiad.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60284 and previous config saved to /var/cache/conftool/dbconfig/20240410-184633-arnaudb.json
  • 18:34 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/echostore: apply
  • 18:34 eevans@deploy1002: helmfile [staging] START helmfile.d/services/echostore: apply
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60283 and previous config saved to /var/cache/conftool/dbconfig/20240410-183126-arnaudb.json
  • 18:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 18:28 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3071.esams.wmnet with OS bullseye
  • 18:26 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1115.eqiad.wmnet with OS bullseye
  • 18:24 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp3071.esams.wmnet,service=(cdn|ats-be)
  • 18:17 eevans@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 eevans@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P60282 and previous config saved to /var/cache/conftool/dbconfig/20240410-181618-arnaudb.json
  • 18:15 eevans@deploy1002: helmfile [staging] START helmfile.d/services/changeprop-jobqueue: apply
  • 18:08 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
  • 18:05 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1115.eqiad.wmnet with reason: host reimage
  • 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60281 and previous config saved to /var/cache/conftool/dbconfig/20240410-180111-arnaudb.json
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T360332)', diff saved to https://phabricator.wikimedia.org/P60280 and previous config saved to /var/cache/conftool/dbconfig/20240410-175816-arnaudb.json
  • 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60279 and previous config saved to /var/cache/conftool/dbconfig/20240410-175752-arnaudb.json
  • 17:48 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
  • 17:48 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1115.eqiad.wmnet with OS bullseye
  • 17:46 swfrench-wmf: finished updating A:conf hosts to etcd-mirror 0.0.11-1 (T358636)
  • 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60278 and previous config saved to /var/cache/conftool/dbconfig/20240410-174244-arnaudb.json
  • 17:37 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1115.eqiad.wmnet with OS bullseye
  • 17:37 swfrench-wmf: restarting etcd-mirror on conf2005.codfw.wmnet for T358636
  • 17:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:34 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P60277 and previous config saved to /var/cache/conftool/dbconfig/20240410-172736-arnaudb.json
  • 17:21 hashar@deploy1002: Finished scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) (duration: 18m 53s)
  • 17:14 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60276 and previous config saved to /var/cache/conftool/dbconfig/20240410-171229-arnaudb.json
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T360332)', diff saved to https://phabricator.wikimedia.org/P60275 and previous config saved to /var/cache/conftool/dbconfig/20240410-170930-arnaudb.json
  • 17:09 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60274 and previous config saved to /var/cache/conftool/dbconfig/20240410-170907-arnaudb.json
  • 17:07 hashar@deploy1002: hashar: Continuing with sync
  • 17:07 hashar@deploy1002: hashar: Backport for TitleLibrary: Don't register external titles as dependencies (T362222) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 17:06 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:06 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:05 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1115.eqiad.wmnet
  • 17:05 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1115.eqiad.wmnet,service=(cdn|ats-be)
  • 17:04 sukhe: depool cp1115 for firmware downgrade for PXE boot testing: T350179
  • 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:03 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:02 hnowlan: killing long-running videoscaler ffmpegs
  • 17:02 hashar@deploy1002: Started scap: Backport for TitleLibrary: Don't register external titles as dependencies (T362222)
  • 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60272 and previous config saved to /var/cache/conftool/dbconfig/20240410-165359-arnaudb.json
  • 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:50 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P60270 and previous config saved to /var/cache/conftool/dbconfig/20240410-163851-arnaudb.json
  • 16:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60269 and previous config saved to /var/cache/conftool/dbconfig/20240410-162344-arnaudb.json
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T360332)', diff saved to https://phabricator.wikimedia.org/P60268 and previous config saved to /var/cache/conftool/dbconfig/20240410-162101-arnaudb.json
  • 16:20 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 16:20 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60267 and previous config saved to /var/cache/conftool/dbconfig/20240410-162039-arnaudb.json
  • 16:19 elukey@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: sync
  • 16:19 elukey@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: sync
  • 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] DONE helmfile.d/services/termbox: apply
  • 16:16 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [eqiad] START helmfile.d/services/termbox: apply
  • 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] DONE helmfile.d/services/termbox: apply
  • 16:15 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [codfw] START helmfile.d/services/termbox: apply
  • 16:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] DONE helmfile.d/services/termbox: apply
  • 16:13 logmsgbot: lucaswerkmeister-wmde@deploy1002 helmfile [staging] START helmfile.d/services/termbox: apply
  • 16:12 swfrench-wmf: uploaded etcd-mirror 0.0.11-1 to apt.wikimedia.org (T358636)
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60265 and previous config saved to /var/cache/conftool/dbconfig/20240410-160531-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P60264 and previous config saved to /var/cache/conftool/dbconfig/20240410-155024-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60262 and previous config saved to /var/cache/conftool/dbconfig/20240410-153516-arnaudb.json
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T360332)', diff saved to https://phabricator.wikimedia.org/P60261 and previous config saved to /var/cache/conftool/dbconfig/20240410-153229-arnaudb.json
  • 15:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 15:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60260 and previous config saved to /var/cache/conftool/dbconfig/20240410-153207-arnaudb.json
  • 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60259 and previous config saved to /var/cache/conftool/dbconfig/20240410-151659-arnaudb.json
  • 15:14 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:14 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:14 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 15:13 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T356166)', diff saved to https://phabricator.wikimedia.org/P60258 and previous config saved to /var/cache/conftool/dbconfig/20240410-150327-marostegui.json
  • 15:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 15:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 15:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60257 and previous config saved to /var/cache/conftool/dbconfig/20240410-150304-marostegui.json
  • 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P60256 and previous config saved to /var/cache/conftool/dbconfig/20240410-150152-arnaudb.json
  • 14:58 moritzm: installing debian-archive-keyring updates on buster
  • 14:55 akosiaris: kill all ffmpegs on mw1437 and increase weight of mw1347 from 10 to 30 to direct most queries to it while the other 3 videoscalers serve the backlog
  • 14:54 akosiaris@cumin1002: conftool action : set/weight=30; selector: name=mw1437.*.wmnet,dc=eqiad
  • 14:51 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:51 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:50 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop-jobqueue: apply
  • 14:50 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop-jobqueue: apply
  • 14:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60255 and previous config saved to /var/cache/conftool/dbconfig/20240410-144757-marostegui.json
  • 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60254 and previous config saved to /var/cache/conftool/dbconfig/20240410-144644-arnaudb.json
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T360332)', diff saved to https://phabricator.wikimedia.org/P60253 and previous config saved to /var/cache/conftool/dbconfig/20240410-144400-arnaudb.json
  • 14:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 14:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60252 and previous config saved to /var/cache/conftool/dbconfig/20240410-144336-arnaudb.json
  • 14:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P60251 and previous config saved to /var/cache/conftool/dbconfig/20240410-143249-marostegui.json
  • 14:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60250 and previous config saved to /var/cache/conftool/dbconfig/20240410-142829-arnaudb.json
  • 14:21 sukhe@cumin1002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60249 and previous config saved to /var/cache/conftool/dbconfig/20240410-141742-marostegui.json
  • 14:17 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P60248 and previous config saved to /var/cache/conftool/dbconfig/20240410-141322-arnaudb.json
  • 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1112.eqiad.wmnet with OS bullseye
  • 13:58 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60246 and previous config saved to /var/cache/conftool/dbconfig/20240410-135814-arnaudb.json
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60245 and previous config saved to /var/cache/conftool/dbconfig/20240410-135525-arnaudb.json
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60244 and previous config saved to /var/cache/conftool/dbconfig/20240410-135502-arnaudb.json
  • 13:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:49 denisse: Delete unused Prometheus TLS certificates - T360414
  • 13:47 moritzm: installing unbound security updates
  • 13:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
  • 13:43 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1112.eqiad.wmnet with reason: host reimage
  • 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60243 and previous config saved to /var/cache/conftool/dbconfig/20240410-133955-arnaudb.json
  • 13:39 eevans@deploy1002: helmfile [staging] DONE helmfile.d/services/sessionstore: apply
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 13:30 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
  • 13:30 bking@cumin2002: START - Cookbook sre.hosts.downtime for 30 days, 0:00:00 on elastic2088.codfw.wmnet with reason: T361525
  • 13:30 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 13:28 eevans@deploy1002: helmfile [staging] START helmfile.d/services/sessionstore: apply
  • 13:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
  • 13:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1112.eqiad.wmnet with OS bullseye
  • 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P60242 and previous config saved to /var/cache/conftool/dbconfig/20240410-132447-arnaudb.json
  • 13:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T356166)', diff saved to https://phabricator.wikimedia.org/P60241 and previous config saved to /var/cache/conftool/dbconfig/20240410-131716-marostegui.json
  • 13:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1112.eqiad.wmnet with OS bullseye
  • 13:16 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60240 and previous config saved to /var/cache/conftool/dbconfig/20240410-131653-marostegui.json
  • 13:16 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1112.eqiad.wmnet,service=(cdn|ats-be)
  • 13:09 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:09 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
  • 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60239 and previous config saved to /var/cache/conftool/dbconfig/20240410-130940-arnaudb.json
  • 13:09 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test restoring dns entry - volans@cumin2002"
  • 13:07 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:07 volans@cumin2002: START - Cookbook sre.dns.netbox
  • 13:07 sukhe: depool cp4052 for PXE boot issue testing
  • 13:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T360332)', diff saved to https://phabricator.wikimedia.org/P60238 and previous config saved to /var/cache/conftool/dbconfig/20240410-130650-arnaudb.json
  • 13:07 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 13:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 13:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60237 and previous config saved to /var/cache/conftool/dbconfig/20240410-130626-arnaudb.json
  • 13:05 volans@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 13:05 volans@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
  • 13:05 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=dns6001.wikimedia.org,service=authdns-update
  • 13:04 volans@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: test removing dns entry - volans@cumin2002"
  • 13:02 volans@cumin2002: START - Cookbook sre.dns.netbox
  • 13:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60236 and previous config saved to /var/cache/conftool/dbconfig/20240410-130145-marostegui.json
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts idp-test2003.wikimedia.org
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:59 slyngshede@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 12:56 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=dns6001.wikimedia.org,service=authdns-update
  • 12:56 slyngshede@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: idp-test2003.wikimedia.org decommissioned, removing all IPs except the asset tag one - slyngshede@cumin1002"
  • 12:53 slyngshede@cumin1002: START - Cookbook sre.dns.netbox
  • 12:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P60235 and previous config saved to /var/cache/conftool/dbconfig/20240410-125119-arnaudb.json
  • 12:48 slyngshede@cumin1002: START - Cookbook sre.hosts.decommission for hosts idp-test2003.wikimedia.org
  • 12:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P60234 and previous config saved to /var/cache/conftool/dbconfig/20240410-124638-marostegui.json
  • 12:45 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
  • 12:44 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "update for latest VMs - jmm@cumin2002"
  • 12:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60231 and previous config saved to /var/cache/conftool/dbconfig/20240410-122518-root.json
  • 12:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 12:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60230 and previous config saved to /var/cache/conftool/dbconfig/20240410-122104-arnaudb.json
  • 12:20 slyngshede@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
  • 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T360332)', diff saved to https://phabricator.wikimedia.org/P60229 and previous config saved to /var/cache/conftool/dbconfig/20240410-121814-arnaudb.json
  • 12:18 slyngshede@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on idp-test1002.wikimedia.org with reason: host reimage
  • 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 12:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60228 and previous config saved to /var/cache/conftool/dbconfig/20240410-121743-arnaudb.json
  • 12:15 Lucas_WMDE: lucaswerkmeister-wmde@mwmaint1002:~$ mwscript extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # succeeded
  • 12:14 Lucas_WMDE: lucaswerkmeister-wmde@deploy1002 ~ $ mwscript-k8s extensions/Wikibase/repo/maintenance/changePropertyDataType.php wikidatawiki --property-id P4496 --new-data-type external-id --summary 'T359297' # failed, will retry with non-k8s mwscript
  • 12:12 cgoubert@cumin1002: conftool action : set/weight=10:pooled=yes; selector: name=(mw1421.eqiad.wmnet|mw1422.eqiad.wmnet|mw1491.eqiad.wmnet|mw1492.eqiad.wmnet|mw1493.eqiad.wmnet),cluster=kubernetes,service=kubesvc
  • 12:11 claime: Pooling and uncordoning mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60227 and previous config saved to /var/cache/conftool/dbconfig/20240410-121012-root.json
  • 12:04 slyngshede@cumin1002: START - Cookbook sre.hosts.reimage for host idp-test1002.wikimedia.org with OS bookworm
  • 12:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60226 and previous config saved to /var/cache/conftool/dbconfig/20240410-120235-arnaudb.json
  • 12:01 claime: Running homer 'cr*eqiad*' commit 'T351074' and homer 'lsw1-e3-eqiad*' commit 'T351074'
  • 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60225 and previous config saved to /var/cache/conftool/dbconfig/20240410-115506-root.json
  • 11:54 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1492.eqiad.wmnet with OS bullseye
  • 11:53 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1491.eqiad.wmnet with OS bullseye
  • 11:49 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1422.eqiad.wmnet with OS bullseye
  • 11:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P60224 and previous config saved to /var/cache/conftool/dbconfig/20240410-114728-arnaudb.json
  • 11:45 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1493.eqiad.wmnet with OS bullseye
  • 11:42 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mw1421.eqiad.wmnet with OS bullseye
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60223 and previous config saved to /var/cache/conftool/dbconfig/20240410-114001-root.json
  • 11:38 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
  • 11:34 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
  • 11:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60222 and previous config saved to /var/cache/conftool/dbconfig/20240410-113220-arnaudb.json
  • 11:31 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
  • 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T360332)', diff saved to https://phabricator.wikimedia.org/P60221 and previous config saved to /var/cache/conftool/dbconfig/20240410-112929-arnaudb.json
  • 11:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 11:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 11:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60220 and previous config saved to /var/cache/conftool/dbconfig/20240410-112907-arnaudb.json
  • 11:28 jiji@deploy1002: Finished scap: Deploy chart changes in gerrit:1015342 (duration: 08m 18s)
  • 11:27 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
  • 11:24 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60219 and previous config saved to /var/cache/conftool/dbconfig/20240410-112455-root.json
  • 11:24 cgoubert@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
  • 11:23 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1493.eqiad.wmnet with reason: host reimage
  • 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1492.eqiad.wmnet with reason: host reimage
  • 11:22 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1491.eqiad.wmnet with reason: host reimage
  • 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1422.eqiad.wmnet with reason: host reimage
  • 11:21 cgoubert@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mw1421.eqiad.wmnet with reason: host reimage
  • 11:19 jiji@deploy1002: Started scap: Deploy chart changes in gerrit:1015342
  • 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 11:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60218 and previous config saved to /var/cache/conftool/dbconfig/20240410-111400-arnaudb.json
  • 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 11:12 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 11:12 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 11:10 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1493.eqiad.wmnet with OS bullseye
  • 11:09 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60217 and previous config saved to /var/cache/conftool/dbconfig/20240410-110949-root.json
  • 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1492.eqiad.wmnet with OS bullseye
  • 11:09 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1491.eqiad.wmnet with OS bullseye
  • 11:08 jiji@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-debug: apply
  • 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1422.eqiad.wmnet with OS bullseye
  • 11:08 cgoubert@cumin1002: START - Cookbook sre.hosts.reimage for host mw1421.eqiad.wmnet with OS bullseye
  • 11:07 jiji@deploy1002: helmfile [codfw] START helmfile.d/services/mw-debug: apply
  • 11:07 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:07 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:03 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 11:02 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:59 claime: Depooling mw1421.eqiad.wmnet,mw1422.eqiad.wmnet,mw1491.eqiad.wmnet,mw1492.eqiad.wmnet,mw1493.eqiad.wmnet - T351074
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P60216 and previous config saved to /var/cache/conftool/dbconfig/20240410-105852-arnaudb.json
  • 10:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1175.eqiad.wmnet with OS bookworm
  • 10:54 marostegui@cumin1002: dbctl commit (dc=all): 'db1175 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60215 and previous config saved to /var/cache/conftool/dbconfig/20240410-105444-root.json
  • 10:53 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 10:53 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 10:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60214 and previous config saved to /var/cache/conftool/dbconfig/20240410-104345-arnaudb.json
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T360332)', diff saved to https://phabricator.wikimedia.org/P60213 and previous config saved to /var/cache/conftool/dbconfig/20240410-104053-arnaudb.json
  • 10:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 10:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60212 and previous config saved to /var/cache/conftool/dbconfig/20240410-104030-arnaudb.json
  • 10:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
  • 10:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1175.eqiad.wmnet with reason: host reimage
  • 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60211 and previous config saved to /var/cache/conftool/dbconfig/20240410-102523-arnaudb.json
  • 10:21 claime: Enabling and running puppet on O:docker_registry_ha::registry - T360636
  • 10:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1175.eqiad.wmnet with OS bookworm
  • 10:18 claime: Enabling and running puppet on registry1003.eqiad.wmnet - T360636
  • 10:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1175 T362036', diff saved to https://phabricator.wikimedia.org/P60210 and previous config saved to /var/cache/conftool/dbconfig/20240410-101746-root.json
  • 10:16 claime: Disabling puppet on O:docker_registry_ha::registry - T360636
  • 10:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_master
  • 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P60209 and previous config saved to /var/cache/conftool/dbconfig/20240410-101015-arnaudb.json
  • 10:08 jiji@deploy1002: Finished scap: (no justification provided) (duration: 27m 59s)
  • 09:58 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_master
  • 09:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60208 and previous config saved to /var/cache/conftool/dbconfig/20240410-095508-arnaudb.json
  • 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T360332)', diff saved to https://phabricator.wikimedia.org/P60207 and previous config saved to /var/cache/conftool/dbconfig/20240410-095214-arnaudb.json
  • 09:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 09:42 effie: running scap sync-world to rebuild mw image and pick up gerrit:1015338
  • 09:40 jiji@deploy1002: Started scap: (no justification provided)
  • 08:49 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3070.esams.wmnet
  • 08:42 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3070.esams.wmnet with OS bullseye
  • 08:38 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60206 and previous config saved to /var/cache/conftool/dbconfig/20240410-083822-arnaudb.json
  • 08:35 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 08:34 gmodena@deploy1002: Finished deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648 (duration: 00m 33s)
  • 08:34 hashar@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.26 refs T360158 (duration: 13m 05s)
  • 08:34 gmodena@deploy1002: Started deploy [airflow-dags/analytics@46818a3]: Deploying cassandra_load_pageview_top_articles changes MR#648
  • 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:25 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 08:24 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 08:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60205 and previous config saved to /var/cache/conftool/dbconfig/20240410-082316-arnaudb.json
  • 08:21 hashar@deploy1002: rebuilt and synchronized wikiversions files: group1 wikis to 1.42.0-wmf.26 refs T360158
  • 08:18 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
  • 08:15 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3070.esams.wmnet with reason: host reimage
  • 08:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60204 and previous config saved to /var/cache/conftool/dbconfig/20240410-080810-arnaudb.json
  • 07:56 moritzm: installing glibc security updates on bullseye
  • 07:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60203 and previous config saved to /var/cache/conftool/dbconfig/20240410-075304-arnaudb.json
  • 07:52 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3070.esams.wmnet with OS bullseye
  • 07:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60202 and previous config saved to /var/cache/conftool/dbconfig/20240410-075150-arnaudb.json
  • 07:50 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3070.esams.wmnet
  • 07:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60201 and previous config saved to /var/cache/conftool/dbconfig/20240410-073759-arnaudb.json
  • 07:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60200 and previous config saved to /var/cache/conftool/dbconfig/20240410-073644-arnaudb.json
  • 07:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: dumps::generation::server::spare
  • 07:29 akosiaris@deploy1002: Synchronized wmf-config/mc.php: Dummy sync for https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1018332 (duration: 14m 03s)
  • 07:25 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: dumps::generation::server::spare
  • 07:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60199 and previous config saved to /var/cache/conftool/dbconfig/20240410-072253-arnaudb.json
  • 07:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60198 and previous config saved to /var/cache/conftool/dbconfig/20240410-072137-arnaudb.json
  • 07:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60197 and previous config saved to /var/cache/conftool/dbconfig/20240410-070745-arnaudb.json
  • 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60196 and previous config saved to /var/cache/conftool/dbconfig/20240410-070631-arnaudb.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60195 and previous config saved to /var/cache/conftool/dbconfig/20240410-065929-root.json
  • 06:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60194 and previous config saved to /var/cache/conftool/dbconfig/20240410-065239-arnaudb.json
  • 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 20%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60193 and previous config saved to /var/cache/conftool/dbconfig/20240410-065125-arnaudb.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60192 and previous config saved to /var/cache/conftool/dbconfig/20240410-064423-root.json
  • 06:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2212 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P60191 and previous config saved to /var/cache/conftool/dbconfig/20240410-063734-arnaudb.json
  • 06:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 10%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60190 and previous config saved to /var/cache/conftool/dbconfig/20240410-063620-arnaudb.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60189 and previous config saved to /var/cache/conftool/dbconfig/20240410-062917-root.json
  • 06:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2112 (re)pooling @ 5%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P60188 and previous config saved to /var/cache/conftool/dbconfig/20240410-062114-arnaudb.json
  • 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60187 and previous config saved to /var/cache/conftool/dbconfig/20240410-062003-root.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60186 and previous config saved to /var/cache/conftool/dbconfig/20240410-061411-root.json
  • 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60185 and previous config saved to /var/cache/conftool/dbconfig/20240410-060457-root.json
  • 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60184 and previous config saved to /var/cache/conftool/dbconfig/20240410-055906-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60183 and previous config saved to /var/cache/conftool/dbconfig/20240410-054952-root.json
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60182 and previous config saved to /var/cache/conftool/dbconfig/20240410-054400-root.json
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60181 and previous config saved to /var/cache/conftool/dbconfig/20240410-053445-root.json
  • 05:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1166.eqiad.wmnet with OS bookworm
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1166 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60180 and previous config saved to /var/cache/conftool/dbconfig/20240410-052854-root.json
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60179 and previous config saved to /var/cache/conftool/dbconfig/20240410-051939-root.json
  • 05:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
  • 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1166.eqiad.wmnet with reason: host reimage
  • 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60178 and previous config saved to /var/cache/conftool/dbconfig/20240410-050434-root.json
  • 04:58 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1166.eqiad.wmnet with OS bookworm
  • 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1166 T362134', diff saved to https://phabricator.wikimedia.org/P60177 and previous config saved to /var/cache/conftool/dbconfig/20240410-045710-marostegui.json
  • 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repool db1223', diff saved to https://phabricator.wikimedia.org/P60176 and previous config saved to /var/cache/conftool/dbconfig/20240410-045632-marostegui.json
  • 04:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1223 T362134', diff saved to https://phabricator.wikimedia.org/P60175 and previous config saved to /var/cache/conftool/dbconfig/20240410-045534-marostegui.json
  • 04:49 marostegui@cumin1002: dbctl commit (dc=all): 'db1246 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60174 and previous config saved to /var/cache/conftool/dbconfig/20240410-044928-root.json
  • 04:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 6 hosts with reason: Kernel reboot
  • 04:46 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 6 hosts with reason: Kernel reboot
  • 04:16 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T356166)', diff saved to https://phabricator.wikimedia.org/P60173 and previous config saved to /var/cache/conftool/dbconfig/20240410-041604-marostegui.json
  • 04:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 04:15 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 04:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60172 and previous config saved to /var/cache/conftool/dbconfig/20240410-041541-marostegui.json
  • 04:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60171 and previous config saved to /var/cache/conftool/dbconfig/20240410-040033-marostegui.json
  • 03:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P60170 and previous config saved to /var/cache/conftool/dbconfig/20240410-034526-marostegui.json
  • 03:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60169 and previous config saved to /var/cache/conftool/dbconfig/20240410-033019-marostegui.json

2024-04-09

  • 23:17 eileen: config revision changed from 7908b55e to 974afe9c
  • 23:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60168 and previous config saved to /var/cache/conftool/dbconfig/20240409-230828-arnaudb.json
  • 23:08 eileen: config revision changed from 064d18b0 to 7908b55e
  • 22:58 eileen: config revision changed from 8fb02f33 to 064d18b0
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60167 and previous config saved to /var/cache/conftool/dbconfig/20240409-225321-arnaudb.json
  • 22:51 eileen: config revision changed from df416a50 to 8fb02f33
  • 22:48 eileen: config revision changed from cea14e30 to df416a50
  • 22:42 eileen: config revision changed from 075ddd44 to cea14e30
  • 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209', diff saved to https://phabricator.wikimedia.org/P60166 and previous config saved to /var/cache/conftool/dbconfig/20240409-223813-arnaudb.json
  • 22:24 eileen: config revision changed from 3c1a0267 to 4638a4d2
  • 22:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60165 and previous config saved to /var/cache/conftool/dbconfig/20240409-222306-arnaudb.json
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2209 (T360332)', diff saved to https://phabricator.wikimedia.org/P60164 and previous config saved to /var/cache/conftool/dbconfig/20240409-220755-arnaudb.json
  • 22:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 22:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2209.codfw.wmnet with reason: Maintenance
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60163 and previous config saved to /var/cache/conftool/dbconfig/20240409-220732-arnaudb.json
  • 22:03 eileen: civicrm upgraded from b05fd08f to a382a7b0
  • 21:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60162 and previous config saved to /var/cache/conftool/dbconfig/20240409-215225-arnaudb.json
  • 21:38 eileen: civicrm upgraded from 8c7cc208 to b05fd08f
  • 21:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205', diff saved to https://phabricator.wikimedia.org/P60161 and previous config saved to /var/cache/conftool/dbconfig/20240409-213717-arnaudb.json
  • 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60160 and previous config saved to /var/cache/conftool/dbconfig/20240409-212210-arnaudb.json
  • 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2205 (T360332)', diff saved to https://phabricator.wikimedia.org/P60159 and previous config saved to /var/cache/conftool/dbconfig/20240409-210656-arnaudb.json
  • 21:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 21:06 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2205.codfw.wmnet with reason: Maintenance
  • 21:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60158 and previous config saved to /var/cache/conftool/dbconfig/20240409-210633-arnaudb.json
  • 21:01 cjming: end of UTC late backport window
  • 21:00 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 15m 27s)
  • 20:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60157 and previous config saved to /var/cache/conftool/dbconfig/20240409-205125-arnaudb.json
  • 20:48 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
  • 20:47 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:45 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:45 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
  • 20:44 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:44 cjming@deploy1002: Finished scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) (duration: 16m 16s)
  • 20:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194', diff saved to https://phabricator.wikimedia.org/P60156 and previous config saved to /var/cache/conftool/dbconfig/20240409-203617-arnaudb.json
  • 20:32 cjming@deploy1002: jjmc89 and cjming: Continuing with sync
  • 20:30 cjming@deploy1002: jjmc89 and cjming: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:28 cjming@deploy1002: Started scap: Backport for extension.json: add pagetriage-copyvio right to the highvolume grant (T362188)
  • 20:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60155 and previous config saved to /var/cache/conftool/dbconfig/20240409-202410-arnaudb.json
  • 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60154 and previous config saved to /var/cache/conftool/dbconfig/20240409-202110-arnaudb.json
  • 20:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60153 and previous config saved to /var/cache/conftool/dbconfig/20240409-200903-arnaudb.json
  • 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2194 (T360332)', diff saved to https://phabricator.wikimedia.org/P60152 and previous config saved to /var/cache/conftool/dbconfig/20240409-200556-arnaudb.json
  • 20:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 20:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2194.codfw.wmnet with reason: Maintenance
  • 20:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60151 and previous config saved to /var/cache/conftool/dbconfig/20240409-200533-arnaudb.json
  • 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 20:02 eileen: config revision changed from abccfdc0 to 3c1a0267
  • 20:02 cdanis@deploy2002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207', diff saved to https://phabricator.wikimedia.org/P60150 and previous config saved to /var/cache/conftool/dbconfig/20240409-195355-arnaudb.json
  • 19:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60149 and previous config saved to /var/cache/conftool/dbconfig/20240409-195024-arnaudb.json
  • 19:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60148 and previous config saved to /var/cache/conftool/dbconfig/20240409-193848-arnaudb.json
  • 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2207 (T360332)', diff saved to https://phabricator.wikimedia.org/P60147 and previous config saved to /var/cache/conftool/dbconfig/20240409-193611-arnaudb.json
  • 19:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2207.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190', diff saved to https://phabricator.wikimedia.org/P60146 and previous config saved to /var/cache/conftool/dbconfig/20240409-193517-arnaudb.json
  • 19:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2197.codfw.wmnet with reason: Maintenance
  • 19:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60145 and previous config saved to /var/cache/conftool/dbconfig/20240409-193507-arnaudb.json
  • 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60143 and previous config saved to /var/cache/conftool/dbconfig/20240409-192010-arnaudb.json
  • 19:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60142 and previous config saved to /var/cache/conftool/dbconfig/20240409-192000-arnaudb.json
  • 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2190 (T360332)', diff saved to https://phabricator.wikimedia.org/P60141 and previous config saved to /var/cache/conftool/dbconfig/20240409-190459-arnaudb.json
  • 19:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189', diff saved to https://phabricator.wikimedia.org/P60140 and previous config saved to /var/cache/conftool/dbconfig/20240409-190452-arnaudb.json
  • 19:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 19:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2190.codfw.wmnet with reason: Maintenance
  • 19:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60139 and previous config saved to /var/cache/conftool/dbconfig/20240409-190436-arnaudb.json
  • 18:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T356166)', diff saved to https://phabricator.wikimedia.org/P60138 and previous config saved to /var/cache/conftool/dbconfig/20240409-185817-marostegui.json
  • 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 18:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 18:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 18:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60137 and previous config saved to /var/cache/conftool/dbconfig/20240409-185736-marostegui.json
  • 18:56 eoghan@cumin1002: END (PASS) - Cookbook sre.gitlab.failover (exit_code=0) Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
  • 18:54 eoghan@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
  • 18:54 eoghan@cumin1002: START - Cookbook sre.dns.wipe-cache 'https://gitlab-replica.wikimedia.org/ https://gitlab-replica-old.wikimedia.org/' on all recursors
  • 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60136 and previous config saved to /var/cache/conftool/dbconfig/20240409-184944-arnaudb.json
  • 18:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60135 and previous config saved to /var/cache/conftool/dbconfig/20240409-184929-arnaudb.json
  • 18:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60134 and previous config saved to /var/cache/conftool/dbconfig/20240409-184702-arnaudb.json
  • 18:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2189.codfw.wmnet with reason: Maintenance
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60133 and previous config saved to /var/cache/conftool/dbconfig/20240409-184640-arnaudb.json
  • 18:43 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@875e0d2]: (no justification provided) (duration: 00m 26s)
  • 18:43 ebysans@deploy1002: Started deploy [airflow-dags/analytics@875e0d2]: (no justification provided)
  • 18:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60132 and previous config saved to /var/cache/conftool/dbconfig/20240409-184228-marostegui.json
  • 18:41 SandraEbele_: deploying airflow dag to fix mediawiki_history_metrics_monthly dag
  • 18:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177', diff saved to https://phabricator.wikimedia.org/P60131 and previous config saved to /var/cache/conftool/dbconfig/20240409-183421-arnaudb.json
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60130 and previous config saved to /var/cache/conftool/dbconfig/20240409-183132-arnaudb.json
  • 18:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P60129 and previous config saved to /var/cache/conftool/dbconfig/20240409-182721-marostegui.json
  • 18:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60128 and previous config saved to /var/cache/conftool/dbconfig/20240409-181914-arnaudb.json
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175', diff saved to https://phabricator.wikimedia.org/P60127 and previous config saved to /var/cache/conftool/dbconfig/20240409-181625-arnaudb.json
  • 18:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60126 and previous config saved to /var/cache/conftool/dbconfig/20240409-181213-marostegui.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2177 (T360332)', diff saved to https://phabricator.wikimedia.org/P60125 and previous config saved to /var/cache/conftool/dbconfig/20240409-180306-arnaudb.json
  • 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2177.codfw.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60124 and previous config saved to /var/cache/conftool/dbconfig/20240409-180242-arnaudb.json
  • 18:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60123 and previous config saved to /var/cache/conftool/dbconfig/20240409-180117-arnaudb.json
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60122 and previous config saved to /var/cache/conftool/dbconfig/20240409-175837-arnaudb.json
  • 17:58 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2175.codfw.wmnet with reason: Maintenance
  • 17:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60121 and previous config saved to /var/cache/conftool/dbconfig/20240409-175813-arnaudb.json
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60120 and previous config saved to /var/cache/conftool/dbconfig/20240409-174734-arnaudb.json
  • 17:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60119 and previous config saved to /var/cache/conftool/dbconfig/20240409-174306-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156', diff saved to https://phabricator.wikimedia.org/P60118 and previous config saved to /var/cache/conftool/dbconfig/20240409-173226-arnaudb.json
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148', diff saved to https://phabricator.wikimedia.org/P60117 and previous config saved to /var/cache/conftool/dbconfig/20240409-172758-arnaudb.json
  • 17:25 denisse: Delete unused grafana-labs TLS certificates - T360414
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60116 and previous config saved to /var/cache/conftool/dbconfig/20240409-171719-arnaudb.json
  • 17:16 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
  • 17:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp1113.eqiad.wmnet with OS bullseye
  • 17:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60115 and previous config saved to /var/cache/conftool/dbconfig/20240409-171251-arnaudb.json
  • 17:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2148 (T360332)', diff saved to https://phabricator.wikimedia.org/P60114 and previous config saved to /var/cache/conftool/dbconfig/20240409-171009-arnaudb.json
  • 17:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2148.codfw.wmnet with reason: Maintenance
  • 17:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60113 and previous config saved to /var/cache/conftool/dbconfig/20240409-170946-arnaudb.json
  • 17:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60112 and previous config saved to /var/cache/conftool/dbconfig/20240409-170234-arnaudb.json
  • 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2156.codfw.wmnet with reason: Maintenance
  • 17:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60111 and previous config saved to /var/cache/conftool/dbconfig/20240409-170155-arnaudb.json
  • 16:56 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
  • 16:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60110 and previous config saved to /var/cache/conftool/dbconfig/20240409-165438-arnaudb.json
  • 16:54 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: host reimage
  • 16:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60109 and previous config saved to /var/cache/conftool/dbconfig/20240409-164647-arnaudb.json
  • 16:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138', diff saved to https://phabricator.wikimedia.org/P60108 and previous config saved to /var/cache/conftool/dbconfig/20240409-163931-arnaudb.json
  • 16:38 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
  • 16:37 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp1113.eqiad.wmnet with OS bullseye
  • 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149', diff saved to https://phabricator.wikimedia.org/P60107 and previous config saved to /var/cache/conftool/dbconfig/20240409-163140-arnaudb.json
  • 16:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60106 and previous config saved to /var/cache/conftool/dbconfig/20240409-162423-arnaudb.json
  • 16:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp1113.eqiad.wmnet with OS bullseye
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2138 (T360332)', diff saved to https://phabricator.wikimedia.org/P60105 and previous config saved to /var/cache/conftool/dbconfig/20240409-162142-arnaudb.json
  • 16:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 16:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2138.codfw.wmnet with reason: Maintenance
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60104 and previous config saved to /var/cache/conftool/dbconfig/20240409-162119-arnaudb.json
  • 16:21 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp1113.eqiad.wmnet
  • 16:20 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp1113.eqiad.wmnet
  • 16:20 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
  • 16:19 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp1113.eqiad.wmnet with reason: NIC firmware upgrade and reimage
  • 16:19 sukhe@puppetmaster1001: conftool action : set/pooled=no; selector: name=cp1113.eqiad.wmnet,service=(cdn|ats-be)
  • 16:16 sukhe: depool cp1113 for PXE boot issue related testing T350179
  • 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60103 and previous config saved to /var/cache/conftool/dbconfig/20240409-161632-arnaudb.json
  • 16:13 denisse: Deleting unused webperf TLS certificates - T360414
  • 16:13 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 16:06 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 16:06 sukhe: pool cp4052 after reimaging and new NIC firmware
  • 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60102 and previous config saved to /var/cache/conftool/dbconfig/20240409-160612-arnaudb.json
  • 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2149 (T360332)', diff saved to https://phabricator.wikimedia.org/P60101 and previous config saved to /var/cache/conftool/dbconfig/20240409-160225-arnaudb.json
  • 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2149.codfw.wmnet with reason: Maintenance
  • 16:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:56 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
  • 15:54 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:54 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1005.eqiad.wmnet with reason: host reimage
  • 15:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126', diff saved to https://phabricator.wikimedia.org/P60100 and previous config saved to /var/cache/conftool/dbconfig/20240409-155104-arnaudb.json
  • 15:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 15:45 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:42 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on matomo1003.eqiad.wmnet with reason: host reimage
  • 15:40 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 15:39 moritzm: installing python2.7 security updates
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60099 and previous config saved to /var/cache/conftool/dbconfig/20240409-153557-arnaudb.json
  • 15:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 15:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2109.codfw.wmnet with reason: Maintenance
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60098 and previous config saved to /var/cache/conftool/dbconfig/20240409-153512-arnaudb.json
  • 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2126 (T360332)', diff saved to https://phabricator.wikimedia.org/P60097 and previous config saved to /var/cache/conftool/dbconfig/20240409-153315-arnaudb.json
  • 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2126.codfw.wmnet with reason: Maintenance
  • 15:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60096 and previous config saved to /var/cache/conftool/dbconfig/20240409-153257-arnaudb.json
  • 15:32 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 15:32 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 15:31 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
  • 15:26 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 15:24 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 15:22 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/services/mw-api-int: apply
  • 15:22 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/services/mw-api-int: apply
  • 15:20 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-api-int: apply
  • 15:20 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-api-int: apply
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60095 and previous config saved to /var/cache/conftool/dbconfig/20240409-152005-arnaudb.json
  • 15:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60094 and previous config saved to /var/cache/conftool/dbconfig/20240409-151750-arnaudb.json
  • 15:16 eoghan@cumin1002: START - Cookbook sre.gitlab.failover Failover of gitlab from gitlab1003.wikimedia.org to gitlab1004.wikimedia.org
  • 15:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 15:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105', diff saved to https://phabricator.wikimedia.org/P60093 and previous config saved to /var/cache/conftool/dbconfig/20240409-150458-arnaudb.json
  • 15:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125', diff saved to https://phabricator.wikimedia.org/P60092 and previous config saved to /var/cache/conftool/dbconfig/20240409-150242-arnaudb.json
  • 15:02 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:01 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on A:cp-ulsfo and not P{cp[4037,4041,4045,4049,4052].ulsfo.wmnet} and A:cp
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60091 and previous config saved to /var/cache/conftool/dbconfig/20240409-144950-arnaudb.json
  • 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60090 and previous config saved to /var/cache/conftool/dbconfig/20240409-144735-arnaudb.json
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2125 (T360332)', diff saved to https://phabricator.wikimedia.org/P60089 and previous config saved to /var/cache/conftool/dbconfig/20240409-144447-arnaudb.json
  • 14:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 14:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2125.codfw.wmnet with reason: Maintenance
  • 14:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60088 and previous config saved to /var/cache/conftool/dbconfig/20240409-144424-arnaudb.json
  • 14:35 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2105 (T360332)', diff saved to https://phabricator.wikimedia.org/P60087 and previous config saved to /var/cache/conftool/dbconfig/20240409-143445-arnaudb.json
  • 14:34 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 14:34 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2105.codfw.wmnet with reason: Maintenance
  • 14:33 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=99) for new host matomo1003.eqiad.wmnet
  • 14:33 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host matomo1003.eqiad.wmnet with OS bookworm
  • 14:30 sukhe@cumin1002: END (ERROR) - Cookbook sre.hardware.upgrade-firmware (exit_code=97) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:30 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 14:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60086 and previous config saved to /var/cache/conftool/dbconfig/20240409-142916-arnaudb.json
  • 14:28 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 14:24 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 14:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60085 and previous config saved to /var/cache/conftool/dbconfig/20240409-142414-arnaudb.json
  • 14:18 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
  • 14:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107', diff saved to https://phabricator.wikimedia.org/P60084 and previous config saved to /var/cache/conftool/dbconfig/20240409-141410-arnaudb.json
  • 14:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60083 and previous config saved to /var/cache/conftool/dbconfig/20240409-140906-arnaudb.json
  • 14:07 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:07 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1005.eqiad.wmnet with OS bullseye
  • 14:04 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:03 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 14:02 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60082 and previous config saved to /var/cache/conftool/dbconfig/20240409-135902-arnaudb.json
  • 13:57 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
  • 13:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2107 (T360332)', diff saved to https://phabricator.wikimedia.org/P60081 and previous config saved to /var/cache/conftool/dbconfig/20240409-135621-arnaudb.json
  • 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2107.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 13:54 cgoubert@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-debug: apply
  • 13:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1246.eqiad.wmnet with reason: Maintenance
  • 13:54 cgoubert@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-debug: apply
  • 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P60080 and previous config saved to /var/cache/conftool/dbconfig/20240409-135359-arnaudb.json
  • 13:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 13:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 13:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60079 and previous config saved to /var/cache/conftool/dbconfig/20240409-135335-arnaudb.json
  • 13:42 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:42 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
  • 13:38 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dumpsdata1004.eqiad.wmnet with reason: host reimage
  • 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60078 and previous config saved to /var/cache/conftool/dbconfig/20240409-133852-arnaudb.json
  • 13:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60077 and previous config saved to /var/cache/conftool/dbconfig/20240409-133827-arnaudb.json
  • 13:35 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 13:33 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reboot-single (exit_code=0) for host cp4052.ulsfo.wmnet
  • 13:32 jforrester@deploy1002: Finished scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) (duration: 14m 31s)
  • 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T360332)', diff saved to https://phabricator.wikimedia.org/P60076 and previous config saved to /var/cache/conftool/dbconfig/20240409-133135-arnaudb.json
  • 13:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 13:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 13:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60075 and previous config saved to /var/cache/conftool/dbconfig/20240409-133112-arnaudb.json
  • 13:25 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 13:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233', diff saved to https://phabricator.wikimedia.org/P60074 and previous config saved to /var/cache/conftool/dbconfig/20240409-132320-arnaudb.json
  • 13:21 jforrester@deploy1002: sdhehua and jforrester: Continuing with sync
  • 13:21 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host matomo1003.eqiad.wmnet with OS bookworm
  • 13:20 jforrester@deploy1002: sdhehua and jforrester: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:18 sukhe@cumin1002: START - Cookbook sre.hosts.reboot-single for host cp4052.ulsfo.wmnet
  • 13:17 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
  • 13:17 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: BIOS firmware upgrade
  • 13:17 jforrester@deploy1002: Started scap: Backport for zhwiki:Add centralauth-createlocal to ipblock exempt granter (T361184)
  • 13:16 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 13:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60073 and previous config saved to /var/cache/conftool/dbconfig/20240409-131605-arnaudb.json
  • 13:16 sukhe: depool cp4052 for firmware upgrade
  • 13:16 jforrester@deploy1002: Finished scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) (duration: 14m 09s)
  • 13:14 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 13:14 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on stat1010.eqiad.wmnet with reason: Connecting GPU power cable
  • 13:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60072 and previous config saved to /var/cache/conftool/dbconfig/20240409-130812-arnaudb.json
  • 13:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1233 (T360332)', diff saved to https://phabricator.wikimedia.org/P60071 and previous config saved to /var/cache/conftool/dbconfig/20240409-130543-arnaudb.json
  • 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1233.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60070 and previous config saved to /var/cache/conftool/dbconfig/20240409-130520-arnaudb.json
  • 13:04 jforrester@deploy1002: jforrester: Continuing with sync
  • 13:04 jforrester@deploy1002: jforrester: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:01 jforrester@deploy1002: Started scap: Backport for ExtensionDistributor: Add REL1_42 as a beta (T359844)
  • 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P60069 and previous config saved to /var/cache/conftool/dbconfig/20240409-130057-arnaudb.json
  • 12:53 vgutierrez: uploaded golang-github-gopacket-gopacket_1.2.0-2~wmf1 to apt.wm.o (bookworm)
  • 12:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: wmcs::openstack::codfw1dev::cinder_backups
  • 12:50 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 12:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229', diff saved to https://phabricator.wikimedia.org/P60068 and previous config saved to /var/cache/conftool/dbconfig/20240409-125012-arnaudb.json
  • 12:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T360332)', diff saved to https://phabricator.wikimedia.org/P60067 and previous config saved to /var/cache/conftool/dbconfig/20240409-124550-arnaudb.json
  • 12:45 elukey@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 12:44 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-eqiad: Deploy new Truststore - elukey@cumin1002
  • 12:43 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: wmcs::openstack::codfw1dev::cinder_backups
  • 12:25 gmodena@deploy1002: Finished deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6] (duration: 15m 41s)
  • 12:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host cloudbackup1001-dev.eqiad.wmnet
  • 12:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60063 and previous config saved to /var/cache/conftool/dbconfig/20240409-122050-arnaudb.json
  • 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60062 and previous config saved to /var/cache/conftool/dbconfig/20240409-121958-arnaudb.json
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1229 (T360332)', diff saved to https://phabricator.wikimedia.org/P60061 and previous config saved to /var/cache/conftool/dbconfig/20240409-121722-arnaudb.json
  • 12:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 12:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1229.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60060 and previous config saved to /var/cache/conftool/dbconfig/20240409-121622-arnaudb.json
  • 12:09 gmodena@deploy1002: Started deploy [analytics/refinery@d45a15b]: Regular analytics weekly train [analytics/refinery@d45a15b6]
  • 12:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P60059 and previous config saved to /var/cache/conftool/dbconfig/20240409-120542-arnaudb.json
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60058 and previous config saved to /var/cache/conftool/dbconfig/20240409-120115-arnaudb.json
  • 11:56 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:55 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) matomo1003.eqiad.wmnet on all recursors
  • 11:54 btullis@cumin1002: START - Cookbook sre.dns.wipe-cache matomo1003.eqiad.wmnet on all recursors
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 11:54 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60057 and previous config saved to /var/cache/conftool/dbconfig/20240409-115035-arnaudb.json
  • 11:49 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:48 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:47 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:47 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197', diff saved to https://phabricator.wikimedia.org/P60056 and previous config saved to /var/cache/conftool/dbconfig/20240409-114607-arnaudb.json
  • 11:44 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2112.codfw.wmnet onto db2212.codfw.wmnet
  • 11:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T360332)', diff saved to https://phabricator.wikimedia.org/P60055 and previous config saved to /var/cache/conftool/dbconfig/20240409-114411-arnaudb.json
  • 11:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:43 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60054 and previous config saved to /var/cache/conftool/dbconfig/20240409-114349-arnaudb.json
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2112 in db2212 for T355422', diff saved to https://phabricator.wikimedia.org/P60053 and previous config saved to /var/cache/conftool/dbconfig/20240409-114302-arnaudb.json
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:42 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2212.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2112.codfw.wmnet with reason: provisionning db2212.codfw.wmnet - T355422
  • 11:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60051 and previous config saved to /var/cache/conftool/dbconfig/20240409-113100-arnaudb.json
  • 11:29 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM matomo1003.eqiad.wmnet - btullis@cumin1002"
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60050 and previous config saved to /var/cache/conftool/dbconfig/20240409-112841-arnaudb.json
  • 11:27 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 11:27 btullis@cumin1002: START - Cookbook sre.ganeti.makevm for new host matomo1003.eqiad.wmnet
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1197 (T360332)', diff saved to https://phabricator.wikimedia.org/P60049 and previous config saved to /var/cache/conftool/dbconfig/20240409-112728-arnaudb.json
  • 11:27 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1197.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60048 and previous config saved to /var/cache/conftool/dbconfig/20240409-112705-arnaudb.json
  • 11:15 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host dumpsdata1004.eqiad.wmnet with OS bullseye
  • 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P60047 and previous config saved to /var/cache/conftool/dbconfig/20240409-111334-arnaudb.json
  • 11:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60046 and previous config saved to /var/cache/conftool/dbconfig/20240409-111157-arnaudb.json
  • 11:01 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host snapshot1015.eqiad.wmnet with OS bullseye
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60045 and previous config saved to /var/cache/conftool/dbconfig/20240409-105827-arnaudb.json
  • 10:58 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P60044 and previous config saved to /var/cache/conftool/dbconfig/20240409-105814-root.json
  • 10:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188', diff saved to https://phabricator.wikimedia.org/P60043 and previous config saved to /var/cache/conftool/dbconfig/20240409-105650-arnaudb.json
  • 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T360332)', diff saved to https://phabricator.wikimedia.org/P60042 and previous config saved to /var/cache/conftool/dbconfig/20240409-104853-arnaudb.json
  • 10:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 10:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 10:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60041 and previous config saved to /var/cache/conftool/dbconfig/20240409-104830-arnaudb.json
  • 10:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 10:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 10:44 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 10:43 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P60040 and previous config saved to /var/cache/conftool/dbconfig/20240409-104308-root.json
  • 10:42 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 10:41 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 10:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60039 and previous config saved to /var/cache/conftool/dbconfig/20240409-104143-arnaudb.json
  • 10:41 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1188 (T360332)', diff saved to https://phabricator.wikimedia.org/P60038 and previous config saved to /var/cache/conftool/dbconfig/20240409-103908-arnaudb.json
  • 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 10:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1188.eqiad.wmnet with reason: Maintenance
  • 10:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60037 and previous config saved to /var/cache/conftool/dbconfig/20240409-103845-arnaudb.json
  • 10:34 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
  • 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60036 and previous config saved to /var/cache/conftool/dbconfig/20240409-103323-arnaudb.json
  • 10:32 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
  • 10:31 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 14 days, 0:00:00 on db2097.codfw.wmnet with reason: host weirdness and possible decom
  • 10:31 btullis@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on snapshot1015.eqiad.wmnet with reason: host reimage
  • 10:28 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P60035 and previous config saved to /var/cache/conftool/dbconfig/20240409-102803-root.json
  • 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60034 and previous config saved to /var/cache/conftool/dbconfig/20240409-102337-arnaudb.json
  • 10:23 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60033 and previous config saved to /var/cache/conftool/dbconfig/20240409-102312-arnaudb.json
  • 10:19 btullis@cumin1002: START - Cookbook sre.hosts.reimage for host snapshot1015.eqiad.wmnet with OS bullseye
  • 10:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P60032 and previous config saved to /var/cache/conftool/dbconfig/20240409-101815-arnaudb.json
  • 10:12 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P60031 and previous config saved to /var/cache/conftool/dbconfig/20240409-101257-root.json
  • 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182', diff saved to https://phabricator.wikimedia.org/P60030 and previous config saved to /var/cache/conftool/dbconfig/20240409-100830-arnaudb.json
  • 10:08 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60029 and previous config saved to /var/cache/conftool/dbconfig/20240409-100806-arnaudb.json
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T356166)', diff saved to https://phabricator.wikimedia.org/P60028 and previous config saved to /var/cache/conftool/dbconfig/20240409-100635-marostegui.json
  • 10:06 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60027 and previous config saved to /var/cache/conftool/dbconfig/20240409-100612-marostegui.json
  • 10:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60026 and previous config saved to /var/cache/conftool/dbconfig/20240409-100308-arnaudb.json
  • 10:02 Emperor: puppet cert clean swift_eqiad T361844
  • 09:57 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P60025 and previous config saved to /var/cache/conftool/dbconfig/20240409-095751-root.json
  • 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T360332)', diff saved to https://phabricator.wikimedia.org/P60024 and previous config saved to /var/cache/conftool/dbconfig/20240409-095642-arnaudb.json
  • 09:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 09:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 09:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60023 and previous config saved to /var/cache/conftool/dbconfig/20240409-095620-arnaudb.json
  • 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60022 and previous config saved to /var/cache/conftool/dbconfig/20240409-095323-arnaudb.json
  • 09:53 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60021 and previous config saved to /var/cache/conftool/dbconfig/20240409-095300-arnaudb.json
  • 09:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60020 and previous config saved to /var/cache/conftool/dbconfig/20240409-095105-marostegui.json
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1182 (T360332)', diff saved to https://phabricator.wikimedia.org/P60019 and previous config saved to /var/cache/conftool/dbconfig/20240409-095054-arnaudb.json
  • 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1182.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1162.eqiad.wmnet with reason: Maintenance
  • 09:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60018 and previous config saved to /var/cache/conftool/dbconfig/20240409-095004-arnaudb.json
  • 09:46 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:45 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P60017 and previous config saved to /var/cache/conftool/dbconfig/20240409-094246-root.json
  • 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60016 and previous config saved to /var/cache/conftool/dbconfig/20240409-094113-arnaudb.json
  • 09:40 hashar@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.26 refs T360158
  • 09:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60015 and previous config saved to /var/cache/conftool/dbconfig/20240409-093755-arnaudb.json
  • 09:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P60014 and previous config saved to /var/cache/conftool/dbconfig/20240409-093551-marostegui.json
  • 09:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60013 and previous config saved to /var/cache/conftool/dbconfig/20240409-093457-arnaudb.json
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db1162 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P60012 and previous config saved to /var/cache/conftool/dbconfig/20240409-092740-root.json
  • 09:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P60011 and previous config saved to /var/cache/conftool/dbconfig/20240409-092605-arnaudb.json
  • 09:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 16%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60010 and previous config saved to /var/cache/conftool/dbconfig/20240409-092249-arnaudb.json
  • 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P60009 and previous config saved to /var/cache/conftool/dbconfig/20240409-092043-marostegui.json
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156', diff saved to https://phabricator.wikimedia.org/P60008 and previous config saved to /var/cache/conftool/dbconfig/20240409-091949-arnaudb.json
  • 09:18 hashar@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.26 refs T360158 (duration: 50m 11s)
  • 09:11 moritzm: installing postgresql-13 security updates
  • 09:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60007 and previous config saved to /var/cache/conftool/dbconfig/20240409-091057-arnaudb.json
  • 09:07 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 8%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60006 and previous config saved to /var/cache/conftool/dbconfig/20240409-090744-arnaudb.json
  • 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60005 and previous config saved to /var/cache/conftool/dbconfig/20240409-090442-arnaudb.json
  • 09:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T360332)', diff saved to https://phabricator.wikimedia.org/P60004 and previous config saved to /var/cache/conftool/dbconfig/20240409-090435-arnaudb.json
  • 09:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 09:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1156 (T360332)', diff saved to https://phabricator.wikimedia.org/P60003 and previous config saved to /var/cache/conftool/dbconfig/20240409-090210-arnaudb.json
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 09:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1156.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:56 marostegui@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
  • 08:52 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 4%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60002 and previous config saved to /var/cache/conftool/dbconfig/20240409-085238-arnaudb.json
  • 08:37 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 2%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60001 and previous config saved to /var/cache/conftool/dbconfig/20240409-083733-arnaudb.json
  • 08:28 hashar@deploy1002: Started scap: testwikis wikis to 1.42.0-wmf.26 refs T360158
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 1%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P60000 and previous config saved to /var/cache/conftool/dbconfig/20240409-082227-arnaudb.json
  • 08:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2113.codfw.wmnet with OS bookworm
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
  • 07:54 Emperor: puppet cert clean swift_codfw T361844
  • 07:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2113.codfw.wmnet with reason: host reimage
  • 07:37 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2113.codfw.wmnet with OS bookworm
  • 07:34 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 depool for reimage T360116', diff saved to https://phabricator.wikimedia.org/P59999 and previous config saved to /var/cache/conftool/dbconfig/20240409-073406-arnaudb.json
  • 07:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
  • 07:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2213.codfw.wmnet with reason: Silence for reimage
  • 07:29 kartik@deploy1002: Finished scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) (duration: 25m 30s)
  • 07:16 kartik@deploy1002: kartik: Continuing with sync
  • 07:09 kartik@deploy1002: kartik: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 07:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 07:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59998 and previous config saved to /var/cache/conftool/dbconfig/20240409-070613-arnaudb.json
  • 07:04 kartik@deploy1002: Started scap: Backport for ContentTranslation: Limit publishing in zhwiki for extendedconfirmed users only (T349959)
  • 07:01 marostegui@cumin1002: START - Cookbook sre.mysql.clone of db1162.eqiad.wmnet onto db1246.eqiad.wmnet
  • 06:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59997 and previous config saved to /var/cache/conftool/dbconfig/20240409-065105-arnaudb.json
  • 06:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219', diff saved to https://phabricator.wikimedia.org/P59996 and previous config saved to /var/cache/conftool/dbconfig/20240409-063558-arnaudb.json
  • 06:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59995 and previous config saved to /var/cache/conftool/dbconfig/20240409-062050-arnaudb.json
  • 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2219 (T360332)', diff saved to https://phabricator.wikimedia.org/P59994 and previous config saved to /var/cache/conftool/dbconfig/20240409-061830-arnaudb.json
  • 06:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
  • 06:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2219.codfw.wmnet with reason: Maintenance
  • 06:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59993 and previous config saved to /var/cache/conftool/dbconfig/20240409-061807-arnaudb.json
  • 06:15 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1246.eqiad.wmnet with OS bookworm
  • 06:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1162.eqiad.wmnet with OS bookworm
  • 06:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59992 and previous config saved to /var/cache/conftool/dbconfig/20240409-060300-arnaudb.json
  • 05:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
  • 05:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1246.eqiad.wmnet with reason: host reimage
  • 05:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
  • 05:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210', diff saved to https://phabricator.wikimedia.org/P59991 and previous config saved to /var/cache/conftool/dbconfig/20240409-054752-arnaudb.json
  • 05:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1162.eqiad.wmnet with reason: host reimage
  • 05:39 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1246.eqiad.wmnet with OS bookworm
  • 05:33 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1162.eqiad.wmnet with OS bookworm
  • 05:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59990 and previous config saved to /var/cache/conftool/dbconfig/20240409-053245-arnaudb.json
  • 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59989 and previous config saved to /var/cache/conftool/dbconfig/20240409-053024-arnaudb.json
  • 05:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
  • 05:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1162 T362036', diff saved to https://phabricator.wikimedia.org/P59988 and previous config saved to /var/cache/conftool/dbconfig/20240409-053005-root.json
  • 05:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2210.codfw.wmnet with reason: Maintenance
  • 05:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59987 and previous config saved to /var/cache/conftool/dbconfig/20240409-053001-arnaudb.json
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Promote db1222 to s2 primary and set section read-write T362036', diff saved to https://phabricator.wikimedia.org/P59986 and previous config saved to /var/cache/conftool/dbconfig/20240409-052855-marostegui.json
  • 05:28 marostegui@cumin1002: dbctl commit (dc=all): 'Set s2 eqiad as read-only for maintenance - T362036', diff saved to https://phabricator.wikimedia.org/P59985 and previous config saved to /var/cache/conftool/dbconfig/20240409-052827-marostegui.json
  • 05:28 marostegui: Starting s2 eqiad failover from db1162 to db1222 - T362036
  • 05:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59984 and previous config saved to /var/cache/conftool/dbconfig/20240409-051454-arnaudb.json
  • 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Set db1222 with weight 0 T362036', diff saved to https://phabricator.wikimedia.org/P59983 and previous config saved to /var/cache/conftool/dbconfig/20240409-051027-marostegui.json
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
  • 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 28 hosts with reason: Primary switchover s2 T362036
  • 04:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206', diff saved to https://phabricator.wikimedia.org/P59982 and previous config saved to /var/cache/conftool/dbconfig/20240409-045946-arnaudb.json
  • 04:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59981 and previous config saved to /var/cache/conftool/dbconfig/20240409-044438-arnaudb.json
  • 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2206 (T360332)', diff saved to https://phabricator.wikimedia.org/P59980 and previous config saved to /var/cache/conftool/dbconfig/20240409-044216-arnaudb.json
  • 04:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
  • 04:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2206.codfw.wmnet with reason: Maintenance
  • 04:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59979 and previous config saved to /var/cache/conftool/dbconfig/20240409-044204-arnaudb.json
  • 04:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59978 and previous config saved to /var/cache/conftool/dbconfig/20240409-042657-arnaudb.json
  • 04:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172', diff saved to https://phabricator.wikimedia.org/P59977 and previous config saved to /var/cache/conftool/dbconfig/20240409-041149-arnaudb.json
  • 03:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59976 and previous config saved to /var/cache/conftool/dbconfig/20240409-035641-arnaudb.json
  • 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59975 and previous config saved to /var/cache/conftool/dbconfig/20240409-035422-arnaudb.json
  • 03:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 03:54 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2172.codfw.wmnet with reason: Maintenance
  • 03:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59974 and previous config saved to /var/cache/conftool/dbconfig/20240409-035359-arnaudb.json
  • 03:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59973 and previous config saved to /var/cache/conftool/dbconfig/20240409-033851-arnaudb.json
  • 03:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155', diff saved to https://phabricator.wikimedia.org/P59972 and previous config saved to /var/cache/conftool/dbconfig/20240409-032344-arnaudb.json
  • 03:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59971 and previous config saved to /var/cache/conftool/dbconfig/20240409-030836-arnaudb.json
  • 03:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2155 (T360332)', diff saved to https://phabricator.wikimedia.org/P59970 and previous config saved to /var/cache/conftool/dbconfig/20240409-030617-arnaudb.json
  • 03:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2155.codfw.wmnet with reason: Maintenance
  • 03:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59969 and previous config saved to /var/cache/conftool/dbconfig/20240409-030537-arnaudb.json
  • 02:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59968 and previous config saved to /var/cache/conftool/dbconfig/20240409-025030-arnaudb.json
  • 02:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147', diff saved to https://phabricator.wikimedia.org/P59967 and previous config saved to /var/cache/conftool/dbconfig/20240409-023522-arnaudb.json
  • 02:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59966 and previous config saved to /var/cache/conftool/dbconfig/20240409-022015-arnaudb.json
  • 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2147 (T360332)', diff saved to https://phabricator.wikimedia.org/P59965 and previous config saved to /var/cache/conftool/dbconfig/20240409-021755-arnaudb.json
  • 02:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 02:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2147.codfw.wmnet with reason: Maintenance
  • 02:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59964 and previous config saved to /var/cache/conftool/dbconfig/20240409-021731-arnaudb.json
  • 02:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59963 and previous config saved to /var/cache/conftool/dbconfig/20240409-020223-arnaudb.json
  • 01:49 ebysans@deploy1002: Finished deploy [airflow-dags/analytics@6bb821b]: (no justification provided) (duration: 00m 31s)
  • 01:49 ebysans@deploy1002: Started deploy [airflow-dags/analytics@6bb821b]: (no justification provided)
  • 01:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140', diff saved to https://phabricator.wikimedia.org/P59962 and previous config saved to /var/cache/conftool/dbconfig/20240409-014716-arnaudb.json
  • 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T356166)', diff saved to https://phabricator.wikimedia.org/P59961 and previous config saved to /var/cache/conftool/dbconfig/20240409-013231-marostegui.json
  • 01:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 01:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59960 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-arnaudb.json
  • 01:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59959 and previous config saved to /var/cache/conftool/dbconfig/20240409-013208-marostegui.json
  • 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2140 (T360332)', diff saved to https://phabricator.wikimedia.org/P59958 and previous config saved to /var/cache/conftool/dbconfig/20240409-012949-arnaudb.json
  • 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2140.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2139.codfw.wmnet with reason: Maintenance
  • 01:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59957 and previous config saved to /var/cache/conftool/dbconfig/20240409-012858-arnaudb.json
  • 01:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59956 and previous config saved to /var/cache/conftool/dbconfig/20240409-011700-marostegui.json
  • 01:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59955 and previous config saved to /var/cache/conftool/dbconfig/20240409-011351-arnaudb.json
  • 01:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59954 and previous config saved to /var/cache/conftool/dbconfig/20240409-010152-marostegui.json
  • 00:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137', diff saved to https://phabricator.wikimedia.org/P59953 and previous config saved to /var/cache/conftool/dbconfig/20240409-005843-arnaudb.json
  • 00:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59952 and previous config saved to /var/cache/conftool/dbconfig/20240409-004645-marostegui.json
  • 00:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59951 and previous config saved to /var/cache/conftool/dbconfig/20240409-004336-arnaudb.json
  • 00:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2137 (T360332)', diff saved to https://phabricator.wikimedia.org/P59950 and previous config saved to /var/cache/conftool/dbconfig/20240409-004115-arnaudb.json
  • 00:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 00:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2137.codfw.wmnet with reason: Maintenance
  • 00:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59949 and previous config saved to /var/cache/conftool/dbconfig/20240409-004052-arnaudb.json
  • 00:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59948 and previous config saved to /var/cache/conftool/dbconfig/20240409-002545-arnaudb.json
  • 00:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136', diff saved to https://phabricator.wikimedia.org/P59947 and previous config saved to /var/cache/conftool/dbconfig/20240409-001037-arnaudb.json

2024-04-08

  • 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59946 and previous config saved to /var/cache/conftool/dbconfig/20240408-235530-arnaudb.json
  • 23:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2136 (T360332)', diff saved to https://phabricator.wikimedia.org/P59945 and previous config saved to /var/cache/conftool/dbconfig/20240408-235309-arnaudb.json
  • 23:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 23:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2136.codfw.wmnet with reason: Maintenance
  • 23:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59944 and previous config saved to /var/cache/conftool/dbconfig/20240408-235247-arnaudb.json
  • 23:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59943 and previous config saved to /var/cache/conftool/dbconfig/20240408-233739-arnaudb.json
  • 23:25 tstarling@deploy1002: Synchronized wmf-config/CommonSettings.php: stop writing to ipblocks table T355034 (duration: 12m 32s)
  • 23:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119', diff saved to https://phabricator.wikimedia.org/P59942 and previous config saved to /var/cache/conftool/dbconfig/20240408-232231-arnaudb.json
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59941 and previous config saved to /var/cache/conftool/dbconfig/20240408-230723-arnaudb.json
  • 23:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2119 (T360332)', diff saved to https://phabricator.wikimedia.org/P59940 and previous config saved to /var/cache/conftool/dbconfig/20240408-230502-arnaudb.json
  • 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2119.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2110.codfw.wmnet with reason: Maintenance
  • 23:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59939 and previous config saved to /var/cache/conftool/dbconfig/20240408-230410-arnaudb.json
  • 22:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59938 and previous config saved to /var/cache/conftool/dbconfig/20240408-224903-arnaudb.json
  • 22:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106', diff saved to https://phabricator.wikimedia.org/P59937 and previous config saved to /var/cache/conftool/dbconfig/20240408-223355-arnaudb.json
  • 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59935 and previous config saved to /var/cache/conftool/dbconfig/20240408-221847-arnaudb.json
  • 22:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2106 (T360332)', diff saved to https://phabricator.wikimedia.org/P59934 and previous config saved to /var/cache/conftool/dbconfig/20240408-221626-arnaudb.json
  • 22:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 22:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2106.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2099.codfw.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 22:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59933 and previous config saved to /var/cache/conftool/dbconfig/20240408-221509-arnaudb.json
  • 22:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59931 and previous config saved to /var/cache/conftool/dbconfig/20240408-220001-arnaudb.json
  • 21:55 eileen: config revision changed from fff12a6a to 3c1a0267
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts wdqs1025.eqiad.wmnet
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 21:48 ryankemper@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
  • 21:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249', diff saved to https://phabricator.wikimedia.org/P59930 and previous config saved to /var/cache/conftool/dbconfig/20240408-214454-arnaudb.json
  • 21:38 eileen: config revision changed from b08eb273 to fff12a6a
  • 21:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59929 and previous config saved to /var/cache/conftool/dbconfig/20240408-212946-arnaudb.json
  • 21:28 ryankemper@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: wdqs1025.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - ryankemper@cumin2002"
  • 21:27 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1249 (T360332)', diff saved to https://phabricator.wikimedia.org/P59928 and previous config saved to /var/cache/conftool/dbconfig/20240408-212628-arnaudb.json
  • 21:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 21:26 ryankemper@cumin2002: START - Cookbook sre.dns.netbox
  • 21:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1249.eqiad.wmnet with reason: Maintenance
  • 21:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59927 and previous config saved to /var/cache/conftool/dbconfig/20240408-212605-arnaudb.json
  • 21:20 ryankemper@cumin2002: START - Cookbook sre.hosts.decommission for hosts wdqs1025.eqiad.wmnet
  • 21:18 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:17 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59926 and previous config saved to /var/cache/conftool/dbconfig/20240408-211056-arnaudb.json
  • 21:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:01 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248', diff saved to https://phabricator.wikimedia.org/P59925 and previous config saved to /var/cache/conftool/dbconfig/20240408-205548-arnaudb.json
  • 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:53 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:52 ebernhardson@deploy1002: Finished scap: Backport for cirrus: Restore traffic to codfw clusters (duration: 15m 25s)
  • 20:42 ebernhardson@deploy1002: ebernhardson: Continuing with sync
  • 20:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59924 and previous config saved to /var/cache/conftool/dbconfig/20240408-204041-arnaudb.json
  • 20:39 ebernhardson@deploy1002: ebernhardson: Backport for cirrus: Restore traffic to codfw clusters synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1248 (T360332)', diff saved to https://phabricator.wikimedia.org/P59923 and previous config saved to /var/cache/conftool/dbconfig/20240408-203723-arnaudb.json
  • 20:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 20:37 ebernhardson@deploy1002: Started scap: Backport for cirrus: Restore traffic to codfw clusters
  • 20:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1248.eqiad.wmnet with reason: Maintenance
  • 20:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59922 and previous config saved to /var/cache/conftool/dbconfig/20240408-203700-arnaudb.json
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59921 and previous config saved to /var/cache/conftool/dbconfig/20240408-202955-arnaudb.json
  • 20:25 urbanecm@deploy1002: Finished scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) (duration: 16m 25s)
  • 20:24 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:23 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59920 and previous config saved to /var/cache/conftool/dbconfig/20240408-202153-arnaudb.json
  • 20:15 urbanecm@deploy1002: urbanecm and matmarex: Continuing with sync
  • 20:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59919 and previous config saved to /var/cache/conftool/dbconfig/20240408-201447-arnaudb.json
  • 20:11 urbanecm@deploy1002: urbanecm and matmarex: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:09 urbanecm@deploy1002: Started scap: Backport for Revert "Mark all autoreviewed edits in PageSaveComplete hook" (T361918 T361940 T361960)
  • 20:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247', diff saved to https://phabricator.wikimedia.org/P59918 and previous config saved to /var/cache/conftool/dbconfig/20240408-200645-arnaudb.json
  • 19:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213', diff saved to https://phabricator.wikimedia.org/P59917 and previous config saved to /var/cache/conftool/dbconfig/20240408-195940-arnaudb.json
  • 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59916 and previous config saved to /var/cache/conftool/dbconfig/20240408-195138-arnaudb.json
  • 19:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1247 (T360332)', diff saved to https://phabricator.wikimedia.org/P59915 and previous config saved to /var/cache/conftool/dbconfig/20240408-194919-arnaudb.json
  • 19:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1247.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 19:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59914 and previous config saved to /var/cache/conftool/dbconfig/20240408-194843-arnaudb.json
  • 19:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59913 and previous config saved to /var/cache/conftool/dbconfig/20240408-194432-arnaudb.json
  • 19:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59912 and previous config saved to /var/cache/conftool/dbconfig/20240408-194113-arnaudb.json
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2213.codfw.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59911 and previous config saved to /var/cache/conftool/dbconfig/20240408-194050-arnaudb.json
  • 19:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59910 and previous config saved to /var/cache/conftool/dbconfig/20240408-193336-arnaudb.json
  • 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59909 and previous config saved to /var/cache/conftool/dbconfig/20240408-192543-arnaudb.json
  • 19:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244', diff saved to https://phabricator.wikimedia.org/P59908 and previous config saved to /var/cache/conftool/dbconfig/20240408-191828-arnaudb.json
  • 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211', diff saved to https://phabricator.wikimedia.org/P59907 and previous config saved to /var/cache/conftool/dbconfig/20240408-191035-arnaudb.json
  • 19:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59905 and previous config saved to /var/cache/conftool/dbconfig/20240408-190319-arnaudb.json
  • 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59904 and previous config saved to /var/cache/conftool/dbconfig/20240408-185528-arnaudb.json
  • 18:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59903 and previous config saved to /var/cache/conftool/dbconfig/20240408-185309-arnaudb.json
  • 18:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 18:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2211.codfw.wmnet with reason: Maintenance
  • 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59902 and previous config saved to /var/cache/conftool/dbconfig/20240408-185247-arnaudb.json
  • 18:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59901 and previous config saved to /var/cache/conftool/dbconfig/20240408-183739-arnaudb.json
  • 18:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192', diff saved to https://phabricator.wikimedia.org/P59900 and previous config saved to /var/cache/conftool/dbconfig/20240408-182232-arnaudb.json
  • 18:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59899 and previous config saved to /var/cache/conftool/dbconfig/20240408-180724-arnaudb.json
  • 18:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59898 and previous config saved to /var/cache/conftool/dbconfig/20240408-180406-arnaudb.json
  • 18:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2192.codfw.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59897 and previous config saved to /var/cache/conftool/dbconfig/20240408-180343-arnaudb.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1244 (T360332)', diff saved to https://phabricator.wikimedia.org/P59896 and previous config saved to /var/cache/conftool/dbconfig/20240408-180253-arnaudb.json
  • 18:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1244.eqiad.wmnet with reason: Maintenance
  • 18:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59895 and previous config saved to /var/cache/conftool/dbconfig/20240408-180231-arnaudb.json
  • 17:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59894 and previous config saved to /var/cache/conftool/dbconfig/20240408-174835-arnaudb.json
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59893 and previous config saved to /var/cache/conftool/dbconfig/20240408-174723-arnaudb.json
  • 17:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178', diff saved to https://phabricator.wikimedia.org/P59892 and previous config saved to /var/cache/conftool/dbconfig/20240408-173327-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243', diff saved to https://phabricator.wikimedia.org/P59891 and previous config saved to /var/cache/conftool/dbconfig/20240408-173215-arnaudb.json
  • 17:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59890 and previous config saved to /var/cache/conftool/dbconfig/20240408-171819-arnaudb.json
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59889 and previous config saved to /var/cache/conftool/dbconfig/20240408-171707-arnaudb.json
  • 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59888 and previous config saved to /var/cache/conftool/dbconfig/20240408-171502-arnaudb.json
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1243 (T360332)', diff saved to https://phabricator.wikimedia.org/P59887 and previous config saved to /var/cache/conftool/dbconfig/20240408-171448-arnaudb.json
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2178.codfw.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59886 and previous config saved to /var/cache/conftool/dbconfig/20240408-171439-arnaudb.json
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1243.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59885 and previous config saved to /var/cache/conftool/dbconfig/20240408-171425-arnaudb.json
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59884 and previous config saved to /var/cache/conftool/dbconfig/20240408-165931-arnaudb.json
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59883 and previous config saved to /var/cache/conftool/dbconfig/20240408-165917-arnaudb.json
  • 16:57 elukey@cumin1002: END (PASS) - Cookbook sre.cassandra.roll-restart (exit_code=0) for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
  • 16:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T356166)', diff saved to https://phabricator.wikimedia.org/P59882 and previous config saved to /var/cache/conftool/dbconfig/20240408-164524-marostegui.json
  • 16:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171', diff saved to https://phabricator.wikimedia.org/P59881 and previous config saved to /var/cache/conftool/dbconfig/20240408-164424-arnaudb.json
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242', diff saved to https://phabricator.wikimedia.org/P59880 and previous config saved to /var/cache/conftool/dbconfig/20240408-164410-arnaudb.json
  • 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:32 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59879 and previous config saved to /var/cache/conftool/dbconfig/20240408-162916-arnaudb.json
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59878 and previous config saved to /var/cache/conftool/dbconfig/20240408-162902-arnaudb.json
  • 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2171 (T360332)', diff saved to https://phabricator.wikimedia.org/P59877 and previous config saved to /var/cache/conftool/dbconfig/20240408-162655-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1242 (T360332)', diff saved to https://phabricator.wikimedia.org/P59876 and previous config saved to /var/cache/conftool/dbconfig/20240408-162645-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2171.codfw.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59875 and previous config saved to /var/cache/conftool/dbconfig/20240408-162633-arnaudb.json
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1242.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59874 and previous config saved to /var/cache/conftool/dbconfig/20240408-162621-arnaudb.json
  • 16:19 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching aqs20[08-12]*: Deploy new Truststore - elukey@cumin1002
  • 16:15 elukey: manually dran + restart cassandra-a on aqs2007 - cookbook failed
  • 16:15 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1002.eqiad.wmnet
  • 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:14 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59873 and previous config saved to /var/cache/conftool/dbconfig/20240408-161125-arnaudb.json
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59872 and previous config saved to /var/cache/conftool/dbconfig/20240408-161114-arnaudb.json
  • 16:07 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 16:06 elukey@cumin1002: END (FAIL) - Cookbook sre.cassandra.roll-restart (exit_code=99) for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
  • 16:01 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157', diff saved to https://phabricator.wikimedia.org/P59871 and previous config saved to /var/cache/conftool/dbconfig/20240408-155618-arnaudb.json
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241', diff saved to https://phabricator.wikimedia.org/P59870 and previous config saved to /var/cache/conftool/dbconfig/20240408-155606-arnaudb.json
  • 15:48 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:47 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:43 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1002.eqiad.wmnet
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts dumpsdata1001.eqiad.wmnet
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 15:42 btullis@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59869 and previous config saved to /var/cache/conftool/dbconfig/20240408-154110-arnaudb.json
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59868 and previous config saved to /var/cache/conftool/dbconfig/20240408-154059-arnaudb.json
  • 15:39 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp3069.esams.wmnet,service=(cdn|ats-be)
  • 15:38 btullis@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: dumpsdata1001.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - btullis@cumin1002"
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1241 (T360332)', diff saved to https://phabricator.wikimedia.org/P59867 and previous config saved to /var/cache/conftool/dbconfig/20240408-153842-arnaudb.json
  • 15:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 15:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1241.eqiad.wmnet with reason: Maintenance
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59866 and previous config saved to /var/cache/conftool/dbconfig/20240408-153819-arnaudb.json
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2157 (T360332)', diff saved to https://phabricator.wikimedia.org/P59865 and previous config saved to /var/cache/conftool/dbconfig/20240408-153753-arnaudb.json
  • 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2157.codfw.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59864 and previous config saved to /var/cache/conftool/dbconfig/20240408-153730-arnaudb.json
  • 15:31 btullis@cumin1002: START - Cookbook sre.dns.netbox
  • 15:25 btullis@cumin1002: START - Cookbook sre.hosts.decommission for hosts dumpsdata1001.eqiad.wmnet
  • 15:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59863 and previous config saved to /var/cache/conftool/dbconfig/20240408-152311-arnaudb.json
  • 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59862 and previous config saved to /var/cache/conftool/dbconfig/20240408-152221-arnaudb.json
  • 15:20 vgutierrez: Uploaded golang-gitlab-wikimedia-sre-qemutest-dev 0.1.0 to apt.wm.o (bookworm)
  • 15:17 dancy@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: testing T361608 (duration: 13m 59s)
  • 15:15 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3069.esams.wmnet with OS bullseye
  • 15:15 elukey@cumin1002: START - Cookbook sre.cassandra.roll-restart for nodes matching A:aqs-codfw: Deploy new Truststore - elukey@cumin1002
  • 15:12 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:11 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 elukey: drain and restart cassandra-a on aqs1011 to test the new truststore
  • 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:10 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221', diff saved to https://phabricator.wikimedia.org/P59860 and previous config saved to /var/cache/conftool/dbconfig/20240408-150803-arnaudb.json
  • 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128', diff saved to https://phabricator.wikimedia.org/P59859 and previous config saved to /var/cache/conftool/dbconfig/20240408-150713-arnaudb.json
  • 15:04 godog: kill -9 thanos-store on titan1001
  • 15:03 dancy@deploy1002: Started deploy [restbase/deploy@c4d19d7]: testing T361608
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59858 and previous config saved to /var/cache/conftool/dbconfig/20240408-145256-arnaudb.json
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59857 and previous config saved to /var/cache/conftool/dbconfig/20240408-145205-arnaudb.json
  • 14:51 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
  • 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2128 (T360332)', diff saved to https://phabricator.wikimedia.org/P59856 and previous config saved to /var/cache/conftool/dbconfig/20240408-144847-arnaudb.json
  • 14:48 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3069.esams.wmnet with reason: host reimage
  • 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2128.codfw.wmnet with reason: Maintenance
  • 14:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59855 and previous config saved to /var/cache/conftool/dbconfig/20240408-144808-arnaudb.json
  • 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1221 (T360332)', diff saved to https://phabricator.wikimedia.org/P59854 and previous config saved to /var/cache/conftool/dbconfig/20240408-144738-arnaudb.json
  • 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:47 jayme@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 14:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 14:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1221.eqiad.wmnet with reason: Maintenance
  • 14:47 jayme@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 14:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59853 and previous config saved to /var/cache/conftool/dbconfig/20240408-144657-arnaudb.json
  • 14:44 jayme@deploy1002: helmfile [ml-serve-eqiad] DONE helmfile.d/admin 'apply'.
  • 14:43 jayme@deploy1002: helmfile [ml-serve-eqiad] START helmfile.d/admin 'apply'.
  • 14:43 jayme@deploy1002: helmfile [ml-serve-codfw] DONE helmfile.d/admin 'apply'.
  • 14:42 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 14:42 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 14:42 jayme@deploy1002: helmfile [ml-serve-codfw] START helmfile.d/admin 'apply'.
  • 14:41 jayme@deploy1002: helmfile [ml-staging-codfw] DONE helmfile.d/admin 'apply'.
  • 14:40 jayme@deploy1002: helmfile [ml-staging-codfw] START helmfile.d/admin 'apply'.
  • 14:39 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:38 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:37 godog: bounce thanos-query and thanos-store on titan1002 - stuck on high CPU
  • 14:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59852 and previous config saved to /var/cache/conftool/dbconfig/20240408-143301-arnaudb.json
  • 14:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59851 and previous config saved to /var/cache/conftool/dbconfig/20240408-143149-arnaudb.json
  • 14:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp3069.esams.wmnet with OS bullseye
  • 14:20 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:19 sukhe: depool cp3069 to prepare for reimaging: T360430
  • 14:19 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113', diff saved to https://phabricator.wikimedia.org/P59850 and previous config saved to /var/cache/conftool/dbconfig/20240408-141753-arnaudb.json
  • 14:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199', diff saved to https://phabricator.wikimedia.org/P59849 and previous config saved to /var/cache/conftool/dbconfig/20240408-141641-arnaudb.json
  • 14:06 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:04 isaranto@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'experimental' for release 'main' .
  • 14:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59847 and previous config saved to /var/cache/conftool/dbconfig/20240408-140246-arnaudb.json
  • 14:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59846 and previous config saved to /var/cache/conftool/dbconfig/20240408-140132-arnaudb.json
  • 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2113 (T360332)', diff saved to https://phabricator.wikimedia.org/P59845 and previous config saved to /var/cache/conftool/dbconfig/20240408-135926-arnaudb.json
  • 13:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1199 (T360332)', diff saved to https://phabricator.wikimedia.org/P59844 and previous config saved to /var/cache/conftool/dbconfig/20240408-135915-arnaudb.json
  • 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2113.codfw.wmnet with reason: Maintenance
  • 13:59 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1199.eqiad.wmnet with reason: Maintenance
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59843 and previous config saved to /var/cache/conftool/dbconfig/20240408-135852-arnaudb.json
  • 13:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2111.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2101.codfw.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 13:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 13:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59842 and previous config saved to /var/cache/conftool/dbconfig/20240408-135444-arnaudb.json
  • 13:45 Lucas_WMDE: UTC afternoon backport+config window done
  • 13:44 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1207.eqiad.wmnet
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59841 and previous config saved to /var/cache/conftool/dbconfig/20240408-134345-arnaudb.json
  • 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable abusefilter block at bnwiki (T361852) (duration: 28m 46s)
  • 13:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59840 and previous config saved to /var/cache/conftool/dbconfig/20240408-133936-arnaudb.json
  • 13:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1207.eqiad.wmnet
  • 13:35 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1229.eqiad.wmnet
  • 13:33 vgutierrez@cumin1002: END (PASS) - Cookbook sre.cdn.roll-upgrade-haproxy (exit_code=0) rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
  • 13:32 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Continuing with sync
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190', diff saved to https://phabricator.wikimedia.org/P59839 and previous config saved to /var/cache/conftool/dbconfig/20240408-132838-arnaudb.json
  • 13:28 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1229.eqiad.wmnet
  • 13:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1175.eqiad.wmnet
  • 13:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
  • 13:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2212.codfw.wmnet with reason: Silence for clone
  • 13:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59838 and previous config saved to /var/cache/conftool/dbconfig/20240408-132429-arnaudb.json
  • 13:23 vgutierrez@cumin1002: START - Cookbook sre.cdn.roll-upgrade-haproxy rolling upgrade of HAProxy on P{cp[4037,4041,4045,4049].ulsfo.wmnet} and A:cp
  • 13:21 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1175.eqiad.wmnet
  • 13:21 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1241.eqiad.wmnet
  • 13:17 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and yahya: Backport for Enable abusefilter block at bnwiki (T361852) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:15 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1241.eqiad.wmnet
  • 13:15 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 13:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 13:14 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable abusefilter block at bnwiki (T361852)
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59837 and previous config saved to /var/cache/conftool/dbconfig/20240408-131331-arnaudb.json
  • 13:12 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1200.eqiad.wmnet
  • 13:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1190 (T360332)', diff saved to https://phabricator.wikimedia.org/P59836 and previous config saved to /var/cache/conftool/dbconfig/20240408-131113-arnaudb.json
  • 13:11 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 13:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1190.eqiad.wmnet with reason: Maintenance
  • 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59835 and previous config saved to /var/cache/conftool/dbconfig/20240408-131051-arnaudb.json
  • 13:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59834 and previous config saved to /var/cache/conftool/dbconfig/20240408-130921-arnaudb.json
  • 13:05 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1200.eqiad.wmnet
  • 13:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T360332)', diff saved to https://phabricator.wikimedia.org/P59833 and previous config saved to /var/cache/conftool/dbconfig/20240408-130543-arnaudb.json
  • 13:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 13:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59832 and previous config saved to /var/cache/conftool/dbconfig/20240408-130443-arnaudb.json
  • 12:56 elukey: nodetool-a drain + restart of cassandra instances on aqs1010 to pick up the new truststore
  • 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160', diff saved to https://phabricator.wikimedia.org/P59831 and previous config saved to /var/cache/conftool/dbconfig/20240408-125543-arnaudb.json
  • 12:55 elukey@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
  • 12:55 elukey@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on aqs1010.eqiad.wmnet with reason: Replace Java Truststore
  • 12:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1180.eqiad.wmnet
  • 12:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59830 and previous config saved to /var/cache/conftool/dbconfig/20240408-124935-arnaudb.json
  • 12:47 jayme@deploy1002: helmfile [staging-eqiad] DONE helmfile.d/admin 'apply'.
  • 12:46 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1180.eqiad.wmnet
  • 12:46 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1194.eqiad.wmnet
  • 12:46 jayme@deploy1002: helmfile [staging-eqiad] START helmfile.d/admin 'apply'.
  • 12:46 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:44 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:27 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db1237.eqiad.wmnet
  • 12:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59827 and previous config saved to /var/cache/conftool/dbconfig/20240408-122527-arnaudb.json
  • 12:22 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1160 (T360332)', diff saved to https://phabricator.wikimedia.org/P59826 and previous config saved to /var/cache/conftool/dbconfig/20240408-122209-arnaudb.json
  • 12:22 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 12:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1160.eqiad.wmnet with reason: Maintenance
  • 12:19 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59825 and previous config saved to /var/cache/conftool/dbconfig/20240408-121920-arnaudb.json
  • 12:17 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db1237.eqiad.wmnet
  • 12:17 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1022.eqiad.wmnet
  • 12:17 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T360332)', diff saved to https://phabricator.wikimedia.org/P59824 and previous config saved to /var/cache/conftool/dbconfig/20240408-121642-arnaudb.json
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59823 and previous config saved to /var/cache/conftool/dbconfig/20240408-121609-arnaudb.json
  • 12:15 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:14 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:11 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:09 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1022.eqiad.wmnet
  • 12:07 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1031.eqiad.wmnet
  • 12:04 jayme@deploy1002: helmfile [staging-codfw] DONE helmfile.d/admin 'apply'.
  • 12:02 btullis@cumin1002: END (FAIL) - Cookbook sre.hosts.reboot-single (exit_code=99) for host stat1011.eqiad.wmnet
  • 12:02 btullis@cumin1002: START - Cookbook sre.hosts.reboot-single for host stat1011.eqiad.wmnet
  • 12:01 jayme@deploy1002: helmfile [staging-codfw] START helmfile.d/admin 'apply'.
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59822 and previous config saved to /var/cache/conftool/dbconfig/20240408-120101-arnaudb.json
  • 11:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1031.eqiad.wmnet
  • 11:57 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1030.eqiad.wmnet
  • 11:49 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1030.eqiad.wmnet
  • 11:49 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host es1029.eqiad.wmnet
  • 11:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59821 and previous config saved to /var/cache/conftool/dbconfig/20240408-114552-arnaudb.json
  • 11:42 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host es1029.eqiad.wmnet
  • 11:35 moritzm: installing glibc security updates on bullseye
  • 11:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59820 and previous config saved to /var/cache/conftool/dbconfig/20240408-113045-arnaudb.json
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T360332)', diff saved to https://phabricator.wikimedia.org/P59819 and previous config saved to /var/cache/conftool/dbconfig/20240408-112807-arnaudb.json
  • 11:28 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 11:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59818 and previous config saved to /var/cache/conftool/dbconfig/20240408-112744-arnaudb.json
  • 11:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59817 and previous config saved to /var/cache/conftool/dbconfig/20240408-112052-arnaudb.json
  • 11:13 hnowlan@deploy1002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59816 and previous config saved to /var/cache/conftool/dbconfig/20240408-111236-arnaudb.json
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:12 hnowlan@deploy1002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:11 hnowlan@deploy1002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:10 hnowlan@deploy1002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 11:09 hnowlan@deploy1002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 11:09 hnowlan@deploy1002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 11:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59815 and previous config saved to /var/cache/conftool/dbconfig/20240408-110545-arnaudb.json
  • 11:03 btullis: started manual wikidata dump on snapshot1009 for T252396
  • 10:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59814 and previous config saved to /var/cache/conftool/dbconfig/20240408-105729-arnaudb.json
  • 10:51 Dreamy_Jazz: Starting scan on dewiki for MediaModeration to catch-up on monthly limits - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 10:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59813 and previous config saved to /var/cache/conftool/dbconfig/20240408-105036-arnaudb.json
  • 10:49 moritzm: installing postgresql-13 security updates
  • 10:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59812 and previous config saved to /var/cache/conftool/dbconfig/20240408-104221-arnaudb.json
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T360332)', diff saved to https://phabricator.wikimedia.org/P59811 and previous config saved to /var/cache/conftool/dbconfig/20240408-103945-arnaudb.json
  • 10:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 10:39 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 10:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59810 and previous config saved to /var/cache/conftool/dbconfig/20240408-103922-arnaudb.json
  • 10:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59809 and previous config saved to /var/cache/conftool/dbconfig/20240408-103529-arnaudb.json
  • 10:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59808 and previous config saved to /var/cache/conftool/dbconfig/20240408-103313-arnaudb.json
  • 10:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59807 and previous config saved to /var/cache/conftool/dbconfig/20240408-103249-arnaudb.json
  • 10:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59806 and previous config saved to /var/cache/conftool/dbconfig/20240408-102414-arnaudb.json
  • 10:24 Dreamy_Jazz: Starting MediaModeration scanning script (stopped over the weekend due to server instability) - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 10:18 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-host (exit_code=0) for host db2128.codfw.wmnet
  • 10:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59805 and previous config saved to /var/cache/conftool/dbconfig/20240408-101741-arnaudb.json
  • 10:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59804 and previous config saved to /var/cache/conftool/dbconfig/20240408-100906-arnaudb.json
  • 10:07 jmm@cumin2002: START - Cookbook sre.puppet.migrate-host for host db2128.codfw.wmnet
  • 10:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59803 and previous config saved to /var/cache/conftool/dbconfig/20240408-100233-arnaudb.json
  • 09:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59802 and previous config saved to /var/cache/conftool/dbconfig/20240408-095359-arnaudb.json
  • 09:53 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::proxy::master
  • 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T360332)', diff saved to https://phabricator.wikimedia.org/P59801 and previous config saved to /var/cache/conftool/dbconfig/20240408-095123-arnaudb.json
  • 09:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 09:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59800 and previous config saved to /var/cache/conftool/dbconfig/20240408-095100-arnaudb.json
  • 09:49 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 03m 49s)
  • 09:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59799 and previous config saved to /var/cache/conftool/dbconfig/20240408-094726-arnaudb.json
  • 09:46 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
  • 09:45 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/blubberoid: apply
  • 09:45 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/blubberoid: apply
  • 09:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59798 and previous config saved to /var/cache/conftool/dbconfig/20240408-094510-arnaudb.json
  • 09:45 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:45 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/blubberoid: apply
  • 09:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59797 and previous config saved to /var/cache/conftool/dbconfig/20240408-094447-arnaudb.json
  • 09:44 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/blubberoid: apply
  • 09:44 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/services/apertium: apply
  • 09:44 jayme@deploy1002: helmfile [eqiad] START helmfile.d/services/apertium: apply
  • 09:43 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/services/apertium: apply
  • 09:43 jayme@deploy1002: helmfile [codfw] START helmfile.d/services/apertium: apply
  • 09:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 100%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59796 and previous config saved to /var/cache/conftool/dbconfig/20240408-094102-arnaudb.json
  • 09:37 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::proxy::master
  • 09:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59795 and previous config saved to /var/cache/conftool/dbconfig/20240408-093552-arnaudb.json
  • 09:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59794 and previous config saved to /var/cache/conftool/dbconfig/20240408-092939-arnaudb.json
  • 09:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 75%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59793 and previous config saved to /var/cache/conftool/dbconfig/20240408-092557-arnaudb.json
  • 09:22 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/apertium: apply
  • 09:21 jayme@deploy1002: helmfile [staging] START helmfile.d/services/apertium: apply
  • 09:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59792 and previous config saved to /var/cache/conftool/dbconfig/20240408-092045-arnaudb.json
  • 09:20 jayme@deploy1002: helmfile [staging] DONE helmfile.d/services/blubberoid: apply
  • 09:17 jayme@deploy1002: helmfile [staging] START helmfile.d/services/blubberoid: apply
  • 09:17 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 09:16 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 09:14 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 15830
  • 09:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59791 and previous config saved to /var/cache/conftool/dbconfig/20240408-091432-arnaudb.json
  • 09:10 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 15830
  • 09:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 50%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59790 and previous config saved to /var/cache/conftool/dbconfig/20240408-091051-arnaudb.json
  • 09:06 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::sanitarium_multiinstance
  • 09:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59789 and previous config saved to /var/cache/conftool/dbconfig/20240408-090535-arnaudb.json
  • 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59788 and previous config saved to /var/cache/conftool/dbconfig/20240408-090258-arnaudb.json
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 09:02 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 08:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59787 and previous config saved to /var/cache/conftool/dbconfig/20240408-085924-arnaudb.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 08:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59786 and previous config saved to /var/cache/conftool/dbconfig/20240408-085708-arnaudb.json
  • 08:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 08:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 08:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2114 (re)pooling @ 25%: Post clone (src)', diff saved to https://phabricator.wikimedia.org/P59785 and previous config saved to /var/cache/conftool/dbconfig/20240408-085545-arnaudb.json
  • 08:44 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::sanitarium_multiinstance
  • 08:41 godog: grafana upgrade to 9.5.18 - T361830
  • 08:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
  • 08:29 dcausse: restarting blazegraph on wdqs1020 (BlazegraphFreeAllocatorsDecreasingRapidly)
  • 08:26 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset: apply
  • 08:25 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset: apply
  • 08:24 kartik@deploy1002: Finished scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607) (duration: 15m 47s)
  • 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/dse-k8s-services/superset-next: apply
  • 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/dse-k8s-services/superset-next: apply
  • 08:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Bump db2112 weight T361786', diff saved to https://phabricator.wikimedia.org/P59784 and previous config saved to /var/cache/conftool/dbconfig/20240408-081320-arnaudb.json
  • 08:12 kartik@deploy1002: kartik: Continuing with sync
  • 08:12 volans: restarted stashbot that had died few minutes ago
  • 08:09 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2203 to s1 primary T361786', diff saved to https://phabricator.wikimedia.org/P59783 and previous config saved to /var/cache/conftool/dbconfig/20240408-080910-arnaudb.json
  • 08:08 arnaudb: Starting s1 codfw failover from db2112 to db2203 - T361786
  • 08:08 kartik@deploy1002: Started scap: Backport for Enable the unified dashboard on the test instance for all languages (T360607)
  • 07:57 filippo@deploy1002: helmfile [aux-k8s-eqiad] DONE helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 07:57 jayme@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 07:56 filippo@deploy1002: helmfile [aux-k8s-eqiad] START helmfile.d/aus-k8s-eqiad-services/jaeger: apply
  • 07:56 jayme@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 07:56 jayme@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 07:55 jayme@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 07:48 kartik@deploy1002: Finished scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) (duration: 35m 43s)
  • 07:47 moritzm: installing util-linux security updates on bullseye/bookworm
  • 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59782 and previous config saved to /var/cache/conftool/dbconfig/20240408-074448-root.json
  • 07:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2203 with weight 0 T361786', diff saved to https://phabricator.wikimedia.org/P59781 and previous config saved to /var/cache/conftool/dbconfig/20240408-074006-arnaudb.json
  • 07:39 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
  • 07:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 37 hosts with reason: Primary switchover s1 T361786
  • 07:35 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2114.codfw.wmnet onto db2214.codfw.wmnet
  • 07:35 kartik@deploy1002: kartik and ihurbain: Continuing with sync
  • 07:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2114 in db2214 for T355422', diff saved to https://phabricator.wikimedia.org/P59780 and previous config saved to /var/cache/conftool/dbconfig/20240408-073239-arnaudb.json
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2214.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2114.codfw.wmnet with reason: provisionning db2214.codfw.wmnet - T355422
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59779 and previous config saved to /var/cache/conftool/dbconfig/20240408-072942-root.json
  • 07:25 kartik@deploy1002: kartik and ihurbain: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59778 and previous config saved to /var/cache/conftool/dbconfig/20240408-071436-root.json
  • 07:12 kartik@deploy1002: Started scap: Backport for Add Kartographer Parsoid support to hewikivoyage (T342871 T361025)
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59777 and previous config saved to /var/cache/conftool/dbconfig/20240408-065931-root.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59776 and previous config saved to /var/cache/conftool/dbconfig/20240408-064424-root.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59775 and previous config saved to /var/cache/conftool/dbconfig/20240408-062919-root.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1156 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59774 and previous config saved to /var/cache/conftool/dbconfig/20240408-061413-root.json
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1156', diff saved to https://phabricator.wikimedia.org/P59773 and previous config saved to /var/cache/conftool/dbconfig/20240408-060554-root.json
  • 04:02 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog2002
  • 04:01 denisse: Cleaning Prometheus and Thanos-BE log gzips older than 45 days on centrallog1002

2024-04-06

  • 15:33 jhathaway@cumin2002: END (PASS) - Cookbook sre.ganeti.reboot-vm (exit_code=0) for VM mx-out2001.wikimedia.org
  • 15:33 jhathaway@cumin2002: START - Cookbook sre.ganeti.reboot-vm for VM mx-out2001.wikimedia.org
  • 03:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 03:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 03:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59763 and previous config saved to /var/cache/conftool/dbconfig/20240406-034152-arnaudb.json
  • 03:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59762 and previous config saved to /var/cache/conftool/dbconfig/20240406-032644-arnaudb.json
  • 03:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195', diff saved to https://phabricator.wikimedia.org/P59761 and previous config saved to /var/cache/conftool/dbconfig/20240406-031136-arnaudb.json
  • 02:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59760 and previous config saved to /var/cache/conftool/dbconfig/20240406-025629-arnaudb.json
  • 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2195 (T360332)', diff saved to https://phabricator.wikimedia.org/P59759 and previous config saved to /var/cache/conftool/dbconfig/20240406-025411-arnaudb.json
  • 02:54 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 02:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2195.codfw.wmnet with reason: Maintenance
  • 02:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59758 and previous config saved to /var/cache/conftool/dbconfig/20240406-025348-arnaudb.json
  • 02:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59757 and previous config saved to /var/cache/conftool/dbconfig/20240406-023841-arnaudb.json
  • 02:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181', diff saved to https://phabricator.wikimedia.org/P59756 and previous config saved to /var/cache/conftool/dbconfig/20240406-022333-arnaudb.json
  • 02:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59755 and previous config saved to /var/cache/conftool/dbconfig/20240406-020826-arnaudb.json
  • 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2181 (T360332)', diff saved to https://phabricator.wikimedia.org/P59754 and previous config saved to /var/cache/conftool/dbconfig/20240406-020608-arnaudb.json
  • 02:06 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 02:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2181.codfw.wmnet with reason: Maintenance
  • 02:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59753 and previous config saved to /var/cache/conftool/dbconfig/20240406-020545-arnaudb.json
  • 01:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59752 and previous config saved to /var/cache/conftool/dbconfig/20240406-015037-arnaudb.json
  • 01:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167', diff saved to https://phabricator.wikimedia.org/P59751 and previous config saved to /var/cache/conftool/dbconfig/20240406-013530-arnaudb.json
  • 01:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59750 and previous config saved to /var/cache/conftool/dbconfig/20240406-012021-arnaudb.json
  • 01:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59749 and previous config saved to /var/cache/conftool/dbconfig/20240406-011803-arnaudb.json
  • 01:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 01:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2167.codfw.wmnet with reason: Maintenance
  • 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59748 and previous config saved to /var/cache/conftool/dbconfig/20240406-011740-arnaudb.json
  • 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59747 and previous config saved to /var/cache/conftool/dbconfig/20240406-010231-arnaudb.json
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166', diff saved to https://phabricator.wikimedia.org/P59746 and previous config saved to /var/cache/conftool/dbconfig/20240406-004724-arnaudb.json
  • 00:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59745 and previous config saved to /var/cache/conftool/dbconfig/20240406-003216-arnaudb.json
  • 00:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2166 (T360332)', diff saved to https://phabricator.wikimedia.org/P59744 and previous config saved to /var/cache/conftool/dbconfig/20240406-002958-arnaudb.json
  • 00:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 00:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2166.codfw.wmnet with reason: Maintenance
  • 00:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59743 and previous config saved to /var/cache/conftool/dbconfig/20240406-002935-arnaudb.json
  • 00:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59742 and previous config saved to /var/cache/conftool/dbconfig/20240406-001428-arnaudb.json

2024-04-05

  • 23:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164', diff saved to https://phabricator.wikimedia.org/P59741 and previous config saved to /var/cache/conftool/dbconfig/20240405-235920-arnaudb.json
  • 23:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59740 and previous config saved to /var/cache/conftool/dbconfig/20240405-234413-arnaudb.json
  • 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2164 (T360332)', diff saved to https://phabricator.wikimedia.org/P59739 and previous config saved to /var/cache/conftool/dbconfig/20240405-234156-arnaudb.json
  • 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2186.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2164.codfw.wmnet with reason: Maintenance
  • 23:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59738 and previous config saved to /var/cache/conftool/dbconfig/20240405-234117-arnaudb.json
  • 23:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59737 and previous config saved to /var/cache/conftool/dbconfig/20240405-232609-arnaudb.json
  • 23:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163', diff saved to https://phabricator.wikimedia.org/P59736 and previous config saved to /var/cache/conftool/dbconfig/20240405-231102-arnaudb.json
  • 22:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59735 and previous config saved to /var/cache/conftool/dbconfig/20240405-225554-arnaudb.json
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2163 (T360332)', diff saved to https://phabricator.wikimedia.org/P59734 and previous config saved to /var/cache/conftool/dbconfig/20240405-225336-arnaudb.json
  • 22:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 22:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2163.codfw.wmnet with reason: Maintenance
  • 22:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59733 and previous config saved to /var/cache/conftool/dbconfig/20240405-225313-arnaudb.json
  • 22:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59732 and previous config saved to /var/cache/conftool/dbconfig/20240405-223806-arnaudb.json
  • 22:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162', diff saved to https://phabricator.wikimedia.org/P59731 and previous config saved to /var/cache/conftool/dbconfig/20240405-222259-arnaudb.json
  • 22:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59730 and previous config saved to /var/cache/conftool/dbconfig/20240405-220751-arnaudb.json
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2162 (T360332)', diff saved to https://phabricator.wikimedia.org/P59729 and previous config saved to /var/cache/conftool/dbconfig/20240405-220533-arnaudb.json
  • 22:05 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 22:05 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2162.codfw.wmnet with reason: Maintenance
  • 22:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59728 and previous config saved to /var/cache/conftool/dbconfig/20240405-220510-arnaudb.json
  • 21:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59727 and previous config saved to /var/cache/conftool/dbconfig/20240405-215001-arnaudb.json
  • 21:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161', diff saved to https://phabricator.wikimedia.org/P59725 and previous config saved to /var/cache/conftool/dbconfig/20240405-213454-arnaudb.json
  • 21:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59724 and previous config saved to /var/cache/conftool/dbconfig/20240405-211946-arnaudb.json
  • 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2161 (T360332)', diff saved to https://phabricator.wikimedia.org/P59723 and previous config saved to /var/cache/conftool/dbconfig/20240405-211728-arnaudb.json
  • 21:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 21:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2161.codfw.wmnet with reason: Maintenance
  • 21:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59722 and previous config saved to /var/cache/conftool/dbconfig/20240405-211705-arnaudb.json
  • 21:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59721 and previous config saved to /var/cache/conftool/dbconfig/20240405-210157-arnaudb.json
  • 20:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154', diff saved to https://phabricator.wikimedia.org/P59720 and previous config saved to /var/cache/conftool/dbconfig/20240405-204650-arnaudb.json
  • 20:40 jhathaway@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
  • 20:37 jhathaway@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out2001.wikimedia.org with reason: host reimage
  • 20:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59719 and previous config saved to /var/cache/conftool/dbconfig/20240405-203143-arnaudb.json
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2154 (T360332)', diff saved to https://phabricator.wikimedia.org/P59718 and previous config saved to /var/cache/conftool/dbconfig/20240405-202925-arnaudb.json
  • 20:29 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 20:29 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2154.codfw.wmnet with reason: Maintenance
  • 20:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59717 and previous config saved to /var/cache/conftool/dbconfig/20240405-202901-arnaudb.json
  • 20:20 jhathaway@cumin2002: START - Cookbook sre.hosts.reimage for host mx-out2001.wikimedia.org with OS bookworm
  • 20:19 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:18 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out2001.wikimedia.org on all recursors
  • 20:18 jhathaway@cumin2002: START - Cookbook sre.dns.wipe-cache mx-out2001.wikimedia.org on all recursors
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 20:18 jhathaway@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:16 jhathaway@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out2001.wikimedia.org - jhathaway@cumin2002"
  • 20:15 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host mx-out1001.wikimedia.org with OS bookworm
  • 20:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59716 and previous config saved to /var/cache/conftool/dbconfig/20240405-201354-arnaudb.json
  • 20:13 jhathaway@cumin2002: START - Cookbook sre.dns.netbox
  • 20:13 jhathaway@cumin2002: START - Cookbook sre.ganeti.makevm for new host mx-out2001.wikimedia.org
  • 20:02 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 19:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152', diff saved to https://phabricator.wikimedia.org/P59715 and previous config saved to /var/cache/conftool/dbconfig/20240405-195847-arnaudb.json
  • 19:57 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 19:45 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
  • 19:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59714 and previous config saved to /var/cache/conftool/dbconfig/20240405-194339-arnaudb.json
  • 19:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2152 (T360332)', diff saved to https://phabricator.wikimedia.org/P59713 and previous config saved to /var/cache/conftool/dbconfig/20240405-194221-arnaudb.json
  • 19:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 19:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2152.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 19:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 19:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59712 and previous config saved to /var/cache/conftool/dbconfig/20240405-194057-arnaudb.json
  • 19:40 jhathaway@cumin1002: END (FAIL) - Cookbook sre.ganeti.makevm (exit_code=97) for new host mx-out1001.wikimedia.org
  • 19:40 jhathaway@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host mx-out1001.wikimedia.org with OS bookworm
  • 19:27 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
  • 19:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db1246.eqiad.wmnet with reason: Host down
  • 19:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59711 and previous config saved to /var/cache/conftool/dbconfig/20240405-192549-arnaudb.json
  • 19:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59710 and previous config saved to /var/cache/conftool/dbconfig/20240405-191042-arnaudb.json
  • 19:02 mutante: codesearch - puppet trying to restart hound-search after deploying gerrit:1017179 and gerrit:1016480
  • 18:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59709 and previous config saved to /var/cache/conftool/dbconfig/20240405-185533-arnaudb.json
  • 18:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T360332)', diff saved to https://phabricator.wikimedia.org/P59708 and previous config saved to /var/cache/conftool/dbconfig/20240405-185216-arnaudb.json
  • 18:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 18:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59707 and previous config saved to /var/cache/conftool/dbconfig/20240405-185131-arnaudb.json
  • 18:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59706 and previous config saved to /var/cache/conftool/dbconfig/20240405-183623-arnaudb.json
  • 18:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59705 and previous config saved to /var/cache/conftool/dbconfig/20240405-182115-arnaudb.json
  • 18:13 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.remove-downtime (exit_code=0) for db1214.eqiad.wmnet
  • 18:13 sukhe@cumin2002: START - Cookbook sre.hosts.remove-downtime for db1214.eqiad.wmnet
  • 18:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59704 and previous config saved to /var/cache/conftool/dbconfig/20240405-180608-arnaudb.json
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59703 and previous config saved to /var/cache/conftool/dbconfig/20240405-180352-arnaudb.json
  • 18:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59702 and previous config saved to /var/cache/conftool/dbconfig/20240405-180330-arnaudb.json
  • 18:03 dzahn@cumin2002: dbctl commit (dc=all): 'depool db1246', diff saved to https://phabricator.wikimedia.org/P59701 and previous config saved to /var/cache/conftool/dbconfig/20240405-180319-dzahn.json
  • 18:01 mutante: depooling db1246 which went down and paged
  • 17:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59700 and previous config saved to /var/cache/conftool/dbconfig/20240405-174735-arnaudb.json
  • 17:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59699 and previous config saved to /var/cache/conftool/dbconfig/20240405-173227-arnaudb.json
  • 17:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59698 and previous config saved to /var/cache/conftool/dbconfig/20240405-171719-arnaudb.json
  • 17:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T360332)', diff saved to https://phabricator.wikimedia.org/P59697 and previous config saved to /var/cache/conftool/dbconfig/20240405-171502-arnaudb.json
  • 17:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:14 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59696 and previous config saved to /var/cache/conftool/dbconfig/20240405-171439-arnaudb.json
  • 17:14 jhathaway@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 17:11 jhathaway@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on mx-out1001.wikimedia.org with reason: host reimage
  • 16:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59695 and previous config saved to /var/cache/conftool/dbconfig/20240405-165931-arnaudb.json
  • 16:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59694 and previous config saved to /var/cache/conftool/dbconfig/20240405-164424-arnaudb.json
  • 16:29 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59693 and previous config saved to /var/cache/conftool/dbconfig/20240405-162916-arnaudb.json
  • 16:27 jhathaway@cumin1002: START - Cookbook sre.hosts.reimage for host mx-out1001.wikimedia.org with OS bookworm
  • 16:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T360332)', diff saved to https://phabricator.wikimedia.org/P59692 and previous config saved to /var/cache/conftool/dbconfig/20240405-162700-arnaudb.json
  • 16:26 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 16:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59691 and previous config saved to /var/cache/conftool/dbconfig/20240405-162637-arnaudb.json
  • 16:25 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:24 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.ganeti.makevm: created new VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.wipe-cache (exit_code=0) mx-out1001.wikimedia.org on all recursors
  • 16:24 jhathaway@cumin1002: START - Cookbook sre.dns.wipe-cache mx-out1001.wikimedia.org on all recursors
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 16:24 jhathaway@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:18 bking@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:18 bking@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59689 and previous config saved to /var/cache/conftool/dbconfig/20240405-161130-arnaudb.json
  • 16:07 jhathaway@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: Add records for VM mx-out1001.wikimedia.org - jhathaway@cumin1002"
  • 16:03 jhathaway@cumin1002: START - Cookbook sre.dns.netbox
  • 16:03 jhathaway@cumin1002: START - Cookbook sre.ganeti.makevm for new host mx-out1001.wikimedia.org
  • 15:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59688 and previous config saved to /var/cache/conftool/dbconfig/20240405-155622-arnaudb.json
  • 15:45 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 15:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59687 and previous config saved to /var/cache/conftool/dbconfig/20240405-154115-arnaudb.json
  • 15:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59686 and previous config saved to /var/cache/conftool/dbconfig/20240405-153759-arnaudb.json
  • 15:37 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 15:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59685 and previous config saved to /var/cache/conftool/dbconfig/20240405-153736-arnaudb.json
  • 15:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59684 and previous config saved to /var/cache/conftool/dbconfig/20240405-152228-arnaudb.json
  • 15:20 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 15:17 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 15:14 aikochou@deploy1002: helmfile [ml-serve-eqiad] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 15:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59683 and previous config saved to /var/cache/conftool/dbconfig/20240405-150721-arnaudb.json
  • 15:03 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 14:56 dancy@deploy1002: Installation of scap version "4.75.0" completed for 353 hosts
  • 14:55 dancy@deploy1002: Installing scap version "4.75.0" for 353 hosts
  • 14:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59682 and previous config saved to /var/cache/conftool/dbconfig/20240405-145213-arnaudb.json
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T360332)', diff saved to https://phabricator.wikimedia.org/P59681 and previous config saved to /var/cache/conftool/dbconfig/20240405-144957-arnaudb.json
  • 14:49 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 14:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 14:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59680 and previous config saved to /var/cache/conftool/dbconfig/20240405-144934-arnaudb.json
  • 14:36 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1016.eqiad.wmnet
  • 14:36 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1016.eqiad.wmnet
  • 14:35 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1015.eqiad.wmnet
  • 14:35 aikochou@deploy1002: helmfile [ml-serve-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 14:34 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1015.eqiad.wmnet
  • 14:34 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1014.eqiad.wmnet
  • 14:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59679 and previous config saved to /var/cache/conftool/dbconfig/20240405-143427-arnaudb.json
  • 14:33 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1014.eqiad.wmnet
  • 14:33 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1013.eqiad.wmnet
  • 14:29 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1013.eqiad.wmnet
  • 14:26 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1020.eqiad.wmnet
  • 14:25 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1020.eqiad.wmnet
  • 14:25 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1019.eqiad.wmnet
  • 14:22 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1019.eqiad.wmnet
  • 14:22 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1018.eqiad.wmnet
  • 14:20 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1018.eqiad.wmnet
  • 14:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59678 and previous config saved to /var/cache/conftool/dbconfig/20240405-141919-arnaudb.json
  • 14:18 taavi@cumin1002: conftool action : set/pooled=yes; selector: name=clouddb1017.eqiad.wmnet
  • 14:15 taavi@cumin1002: conftool action : set/pooled=no; selector: name=clouddb1017.eqiad.wmnet
  • 14:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59677 and previous config saved to /var/cache/conftool/dbconfig/20240405-140412-arnaudb.json
  • 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T360332)', diff saved to https://phabricator.wikimedia.org/P59676 and previous config saved to /var/cache/conftool/dbconfig/20240405-130347-arnaudb.json
  • 13:03 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 13:03 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 13:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59675 and previous config saved to /var/cache/conftool/dbconfig/20240405-130324-arnaudb.json
  • 12:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59674 and previous config saved to /var/cache/conftool/dbconfig/20240405-124816-arnaudb.json
  • 12:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59673 and previous config saved to /var/cache/conftool/dbconfig/20240405-123309-arnaudb.json
  • 12:32 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp4037.ulsfo.wmnet
  • 12:32 fabfur: repool cp4037 (T361845)
  • 12:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59672 and previous config saved to /var/cache/conftool/dbconfig/20240405-121801-arnaudb.json
  • 12:10 ayounsi@cumin1002: END (FAIL) - Cookbook sre.hosts.decommission (exit_code=1) for hosts testvm2006.codfw.wmnet
  • 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 12:10 ayounsi@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T360332)', diff saved to https://phabricator.wikimedia.org/P59671 and previous config saved to /var/cache/conftool/dbconfig/20240405-111736-arnaudb.json
  • 11:17 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 11:17 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 11:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59670 and previous config saved to /var/cache/conftool/dbconfig/20240405-111713-arnaudb.json
  • 11:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59669 and previous config saved to /var/cache/conftool/dbconfig/20240405-110204-arnaudb.json
  • 10:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59668 and previous config saved to /var/cache/conftool/dbconfig/20240405-104657-arnaudb.json
  • 10:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59667 and previous config saved to /var/cache/conftool/dbconfig/20240405-103149-arnaudb.json
  • 09:51 ayounsi@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: testvm2006.codfw.wmnet decommissioned, removing all IPs except the asset tag one - ayounsi@cumin1002"
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 09:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 09:37 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1239.eqiad.wmnet with reason: Maintenance
  • 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59666 and previous config saved to /var/cache/conftool/dbconfig/20240405-093745-marostegui.json
  • 09:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T360332)', diff saved to https://phabricator.wikimedia.org/P59665 and previous config saved to /var/cache/conftool/dbconfig/20240405-093124-arnaudb.json
  • 09:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 09:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 09:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59664 and previous config saved to /var/cache/conftool/dbconfig/20240405-093038-arnaudb.json
  • 09:24 ayounsi@cumin1002: START - Cookbook sre.dns.netbox
  • 09:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59663 and previous config saved to /var/cache/conftool/dbconfig/20240405-092237-marostegui.json
  • 09:19 ayounsi@cumin1002: START - Cookbook sre.hosts.decommission for hosts testvm2006.codfw.wmnet
  • 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59662 and previous config saved to /var/cache/conftool/dbconfig/20240405-091531-arnaudb.json
  • 09:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235', diff saved to https://phabricator.wikimedia.org/P59661 and previous config saved to /var/cache/conftool/dbconfig/20240405-090730-marostegui.json
  • 09:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59660 and previous config saved to /var/cache/conftool/dbconfig/20240405-090023-arnaudb.json
  • 08:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1235 (T356166)', diff saved to https://phabricator.wikimedia.org/P59659 and previous config saved to /var/cache/conftool/dbconfig/20240405-085222-marostegui.json
  • 08:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59658 and previous config saved to /var/cache/conftool/dbconfig/20240405-084515-arnaudb.json
  • 07:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59655 and previous config saved to /var/cache/conftool/dbconfig/20240405-075831-marostegui.json
  • 07:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T360332)', diff saved to https://phabricator.wikimedia.org/P59654 and previous config saved to /var/cache/conftool/dbconfig/20240405-075646-arnaudb.json
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 07:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234', diff saved to https://phabricator.wikimedia.org/P59653 and previous config saved to /var/cache/conftool/dbconfig/20240405-074323-marostegui.json
  • 07:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59652 and previous config saved to /var/cache/conftool/dbconfig/20240405-072816-marostegui.json
  • 06:35 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 06:35 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1234 (T356166)', diff saved to https://phabricator.wikimedia.org/P59651 and previous config saved to /var/cache/conftool/dbconfig/20240405-044048-marostegui.json
  • 04:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 04:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1234.eqiad.wmnet with reason: Maintenance
  • 04:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59650 and previous config saved to /var/cache/conftool/dbconfig/20240405-044025-marostegui.json
  • 04:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59649 and previous config saved to /var/cache/conftool/dbconfig/20240405-042517-marostegui.json
  • 04:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232', diff saved to https://phabricator.wikimedia.org/P59648 and previous config saved to /var/cache/conftool/dbconfig/20240405-041010-marostegui.json
  • 03:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59647 and previous config saved to /var/cache/conftool/dbconfig/20240405-035829-arnaudb.json
  • 03:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59646 and previous config saved to /var/cache/conftool/dbconfig/20240405-035503-marostegui.json
  • 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1232 (T356166)', diff saved to https://phabricator.wikimedia.org/P59645 and previous config saved to /var/cache/conftool/dbconfig/20240405-035353-marostegui.json
  • 03:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 03:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1232.eqiad.wmnet with reason: Maintenance
  • 03:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59644 and previous config saved to /var/cache/conftool/dbconfig/20240405-035331-marostegui.json
  • 03:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59643 and previous config saved to /var/cache/conftool/dbconfig/20240405-034322-arnaudb.json
  • 03:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59642 and previous config saved to /var/cache/conftool/dbconfig/20240405-033823-marostegui.json
  • 03:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220', diff saved to https://phabricator.wikimedia.org/P59641 and previous config saved to /var/cache/conftool/dbconfig/20240405-032814-arnaudb.json
  • 03:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228', diff saved to https://phabricator.wikimedia.org/P59640 and previous config saved to /var/cache/conftool/dbconfig/20240405-032316-marostegui.json
  • 03:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59639 and previous config saved to /var/cache/conftool/dbconfig/20240405-031307-arnaudb.json
  • 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2220 (T360332)', diff saved to https://phabricator.wikimedia.org/P59638 and previous config saved to /var/cache/conftool/dbconfig/20240405-031028-arnaudb.json
  • 03:10 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
  • 03:10 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2220.codfw.wmnet with reason: Maintenance
  • 03:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59637 and previous config saved to /var/cache/conftool/dbconfig/20240405-031005-arnaudb.json
  • 03:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59636 and previous config saved to /var/cache/conftool/dbconfig/20240405-030809-marostegui.json
  • 02:54 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59635 and previous config saved to /var/cache/conftool/dbconfig/20240405-025458-arnaudb.json
  • 02:39 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218', diff saved to https://phabricator.wikimedia.org/P59634 and previous config saved to /var/cache/conftool/dbconfig/20240405-023949-arnaudb.json
  • 02:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59633 and previous config saved to /var/cache/conftool/dbconfig/20240405-022442-arnaudb.json
  • 02:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2218 (T360332)', diff saved to https://phabricator.wikimedia.org/P59632 and previous config saved to /var/cache/conftool/dbconfig/20240405-022201-arnaudb.json
  • 02:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
  • 02:21 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2218.codfw.wmnet with reason: Maintenance
  • 02:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59631 and previous config saved to /var/cache/conftool/dbconfig/20240405-022138-arnaudb.json
  • 02:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59630 and previous config saved to /var/cache/conftool/dbconfig/20240405-020630-arnaudb.json
  • 01:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208', diff saved to https://phabricator.wikimedia.org/P59629 and previous config saved to /var/cache/conftool/dbconfig/20240405-015123-arnaudb.json
  • 01:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59628 and previous config saved to /var/cache/conftool/dbconfig/20240405-013615-arnaudb.json
  • 01:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2208 (T360332)', diff saved to https://phabricator.wikimedia.org/P59627 and previous config saved to /var/cache/conftool/dbconfig/20240405-013336-arnaudb.json
  • 01:33 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
  • 01:33 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2208.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2198.codfw.wmnet with reason: Maintenance
  • 01:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59626 and previous config saved to /var/cache/conftool/dbconfig/20240405-013227-arnaudb.json
  • 01:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59625 and previous config saved to /var/cache/conftool/dbconfig/20240405-011720-arnaudb.json
  • 01:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182', diff saved to https://phabricator.wikimedia.org/P59623 and previous config saved to /var/cache/conftool/dbconfig/20240405-010212-arnaudb.json
  • 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1228 (T356166)', diff saved to https://phabricator.wikimedia.org/P59622 and previous config saved to /var/cache/conftool/dbconfig/20240405-005341-marostegui.json
  • 00:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 00:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1228.eqiad.wmnet with reason: Maintenance
  • 00:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59621 and previous config saved to /var/cache/conftool/dbconfig/20240405-005318-marostegui.json
  • 00:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59620 and previous config saved to /var/cache/conftool/dbconfig/20240405-004705-arnaudb.json
  • 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2182 (T360332)', diff saved to https://phabricator.wikimedia.org/P59619 and previous config saved to /var/cache/conftool/dbconfig/20240405-004428-arnaudb.json
  • 00:44 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 00:44 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2182.codfw.wmnet with reason: Maintenance
  • 00:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59618 and previous config saved to /var/cache/conftool/dbconfig/20240405-004405-arnaudb.json
  • 00:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59617 and previous config saved to /var/cache/conftool/dbconfig/20240405-003810-marostegui.json
  • 00:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59616 and previous config saved to /var/cache/conftool/dbconfig/20240405-002857-arnaudb.json
  • 00:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219', diff saved to https://phabricator.wikimedia.org/P59615 and previous config saved to /var/cache/conftool/dbconfig/20240405-002303-marostegui.json
  • 00:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168', diff saved to https://phabricator.wikimedia.org/P59614 and previous config saved to /var/cache/conftool/dbconfig/20240405-001350-arnaudb.json
  • 00:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59613 and previous config saved to /var/cache/conftool/dbconfig/20240405-000755-marostegui.json

2024-04-04

  • 23:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59612 and previous config saved to /var/cache/conftool/dbconfig/20240404-235843-arnaudb.json
  • 23:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59611 and previous config saved to /var/cache/conftool/dbconfig/20240404-235606-arnaudb.json
  • 23:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2168.codfw.wmnet with reason: Maintenance
  • 23:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59610 and previous config saved to /var/cache/conftool/dbconfig/20240404-235543-arnaudb.json
  • 23:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59609 and previous config saved to /var/cache/conftool/dbconfig/20240404-234035-arnaudb.json
  • 23:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159', diff saved to https://phabricator.wikimedia.org/P59608 and previous config saved to /var/cache/conftool/dbconfig/20240404-232528-arnaudb.json
  • 23:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59606 and previous config saved to /var/cache/conftool/dbconfig/20240404-231020-arnaudb.json
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2159 (T360332)', diff saved to https://phabricator.wikimedia.org/P59605 and previous config saved to /var/cache/conftool/dbconfig/20240404-230743-arnaudb.json
  • 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2159.codfw.wmnet with reason: Maintenance
  • 23:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59604 and previous config saved to /var/cache/conftool/dbconfig/20240404-230704-arnaudb.json
  • 22:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59603 and previous config saved to /var/cache/conftool/dbconfig/20240404-225156-arnaudb.json
  • 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59602 and previous config saved to /var/cache/conftool/dbconfig/20240404-224119-marostegui.json
  • 22:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150', diff saved to https://phabricator.wikimedia.org/P59601 and previous config saved to /var/cache/conftool/dbconfig/20240404-223649-arnaudb.json
  • 22:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59600 and previous config saved to /var/cache/conftool/dbconfig/20240404-222612-marostegui.json
  • 22:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59599 and previous config saved to /var/cache/conftool/dbconfig/20240404-222141-arnaudb.json
  • 22:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2150 (T360332)', diff saved to https://phabricator.wikimedia.org/P59598 and previous config saved to /var/cache/conftool/dbconfig/20240404-221903-arnaudb.json
  • 22:18 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:18 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2150.codfw.wmnet with reason: Maintenance
  • 22:18 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59597 and previous config saved to /var/cache/conftool/dbconfig/20240404-221839-arnaudb.json
  • 22:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59596 and previous config saved to /var/cache/conftool/dbconfig/20240404-221104-marostegui.json
  • 22:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59595 and previous config saved to /var/cache/conftool/dbconfig/20240404-220331-arnaudb.json
  • 21:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59594 and previous config saved to /var/cache/conftool/dbconfig/20240404-215557-marostegui.json
  • 21:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122', diff saved to https://phabricator.wikimedia.org/P59593 and previous config saved to /var/cache/conftool/dbconfig/20240404-214824-arnaudb.json
  • 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1219 (T356166)', diff saved to https://phabricator.wikimedia.org/P59592 and previous config saved to /var/cache/conftool/dbconfig/20240404-214817-marostegui.json
  • 21:48 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 21:48 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1219.eqiad.wmnet with reason: Maintenance
  • 21:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59591 and previous config saved to /var/cache/conftool/dbconfig/20240404-214753-marostegui.json
  • 21:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59590 and previous config saved to /var/cache/conftool/dbconfig/20240404-213317-arnaudb.json
  • 21:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59589 and previous config saved to /var/cache/conftool/dbconfig/20240404-213245-marostegui.json
  • 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2122 (T360332)', diff saved to https://phabricator.wikimedia.org/P59588 and previous config saved to /var/cache/conftool/dbconfig/20240404-213031-arnaudb.json
  • 21:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2122.codfw.wmnet with reason: Maintenance
  • 21:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59587 and previous config saved to /var/cache/conftool/dbconfig/20240404-213008-arnaudb.json
  • 21:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218', diff saved to https://phabricator.wikimedia.org/P59586 and previous config saved to /var/cache/conftool/dbconfig/20240404-211738-marostegui.json
  • 21:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59585 and previous config saved to /var/cache/conftool/dbconfig/20240404-211501-arnaudb.json
  • 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T355609)', diff saved to https://phabricator.wikimedia.org/P59584 and previous config saved to /var/cache/conftool/dbconfig/20240404-211248-marostegui.json
  • 21:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 21:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 21:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59583 and previous config saved to /var/cache/conftool/dbconfig/20240404-210230-marostegui.json
  • 20:59 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120', diff saved to https://phabricator.wikimedia.org/P59582 and previous config saved to /var/cache/conftool/dbconfig/20240404-205953-arnaudb.json
  • 20:44 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59581 and previous config saved to /var/cache/conftool/dbconfig/20240404-204446-arnaudb.json
  • 20:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2120 (T360332)', diff saved to https://phabricator.wikimedia.org/P59580 and previous config saved to /var/cache/conftool/dbconfig/20240404-204204-arnaudb.json
  • 20:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2120.codfw.wmnet with reason: Maintenance
  • 20:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59579 and previous config saved to /var/cache/conftool/dbconfig/20240404-204141-arnaudb.json
  • 20:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 20:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 20:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59578 and previous config saved to /var/cache/conftool/dbconfig/20240404-203302-marostegui.json
  • 20:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59577 and previous config saved to /var/cache/conftool/dbconfig/20240404-202634-arnaudb.json
  • 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59576 and previous config saved to /var/cache/conftool/dbconfig/20240404-201755-marostegui.json
  • 20:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108', diff saved to https://phabricator.wikimedia.org/P59575 and previous config saved to /var/cache/conftool/dbconfig/20240404-201126-arnaudb.json
  • 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59574 and previous config saved to /var/cache/conftool/dbconfig/20240404-200247-marostegui.json
  • 19:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59573 and previous config saved to /var/cache/conftool/dbconfig/20240404-195615-arnaudb.json
  • 19:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2108 (T360332)', diff saved to https://phabricator.wikimedia.org/P59572 and previous config saved to /var/cache/conftool/dbconfig/20240404-195333-arnaudb.json
  • 19:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 19:53 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2108.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:52 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2098.codfw.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 19:51 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59571 and previous config saved to /var/cache/conftool/dbconfig/20240404-195138-arnaudb.json
  • 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59570 and previous config saved to /var/cache/conftool/dbconfig/20240404-194739-marostegui.json
  • 19:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59569 and previous config saved to /var/cache/conftool/dbconfig/20240404-193631-arnaudb.json
  • 19:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236', diff saved to https://phabricator.wikimedia.org/P59568 and previous config saved to /var/cache/conftool/dbconfig/20240404-192123-arnaudb.json
  • 19:11 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 19:10 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 19:06 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59567 and previous config saved to /var/cache/conftool/dbconfig/20240404-190616-arnaudb.json
  • 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1236 (T360332)', diff saved to https://phabricator.wikimedia.org/P59566 and previous config saved to /var/cache/conftool/dbconfig/20240404-190146-arnaudb.json
  • 19:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 19:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1236.eqiad.wmnet with reason: Maintenance
  • 19:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59565 and previous config saved to /var/cache/conftool/dbconfig/20240404-190123-arnaudb.json
  • 18:55 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T355609)', diff saved to https://phabricator.wikimedia.org/P59564 and previous config saved to /var/cache/conftool/dbconfig/20240404-185458-marostegui.json
  • 18:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 18:54 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59563 and previous config saved to /var/cache/conftool/dbconfig/20240404-185436-marostegui.json
  • 18:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1218 (T356166)', diff saved to https://phabricator.wikimedia.org/P59562 and previous config saved to /var/cache/conftool/dbconfig/20240404-185319-marostegui.json
  • 18:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 18:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1218.eqiad.wmnet with reason: Maintenance
  • 18:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59561 and previous config saved to /var/cache/conftool/dbconfig/20240404-185256-marostegui.json
  • 18:49 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:46 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 18:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59560 and previous config saved to /var/cache/conftool/dbconfig/20240404-184616-arnaudb.json
  • 18:41 denisse@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:40 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on webperf2003.codfw.wmnet,webperf1003.eqiad.wmnet with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 denisse@cumin2002: END (FAIL) - Cookbook sre.hosts.downtime (exit_code=99) for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 denisse@cumin2002: START - Cookbook sre.hosts.downtime for 0:30:00 on performance.wikimedia.org with reason: Downtiming the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:39 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59559 and previous config saved to /var/cache/conftool/dbconfig/20240404-183928-marostegui.json
  • 18:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59558 and previous config saved to /var/cache/conftool/dbconfig/20240404-183748-marostegui.json
  • 18:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227', diff saved to https://phabricator.wikimedia.org/P59557 and previous config saved to /var/cache/conftool/dbconfig/20240404-183108-arnaudb.json
  • 18:31 denisse: Disabling Puppet on the webperf hosts part of the cergen to CFSSL migration - T360414
  • 18:24 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59556 and previous config saved to /var/cache/conftool/dbconfig/20240404-182421-marostegui.json
  • 18:24 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:23 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207', diff saved to https://phabricator.wikimedia.org/P59555 and previous config saved to /var/cache/conftool/dbconfig/20240404-182241-marostegui.json
  • 18:19 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 18:16 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 18:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59554 and previous config saved to /var/cache/conftool/dbconfig/20240404-181601-arnaudb.json
  • 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1227 (T360332)', diff saved to https://phabricator.wikimedia.org/P59553 and previous config saved to /var/cache/conftool/dbconfig/20240404-181326-arnaudb.json
  • 18:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 18:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1227.eqiad.wmnet with reason: Maintenance
  • 18:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59552 and previous config saved to /var/cache/conftool/dbconfig/20240404-181303-arnaudb.json
  • 18:09 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59551 and previous config saved to /var/cache/conftool/dbconfig/20240404-180913-marostegui.json
  • 18:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59550 and previous config saved to /var/cache/conftool/dbconfig/20240404-180733-marostegui.json
  • 18:04 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 18:01 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1002-dev.eqiad.wmnet with reason: host reimage
  • 17:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59549 and previous config saved to /var/cache/conftool/dbconfig/20240404-175756-arnaudb.json
  • 17:49 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1002-dev.eqiad.wmnet with OS bookworm
  • 17:48 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
  • 17:48 sukhe: depool cp4052 to prepare for reimaging
  • 17:46 sukhe@cumin1002: END (PASS) - Cookbook sre.hardware.upgrade-firmware (exit_code=0) upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 17:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202', diff saved to https://phabricator.wikimedia.org/P59548 and previous config saved to /var/cache/conftool/dbconfig/20240404-174246-arnaudb.json
  • 17:37 sukhe@cumin1002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts cp4052.ulsfo.wmnet
  • 17:36 ebernhardson@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 17:35 ebernhardson@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 17:29 moritzm: installing isl bugfix updates from Bookworm point release
  • 17:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59547 and previous config saved to /var/cache/conftool/dbconfig/20240404-172739-arnaudb.json
  • 17:24 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1202 (T360332)', diff saved to https://phabricator.wikimedia.org/P59546 and previous config saved to /var/cache/conftool/dbconfig/20240404-172408-arnaudb.json
  • 17:24 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:23 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1202.eqiad.wmnet with reason: Maintenance
  • 17:23 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59545 and previous config saved to /var/cache/conftool/dbconfig/20240404-172343-arnaudb.json
  • 17:22 moritzm: installing qemu security updates on bookworm
  • 17:19 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 17:17 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 17:17 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 17:15 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T355609)', diff saved to https://phabricator.wikimedia.org/P59544 and previous config saved to /var/cache/conftool/dbconfig/20240404-171347-marostegui.json
  • 17:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 17:13 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59543 and previous config saved to /var/cache/conftool/dbconfig/20240404-171324-marostegui.json
  • 17:13 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 17:12 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 17:09 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3068.esams.wmnet
  • 17:08 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59542 and previous config saved to /var/cache/conftool/dbconfig/20240404-170836-arnaudb.json
  • 16:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59541 and previous config saved to /var/cache/conftool/dbconfig/20240404-165816-marostegui.json
  • 16:58 pmiazga: T355281 executed “mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki --skipclusters=main,echo,growth,mediamoderation,extstore en wikipedia test2wiki test2.wikipedia.beta.wmcloud.org” on deployment-deploy03.deployment-prep
  • 16:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3068.esams.wmnet with OS bullseye
  • 16:54 andrew@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
  • 16:53 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194', diff saved to https://phabricator.wikimedia.org/P59540 and previous config saved to /var/cache/conftool/dbconfig/20240404-165328-arnaudb.json
  • 16:51 andrew@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudbackup1001-dev.eqiad.wmnet with reason: host reimage
  • 16:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59539 and previous config saved to /var/cache/conftool/dbconfig/20240404-164309-marostegui.json
  • 16:42 andrew@cumin1002: START - Cookbook sre.hosts.reimage for host cloudbackup1001-dev.eqiad.wmnet with OS bookworm
  • 16:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59538 and previous config saved to /var/cache/conftool/dbconfig/20240404-163819-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1194 (T360332)', diff saved to https://phabricator.wikimedia.org/P59537 and previous config saved to /var/cache/conftool/dbconfig/20240404-163549-arnaudb.json
  • 16:35 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:35 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1194.eqiad.wmnet with reason: Maintenance
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59536 and previous config saved to /var/cache/conftool/dbconfig/20240404-163526-arnaudb.json
  • 16:32 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
  • 16:31 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59535 and previous config saved to /var/cache/conftool/dbconfig/20240404-163107-arnaudb.json
  • 16:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59534 and previous config saved to /var/cache/conftool/dbconfig/20240404-163053-arnaudb.json
  • 16:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59533 and previous config saved to /var/cache/conftool/dbconfig/20240404-162801-marostegui.json
  • 16:27 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3068.esams.wmnet with reason: host reimage
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59532 and previous config saved to /var/cache/conftool/dbconfig/20240404-162019-arnaudb.json
  • 16:16 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59531 and previous config saved to /var/cache/conftool/dbconfig/20240404-161601-arnaudb.json
  • 16:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59530 and previous config saved to /var/cache/conftool/dbconfig/20240404-161547-arnaudb.json
  • 16:10 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 16:09 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191', diff saved to https://phabricator.wikimedia.org/P59529 and previous config saved to /var/cache/conftool/dbconfig/20240404-160508-arnaudb.json
  • 16:04 jforrester@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifunctions: apply
  • 16:04 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3068.esams.wmnet with OS bullseye
  • 16:03 jforrester@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifunctions: apply
  • 16:03 jforrester@deploy1002: helmfile [codfw] DONE helmfile.d/services/wikifunctions: apply
  • 16:02 jforrester@deploy1002: helmfile [codfw] START helmfile.d/services/wikifunctions: apply
  • 16:01 jforrester@deploy1002: helmfile [staging] DONE helmfile.d/services/wikifunctions: apply
  • 16:01 jforrester@deploy1002: helmfile [staging] START helmfile.d/services/wikifunctions: apply
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59528 and previous config saved to /var/cache/conftool/dbconfig/20240404-160055-arnaudb.json
  • 16:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59527 and previous config saved to /var/cache/conftool/dbconfig/20240404-160041-arnaudb.json
  • 15:55 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3068.esams.wmnet
  • 15:54 fabfur: depooling cp3068 for reimage (T360430)
  • 15:54 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1207 (T356166)', diff saved to https://phabricator.wikimedia.org/P59526 and previous config saved to /var/cache/conftool/dbconfig/20240404-155420-marostegui.json
  • 15:54 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 15:54 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1207.eqiad.wmnet with reason: Maintenance
  • 15:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59525 and previous config saved to /var/cache/conftool/dbconfig/20240404-155357-marostegui.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59524 and previous config saved to /var/cache/conftool/dbconfig/20240404-155000-arnaudb.json
  • 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1191 (T360332)', diff saved to https://phabricator.wikimedia.org/P59523 and previous config saved to /var/cache/conftool/dbconfig/20240404-154730-arnaudb.json
  • 15:47 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:47 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1191.eqiad.wmnet with reason: Maintenance
  • 15:47 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59522 and previous config saved to /var/cache/conftool/dbconfig/20240404-154707-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59521 and previous config saved to /var/cache/conftool/dbconfig/20240404-154549-arnaudb.json
  • 15:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59520 and previous config saved to /var/cache/conftool/dbconfig/20240404-154535-arnaudb.json
  • 15:45 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 15:45 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 15:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59519 and previous config saved to /var/cache/conftool/dbconfig/20240404-153850-marostegui.json
  • 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T355609)', diff saved to https://phabricator.wikimedia.org/P59518 and previous config saved to /var/cache/conftool/dbconfig/20240404-153626-marostegui.json
  • 15:36 sukhe@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
  • 15:36 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 15:36 sukhe@cumin2002: START - Cookbook sre.hosts.downtime for 7 days, 0:00:00 on db2214.codfw.wmnet with reason: depooled, see T361851
  • 15:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 15:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59517 and previous config saved to /var/cache/conftool/dbconfig/20240404-153603-marostegui.json
  • 15:32 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59516 and previous config saved to /var/cache/conftool/dbconfig/20240404-153200-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59515 and previous config saved to /var/cache/conftool/dbconfig/20240404-153043-arnaudb.json
  • 15:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59514 and previous config saved to /var/cache/conftool/dbconfig/20240404-153030-arnaudb.json
  • 15:30 sukhe@cumin2002: dbctl commit (dc=all): 'depool db2214', diff saved to https://phabricator.wikimedia.org/P59513 and previous config saved to /var/cache/conftool/dbconfig/20240404-153023-sukhe.json
  • 15:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206', diff saved to https://phabricator.wikimedia.org/P59512 and previous config saved to /var/cache/conftool/dbconfig/20240404-152341-marostegui.json
  • 15:23 jgiannelos@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mobileapps: apply
  • 15:22 jgiannelos@deploy1002: helmfile [eqiad] START helmfile.d/services/mobileapps: apply
  • 15:22 logmsgbot: dreamyjazz Deployed security patch for T361296
  • 15:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59511 and previous config saved to /var/cache/conftool/dbconfig/20240404-152056-marostegui.json
  • 15:18 jgiannelos@deploy1002: helmfile [codfw] DONE helmfile.d/services/mobileapps: apply
  • 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174', diff saved to https://phabricator.wikimedia.org/P59510 and previous config saved to /var/cache/conftool/dbconfig/20240404-151653-arnaudb.json
  • 15:16 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:16 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:16 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59509 and previous config saved to /var/cache/conftool/dbconfig/20240404-151547-arnaudb.json
  • 15:15 jgiannelos@deploy1002: helmfile [codfw] START helmfile.d/services/mobileapps: apply
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59508 and previous config saved to /var/cache/conftool/dbconfig/20240404-151537-arnaudb.json
  • 15:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59507 and previous config saved to /var/cache/conftool/dbconfig/20240404-151524-arnaudb.json
  • 15:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 100%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59506 and previous config saved to /var/cache/conftool/dbconfig/20240404-151123-arnaudb.json
  • 15:10 herron: beginning rolling hardware upgrades on titan200[12] T361229
  • 15:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59505 and previous config saved to /var/cache/conftool/dbconfig/20240404-150833-marostegui.json
  • 15:08 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 15:08 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 15:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59504 and previous config saved to /var/cache/conftool/dbconfig/20240404-150549-marostegui.json
  • 15:03 logmsgbot: dreamyjazz Deployed security patch for T361295
  • 15:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59503 and previous config saved to /var/cache/conftool/dbconfig/20240404-150145-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59502 and previous config saved to /var/cache/conftool/dbconfig/20240404-150041-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59501 and previous config saved to /var/cache/conftool/dbconfig/20240404-150032-arnaudb.json
  • 15:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59500 and previous config saved to /var/cache/conftool/dbconfig/20240404-150017-arnaudb.json
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1174 (T360332)', diff saved to https://phabricator.wikimedia.org/P59499 and previous config saved to /var/cache/conftool/dbconfig/20240404-145714-arnaudb.json
  • 14:57 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1174.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 70%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59498 and previous config saved to /var/cache/conftool/dbconfig/20240404-145617-arnaudb.json
  • 14:56 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 14:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59497 and previous config saved to /var/cache/conftool/dbconfig/20240404-145613-arnaudb.json
  • 14:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59496 and previous config saved to /var/cache/conftool/dbconfig/20240404-145041-marostegui.json
  • 14:48 dreamyjazz@deploy1002: Finished scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695) (duration: 14m 49s)
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59495 and previous config saved to /var/cache/conftool/dbconfig/20240404-144536-arnaudb.json
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59494 and previous config saved to /var/cache/conftool/dbconfig/20240404-144526-arnaudb.json
  • 14:45 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59493 and previous config saved to /var/cache/conftool/dbconfig/20240404-144511-arnaudb.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 50%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59492 and previous config saved to /var/cache/conftool/dbconfig/20240404-144111-arnaudb.json
  • 14:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59491 and previous config saved to /var/cache/conftool/dbconfig/20240404-144105-arnaudb.json
  • 14:36 dreamyjazz@deploy1002: dreamyjazz: Continuing with sync
  • 14:36 dreamyjazz@deploy1002: dreamyjazz: Backport for Remove sk translation of centralauth-rightslog-name (T361695) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:33 dreamyjazz@deploy1002: Started scap: Backport for Remove sk translation of centralauth-rightslog-name (T361695)
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59490 and previous config saved to /var/cache/conftool/dbconfig/20240404-143030-arnaudb.json
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2213 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59489 and previous config saved to /var/cache/conftool/dbconfig/20240404-143020-arnaudb.json
  • 14:30 arnaudb@cumin1002: dbctl commit (dc=all): 'db2207 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59488 and previous config saved to /var/cache/conftool/dbconfig/20240404-143006-arnaudb.json
  • 14:26 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 30%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59487 and previous config saved to /var/cache/conftool/dbconfig/20240404-142606-arnaudb.json
  • 14:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170', diff saved to https://phabricator.wikimedia.org/P59486 and previous config saved to /var/cache/conftool/dbconfig/20240404-142558-arnaudb.json
  • 14:22 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki (duration: 18m 43s)
  • 14:15 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59485 and previous config saved to /var/cache/conftool/dbconfig/20240404-141517-arnaudb.json
  • 14:11 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 20%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59484 and previous config saved to /var/cache/conftool/dbconfig/20240404-141100-arnaudb.json
  • 14:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59483 and previous config saved to /var/cache/conftool/dbconfig/20240404-141051-arnaudb.json
  • 14:09 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Continuing with sync
  • 14:05 logmsgbot: lucaswerkmeister-wmde@deploy1002 esanders and lucaswerkmeister-wmde: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 14:03 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for Enable wgVisualEditorAllowExternalLinkPaste at collabwiki
  • 14:01 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for End EditCheck add-a-reference A/B test (T361727) (duration: 20m 05s)
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T355609)', diff saved to https://phabricator.wikimedia.org/P59482 and previous config saved to /var/cache/conftool/dbconfig/20240404-140050-marostegui.json
  • 14:00 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 14:00 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59481 and previous config saved to /var/cache/conftool/dbconfig/20240404-140027-marostegui.json
  • 14:00 arnaudb@cumin1002: dbctl commit (dc=all): 'db2113 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59480 and previous config saved to /var/cache/conftool/dbconfig/20240404-140011-arnaudb.json
  • 13:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db2107 (re)pooling @ 10%: Post clone repool (src)', diff saved to https://phabricator.wikimedia.org/P59479 and previous config saved to /var/cache/conftool/dbconfig/20240404-135547-arnaudb.json
  • 13:48 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
  • 13:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1170 (T360332)', diff saved to https://phabricator.wikimedia.org/P59478 and previous config saved to /var/cache/conftool/dbconfig/20240404-134607-arnaudb.json
  • 13:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:45 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1170.eqiad.wmnet with reason: Maintenance
  • 13:45 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59477 and previous config saved to /var/cache/conftool/dbconfig/20240404-134544-arnaudb.json
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59476 and previous config saved to /var/cache/conftool/dbconfig/20240404-134519-marostegui.json
  • 13:43 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for End EditCheck add-a-reference A/B test (T361727) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:41 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for End EditCheck add-a-reference A/B test (T361727)
  • 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
  • 13:39 logmsgbot: lucaswerkmeister-wmde@deploy1002 Finished scap: Backport for DiscussionTools: Remove no-op config (duration: 15m 10s)
  • 13:30 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59475 and previous config saved to /var/cache/conftool/dbconfig/20240404-133037-arnaudb.json
  • 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59474 and previous config saved to /var/cache/conftool/dbconfig/20240404-133012-marostegui.json
  • 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Continuing with sync
  • 13:26 logmsgbot: lucaswerkmeister-wmde@deploy1002 lucaswerkmeister-wmde and esanders: Backport for DiscussionTools: Remove no-op config synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:24 logmsgbot: lucaswerkmeister-wmde@deploy1002 Started scap: Backport for DiscussionTools: Remove no-op config
  • 13:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158', diff saved to https://phabricator.wikimedia.org/P59473 and previous config saved to /var/cache/conftool/dbconfig/20240404-131529-arnaudb.json
  • 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59472 and previous config saved to /var/cache/conftool/dbconfig/20240404-131504-marostegui.json
  • 13:04 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
  • 13:00 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59471 and previous config saved to /var/cache/conftool/dbconfig/20240404-130022-arnaudb.json
  • 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1206 (T356166)', diff saved to https://phabricator.wikimedia.org/P59470 and previous config saved to /var/cache/conftool/dbconfig/20240404-124257-marostegui.json
  • 12:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 12:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1206.eqiad.wmnet with reason: Maintenance
  • 12:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59469 and previous config saved to /var/cache/conftool/dbconfig/20240404-124235-marostegui.json
  • 12:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59468 and previous config saved to /var/cache/conftool/dbconfig/20240404-123645-arnaudb.json
  • 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1014,1018,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:36 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1158.eqiad.wmnet with reason: Maintenance
  • 12:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59467 and previous config saved to /var/cache/conftool/dbconfig/20240404-122727-marostegui.json
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T355609)', diff saved to https://phabricator.wikimedia.org/P59466 and previous config saved to /var/cache/conftool/dbconfig/20240404-122557-marostegui.json
  • 12:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 12:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59465 and previous config saved to /var/cache/conftool/dbconfig/20240404-122535-marostegui.json
  • 12:18 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2113.codfw.wmnet onto db2213.codfw.wmnet
  • 12:17 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2113 in db2213 for T355422', diff saved to https://phabricator.wikimedia.org/P59463 and previous config saved to /var/cache/conftool/dbconfig/20240404-121722-arnaudb.json
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2213.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:16 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2113.codfw.wmnet with reason: provisionning db2213.codfw.wmnet - T355422
  • 12:12 arnaudb@cumin1002: START - Cookbook sre.mysql.clone Will create a clone of db2107.codfw.wmnet onto db2207.codfw.wmnet
  • 12:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196', diff saved to https://phabricator.wikimedia.org/P59462 and previous config saved to /var/cache/conftool/dbconfig/20240404-121218-marostegui.json
  • 12:11 aikochou@deploy1002: helmfile [ml-staging-codfw] Ran 'sync' command on namespace 'revertrisk' for release 'main' .
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59461 and previous config saved to /var/cache/conftool/dbconfig/20240404-121027-marostegui.json
  • 12:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Cloning db2107 in db2207 for T355422', diff saved to https://phabricator.wikimedia.org/P59460 and previous config saved to /var/cache/conftool/dbconfig/20240404-121008-arnaudb.json
  • 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2207.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 12:08 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2107.codfw.wmnet with reason: provisionning db2207.codfw.wmnet - T355422
  • 11:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59459 and previous config saved to /var/cache/conftool/dbconfig/20240404-115709-marostegui.json
  • 11:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59458 and previous config saved to /var/cache/conftool/dbconfig/20240404-115520-marostegui.json
  • 11:49 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.debug (exit_code=0) for Netbox circuit ID 108
  • 11:49 ayounsi@cumin1002: START - Cookbook sre.network.debug for Netbox circuit ID 108
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59457 and previous config saved to /var/cache/conftool/dbconfig/20240404-114012-marostegui.json
  • 10:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T355609)', diff saved to https://phabricator.wikimedia.org/P59456 and previous config saved to /var/cache/conftool/dbconfig/20240404-105158-marostegui.json
  • 10:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 10:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59455 and previous config saved to /var/cache/conftool/dbconfig/20240404-105135-marostegui.json
  • 10:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59453 and previous config saved to /var/cache/conftool/dbconfig/20240404-103628-marostegui.json
  • 10:34 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:34 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:33 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:33 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59451 and previous config saved to /var/cache/conftool/dbconfig/20240404-102120-marostegui.json
  • 10:10 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::objectstash
  • 10:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59450 and previous config saved to /var/cache/conftool/dbconfig/20240404-100612-marostegui.json
  • 10:05 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:05 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 09:57 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::objectstash
  • 09:52 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::misc
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts db2104.codfw.wmnet
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 09:49 marostegui@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
  • 09:48 marostegui@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2104.codfw.wmnet decommissioned, removing all IPs except the asset tag one - marostegui@cumin1002"
  • 09:46 marostegui@cumin1002: START - Cookbook sre.dns.netbox
  • 09:46 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1196 (T356166)', diff saved to https://phabricator.wikimedia.org/P59449 and previous config saved to /var/cache/conftool/dbconfig/20240404-094608-marostegui.json
  • 09:46 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1196.eqiad.wmnet with reason: Maintenance
  • 09:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59448 and previous config saved to /var/cache/conftool/dbconfig/20240404-094536-marostegui.json
  • 09:41 marostegui@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2104.codfw.wmnet
  • 09:31 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::misc
  • 09:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59446 and previous config saved to /var/cache/conftool/dbconfig/20240404-093028-marostegui.json
  • 09:18 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2113 weight', diff saved to https://phabricator.wikimedia.org/P59445 and previous config saved to /var/cache/conftool/dbconfig/20240404-091858-arnaudb.json
  • 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T355609)', diff saved to https://phabricator.wikimedia.org/P59444 and previous config saved to /var/cache/conftool/dbconfig/20240404-091732-marostegui.json
  • 09:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 09:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59443 and previous config saved to /var/cache/conftool/dbconfig/20240404-091709-marostegui.json
  • 09:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186', diff saved to https://phabricator.wikimedia.org/P59442 and previous config saved to /var/cache/conftool/dbconfig/20240404-091521-marostegui.json
  • 09:15 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2123 to s5 primary T361789', diff saved to https://phabricator.wikimedia.org/P59441 and previous config saved to /var/cache/conftool/dbconfig/20240404-091512-arnaudb.json
  • 09:14 arnaudb: Starting s5 codfw failover from db2113 to db2123 - T361789
  • 09:11 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.migrate-role (exit_code=0) for role: mariadb::backup_source
  • 09:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59440 and previous config saved to /var/cache/conftool/dbconfig/20240404-090202-marostegui.json
  • 09:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59439 and previous config saved to /var/cache/conftool/dbconfig/20240404-090007-marostegui.json
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1186 (T356166)', diff saved to https://phabricator.wikimedia.org/P59438 and previous config saved to /var/cache/conftool/dbconfig/20240404-085856-marostegui.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1186.eqiad.wmnet with reason: Maintenance
  • 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59437 and previous config saved to /var/cache/conftool/dbconfig/20240404-085834-marostegui.json
  • 08:56 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2123 with weight 0 T361789', diff saved to https://phabricator.wikimedia.org/P59436 and previous config saved to /var/cache/conftool/dbconfig/20240404-085606-arnaudb.json
  • 08:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
  • 08:55 jmm@cumin2002: START - Cookbook sre.puppet.migrate-role for role: mariadb::backup_source
  • 08:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 27 hosts with reason: Primary switchover s5 T361789
  • 08:46 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59435 and previous config saved to /var/cache/conftool/dbconfig/20240404-084655-marostegui.json
  • 08:43 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59434 and previous config saved to /var/cache/conftool/dbconfig/20240404-084327-marostegui.json
  • 08:31 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59433 and previous config saved to /var/cache/conftool/dbconfig/20240404-083147-marostegui.json
  • 08:28 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1169', diff saved to https://phabricator.wikimedia.org/P59432 and previous config saved to /var/cache/conftool/dbconfig/20240404-082819-marostegui.json
  • 08:25 arnaudb@cumin1002: dbctl commit (dc=all): 'bump db2107 weight', diff saved to https://phabricator.wikimedia.org/P59431 and previous config saved to /var/cache/conftool/dbconfig/20240404-082547-root.json
  • 08:24 brouberol@deploy1002: helmfile [dse-k8s-eqiad] DONE helmfile.d/admin 'apply'.
  • 08:23 brouberol@deploy1002: helmfile [dse-k8s-eqiad] START helmfile.d/admin 'apply'.
  • 08:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Promote db2204 to s2 primary T361682', diff saved to https://phabricator.wikimedia.org/P59430 and previous config saved to /var/cache/conftool/dbconfig/20240404-082200-arnaudb.json
  • 08:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Set db2204 with weight 0 T361682', diff saved to https://phabricator.wikimedia.org/P59427 and previous config saved to /var/cache/conftool/dbconfig/20240404-080408-arnaudb.json
  • 08:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
  • 08:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1:00:00 on 29 hosts with reason: Primary switchover s2 T361682
  • 07:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T355609)', diff saved to https://phabricator.wikimedia.org/P59426 and previous config saved to /var/cache/conftool/dbconfig/20240404-074313-marostegui.json
  • 07:43 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'Remove db2104 from dbctl T361779', diff saved to https://phabricator.wikimedia.org/P59425 and previous config saved to /var/cache/conftool/dbconfig/20240404-073600-root.json
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59424 and previous config saved to /var/cache/conftool/dbconfig/20240404-072928-root.json
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59423 and previous config saved to /var/cache/conftool/dbconfig/20240404-071423-root.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59422 and previous config saved to /var/cache/conftool/dbconfig/20240404-065917-root.json
  • 06:59 moritzm: installing util-linux security updates
  • 06:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:58 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:58 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59421 and previous config saved to /var/cache/conftool/dbconfig/20240404-065758-marostegui.json
  • 06:44 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59420 and previous config saved to /var/cache/conftool/dbconfig/20240404-064411-root.json
  • 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59419 and previous config saved to /var/cache/conftool/dbconfig/20240404-064250-marostegui.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59418 and previous config saved to /var/cache/conftool/dbconfig/20240404-062905-root.json
  • 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59417 and previous config saved to /var/cache/conftool/dbconfig/20240404-062743-marostegui.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59416 and previous config saved to /var/cache/conftool/dbconfig/20240404-061400-root.json
  • 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59415 and previous config saved to /var/cache/conftool/dbconfig/20240404-061234-marostegui.json
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1169 (T356166)', diff saved to https://phabricator.wikimedia.org/P59414 and previous config saved to /var/cache/conftool/dbconfig/20240404-060524-marostegui.json
  • 06:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 06:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1169.eqiad.wmnet with reason: Maintenance
  • 06:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59413 and previous config saved to /var/cache/conftool/dbconfig/20240404-060501-marostegui.json
  • 06:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2126.codfw.wmnet with OS bookworm
  • 05:58 marostegui@cumin1002: dbctl commit (dc=all): 'db2126 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59412 and previous config saved to /var/cache/conftool/dbconfig/20240404-055854-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59411 and previous config saved to /var/cache/conftool/dbconfig/20240404-054953-marostegui.json
  • 05:39 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
  • 05:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2126.codfw.wmnet with reason: host reimage
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163', diff saved to https://phabricator.wikimedia.org/P59410 and previous config saved to /var/cache/conftool/dbconfig/20240404-053446-marostegui.json
  • 05:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T355609)', diff saved to https://phabricator.wikimedia.org/P59409 and previous config saved to /var/cache/conftool/dbconfig/20240404-052338-marostegui.json
  • 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59408 and previous config saved to /var/cache/conftool/dbconfig/20240404-051938-marostegui.json
  • 05:19 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2126.codfw.wmnet with OS bookworm
  • 05:18 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2126 T361543', diff saved to https://phabricator.wikimedia.org/P59407 and previous config saved to /var/cache/conftool/dbconfig/20240404-051758-root.json
  • 05:17 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1163 (T356166)', diff saved to https://phabricator.wikimedia.org/P59406 and previous config saved to /var/cache/conftool/dbconfig/20240404-051728-marostegui.json
  • 05:17 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 05:17 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1163.eqiad.wmnet with reason: Maintenance
  • 02:48 TimStarling: ran maintain-views on clouddb1013-1019 (T355034)
  • 02:29 eileen: civicrm upgraded from a0fb57d3 to 8c7cc208
  • 02:28 eileen: config revision changed from 3ed18c47 to abccfdc0
  • 02:11 eileen: config revision changed from 3ed18c47 to abccfdc0
  • 01:07 TimStarling: on clouddb1020 running maintain-views --all-databases --replace-all --auto-depool (T355034)
  • 00:56 TimStarling: on clouddb1021 ran maintain-views for all databases

2024-04-03

  • 23:33 TimStarling: on clouddb1021 ran maintain-views for enwiki
  • 22:35 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:26 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59405 and previous config saved to /var/cache/conftool/dbconfig/20240403-222610-arnaudb.json
  • 22:11 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59404 and previous config saved to /var/cache/conftool/dbconfig/20240403-221103-arnaudb.json
  • 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:06 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:00 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217', diff saved to https://phabricator.wikimedia.org/P59403 and previous config saved to /var/cache/conftool/dbconfig/20240403-215555-arnaudb.json
  • 21:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59402 and previous config saved to /var/cache/conftool/dbconfig/20240403-214048-arnaudb.json
  • 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2217 (T360332)', diff saved to https://phabricator.wikimedia.org/P59401 and previous config saved to /var/cache/conftool/dbconfig/20240403-213825-arnaudb.json
  • 21:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 21:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2217.codfw.wmnet with reason: Maintenance
  • 21:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59400 and previous config saved to /var/cache/conftool/dbconfig/20240403-213802-arnaudb.json
  • 21:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59399 and previous config saved to /var/cache/conftool/dbconfig/20240403-212255-arnaudb.json
  • 21:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214', diff saved to https://phabricator.wikimedia.org/P59398 and previous config saved to /var/cache/conftool/dbconfig/20240403-210747-arnaudb.json
  • 21:02 jforrester@deploy1002: Finished scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) (duration: 14m 18s)
  • 20:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59397 and previous config saved to /var/cache/conftool/dbconfig/20240403-205240-arnaudb.json
  • 20:51 jforrester@deploy1002: ederporto and jforrester: Continuing with sync
  • 20:50 jforrester@deploy1002: ederporto and jforrester: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2214 (T360332)', diff saved to https://phabricator.wikimedia.org/P59396 and previous config saved to /var/cache/conftool/dbconfig/20240403-205014-arnaudb.json
  • 20:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 20:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2214.codfw.wmnet with reason: Maintenance
  • 20:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59395 and previous config saved to /var/cache/conftool/dbconfig/20240403-204949-arnaudb.json
  • 20:48 jforrester@deploy1002: Started scap: Backport for component: Add SandboxLink to Portuguese Wikiquote (T361447)
  • 20:45 jforrester@deploy1002: Finished scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) (duration: 19m 03s)
  • 20:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59394 and previous config saved to /var/cache/conftool/dbconfig/20240403-203440-arnaudb.json
  • 20:34 jforrester@deploy1002: sfaci and jforrester: Continuing with sync
  • 20:28 jforrester@deploy1002: sfaci and jforrester: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:26 jforrester@deploy1002: Started scap: Backport for Update the WikiLambda instrumentation to use core interaction events (T350497)
  • 20:23 jforrester@deploy1002: Finished scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) (duration: 14m 58s)
  • 20:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193', diff saved to https://phabricator.wikimedia.org/P59393 and previous config saved to /var/cache/conftool/dbconfig/20240403-201933-arnaudb.json
  • 20:12 jforrester@deploy1002: jforrester: Continuing with sync
  • 20:11 jforrester@deploy1002: jforrester: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 20:10 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 20:10 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 20:08 jforrester@deploy1002: Started scap: Backport for Centralize API calls in api.js mixin and fix error handling (T361598 T315432)
  • 20:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59392 and previous config saved to /var/cache/conftool/dbconfig/20240403-200425-arnaudb.json
  • 20:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2193 (T360332)', diff saved to https://phabricator.wikimedia.org/P59391 and previous config saved to /var/cache/conftool/dbconfig/20240403-200201-arnaudb.json
  • 20:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 20:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2193.codfw.wmnet with reason: Maintenance
  • 20:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59390 and previous config saved to /var/cache/conftool/dbconfig/20240403-200137-arnaudb.json
  • 20:00 mutante: stewards1001 - rebooting to switch from iptables to nftables
  • 19:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59388 and previous config saved to /var/cache/conftool/dbconfig/20240403-194630-arnaudb.json
  • 19:38 mutante: stewards2001 - reboot to switch from iptables to nftables
  • 19:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180', diff saved to https://phabricator.wikimedia.org/P59387 and previous config saved to /var/cache/conftool/dbconfig/20240403-193122-arnaudb.json
  • 19:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59386 and previous config saved to /var/cache/conftool/dbconfig/20240403-191615-arnaudb.json
  • 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59385 and previous config saved to /var/cache/conftool/dbconfig/20240403-191351-arnaudb.json
  • 19:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 19:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2180.codfw.wmnet with reason: Maintenance
  • 19:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59384 and previous config saved to /var/cache/conftool/dbconfig/20240403-191328-arnaudb.json
  • 19:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 19:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 19:04 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
  • 19:03 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
  • 19:03 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
  • 19:02 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
  • 19:02 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 19:01 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 18:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59383 and previous config saved to /var/cache/conftool/dbconfig/20240403-185821-arnaudb.json
  • 18:57 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 18:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 18:51 eileen: config revision changed from 821f145b to 3ed18c47
  • 18:49 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 18:49 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 18:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169', diff saved to https://phabricator.wikimedia.org/P59382 and previous config saved to /var/cache/conftool/dbconfig/20240403-184313-arnaudb.json
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts restbase[1019-1027].eqiad.wmnet
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 18:37 eevans@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
  • 18:35 eevans@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: restbase[1019-1027].eqiad.wmnet decommissioned, removing all IPs except the asset tag one - eevans@cumin1002"
  • 18:34 eevans@cumin1002: START - Cookbook sre.dns.netbox
  • 18:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59381 and previous config saved to /var/cache/conftool/dbconfig/20240403-182806-arnaudb.json
  • 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2169 (T360332)', diff saved to https://phabricator.wikimedia.org/P59380 and previous config saved to /var/cache/conftool/dbconfig/20240403-182543-arnaudb.json
  • 18:25 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 18:25 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2169.codfw.wmnet with reason: Maintenance
  • 18:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59379 and previous config saved to /var/cache/conftool/dbconfig/20240403-182520-arnaudb.json
  • 18:13 logmsgbot: dreamyjazz Deployed security patch for T361479
  • 18:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59378 and previous config saved to /var/cache/conftool/dbconfig/20240403-181013-arnaudb.json
  • 18:09 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 18:08 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 18:07 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 18:07 eevans@cumin1002: START - Cookbook sre.hosts.decommission for hosts restbase[1019-1027].eqiad.wmnet
  • 18:06 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 18:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 18:04 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 18:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59377 and previous config saved to /var/cache/conftool/dbconfig/20240403-180323-root.json
  • 17:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158', diff saved to https://phabricator.wikimedia.org/P59375 and previous config saved to /var/cache/conftool/dbconfig/20240403-175505-arnaudb.json
  • 17:48 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59374 and previous config saved to /var/cache/conftool/dbconfig/20240403-174817-root.json
  • 17:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59373 and previous config saved to /var/cache/conftool/dbconfig/20240403-173958-arnaudb.json
  • 17:38 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2158 (T360332)', diff saved to https://phabricator.wikimedia.org/P59372 and previous config saved to /var/cache/conftool/dbconfig/20240403-173835-arnaudb.json
  • 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on db2187.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 17:38 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2158.codfw.wmnet with reason: Maintenance
  • 17:37 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59371 and previous config saved to /var/cache/conftool/dbconfig/20240403-173756-arnaudb.json
  • 17:33 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59370 and previous config saved to /var/cache/conftool/dbconfig/20240403-173312-root.json
  • 17:22 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59369 and previous config saved to /var/cache/conftool/dbconfig/20240403-172249-arnaudb.json
  • 17:18 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59368 and previous config saved to /var/cache/conftool/dbconfig/20240403-171806-root.json
  • 17:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151', diff saved to https://phabricator.wikimedia.org/P59367 and previous config saved to /var/cache/conftool/dbconfig/20240403-170741-arnaudb.json
  • 17:04 herron: performing rolling memory upgrades on prometheus100[56] T360687
  • 17:03 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59366 and previous config saved to /var/cache/conftool/dbconfig/20240403-170300-root.json
  • 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59365 and previous config saved to /var/cache/conftool/dbconfig/20240403-165234-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2151 (T360332)', diff saved to https://phabricator.wikimedia.org/P59364 and previous config saved to /var/cache/conftool/dbconfig/20240403-165011-arnaudb.json
  • 16:50 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 16:49 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2151.codfw.wmnet with reason: Maintenance
  • 16:49 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59363 and previous config saved to /var/cache/conftool/dbconfig/20240403-164948-arnaudb.json
  • 16:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59362 and previous config saved to /var/cache/conftool/dbconfig/20240403-164754-root.json
  • 16:38 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot-master (exit_code=0) rolling restart_daemons on A:maps-master
  • 16:36 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot-master rolling restart_daemons on A:maps-master
  • 16:34 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59361 and previous config saved to /var/cache/conftool/dbconfig/20240403-163440-arnaudb.json
  • 16:32 marostegui@cumin1002: dbctl commit (dc=all): 'db1167 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59360 and previous config saved to /var/cache/conftool/dbconfig/20240403-163249-root.json
  • 16:30 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59359 and previous config saved to /var/cache/conftool/dbconfig/20240403-163004-marostegui.json
  • 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:29 jayme@deploy1002: Finished scap: (no justification provided) (duration: 03m 34s)
  • 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 16:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 16:26 effie: pooling back mw-web-ro in eqiad
  • 16:26 jayme@deploy1002: Started scap: (no justification provided)
  • 16:26 jiji@cumin1002: conftool action : set/pooled=true; selector: dnsdisc=mw-web-ro,name=eqiad
  • 16:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124', diff saved to https://phabricator.wikimedia.org/P59358 and previous config saved to /var/cache/conftool/dbconfig/20240403-161933-arnaudb.json
  • 16:14 jiji@deploy1002: helmfile [eqiad] DONE helmfile.d/services/mw-web: apply
  • 16:12 jiji@deploy1002: helmfile [eqiad] START helmfile.d/services/mw-web: apply
  • 16:08 akosiaris@deploy1002: helmfile [codfw] DONE helmfile.d/services/changeprop: apply
  • 16:07 akosiaris@deploy1002: helmfile [codfw] START helmfile.d/services/changeprop: apply
  • 16:05 akosiaris@deploy1002: helmfile [eqiad] DONE helmfile.d/services/changeprop: apply
  • 16:05 akosiaris@deploy1002: helmfile [eqiad] START helmfile.d/services/changeprop: apply
  • 16:04 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59357 and previous config saved to /var/cache/conftool/dbconfig/20240403-160425-arnaudb.json
  • 16:02 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2124 (T360332)', diff saved to https://phabricator.wikimedia.org/P59356 and previous config saved to /var/cache/conftool/dbconfig/20240403-160159-arnaudb.json
  • 16:02 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 16:01 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2124.codfw.wmnet with reason: Maintenance
  • 16:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59355 and previous config saved to /var/cache/conftool/dbconfig/20240403-160136-arnaudb.json
  • 15:53 jiji@cumin1002: conftool action : set/pooled=false; selector: dnsdisc=mw-web-ro,name=eqiad
  • 15:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59354 and previous config saved to /var/cache/conftool/dbconfig/20240403-154628-arnaudb.json
  • 15:33 jiji@cumin1002: END (PASS) - Cookbook sre.discovery.datacenter (exit_code=0) status all services in all: None - None
  • 15:33 jiji@cumin1002: START - Cookbook sre.discovery.datacenter status all services in all: None - None
  • 15:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114', diff saved to https://phabricator.wikimedia.org/P59353 and previous config saved to /var/cache/conftool/dbconfig/20240403-153121-arnaudb.json
  • 15:22 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:22 Dreamy_Jazz: Starting MediaModeration scanning script again - It crashed due to the outage
  • 15:22 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59352 and previous config saved to /var/cache/conftool/dbconfig/20240403-151614-arnaudb.json
  • 15:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db2114 (T360332)', diff saved to https://phabricator.wikimedia.org/P59351 and previous config saved to /var/cache/conftool/dbconfig/20240403-151349-arnaudb.json
  • 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 15:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2114.codfw.wmnet with reason: Maintenance
  • 15:13 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db2097.codfw.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:12 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59350 and previous config saved to /var/cache/conftool/dbconfig/20240403-151233-arnaudb.json
  • 15:04 jynus@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
  • 15:03 jynus@cumin1002: START - Cookbook sre.hosts.downtime for 4:00:00 on db2098.codfw.wmnet with reason: restart of mysqld
  • 15:02 dreamyjazz@deploy1002: Finished scap: (no justification provided) (duration: 18m 54s)
  • 15:01 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/services/wikifeeds: sync
  • 15:01 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1040.eqiad.wmnet with OS bookworm
  • 15:01 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/services/wikifeeds: sync
  • 14:57 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59349 and previous config saved to /var/cache/conftool/dbconfig/20240403-145725-arnaudb.json
  • 14:44 dreamyjazz@deploy1002: Started scap: (no justification provided)
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59347 and previous config saved to /var/cache/conftool/dbconfig/20240403-144217-arnaudb.json
  • 14:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
  • 14:31 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1040.eqiad.wmnet with reason: host reimage
  • 14:27 hnowlan@deploy1002: helmfile [eqiad] DONE helmfile.d/admin 'apply'.
  • 14:27 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59346 and previous config saved to /var/cache/conftool/dbconfig/20240403-142709-arnaudb.json
  • 14:27 hnowlan@deploy1002: helmfile [eqiad] START helmfile.d/admin 'apply'.
  • 14:26 hnowlan@deploy1002: helmfile [codfw] DONE helmfile.d/admin 'apply'.
  • 14:26 hnowlan@deploy1002: helmfile [codfw] START helmfile.d/admin 'apply'.
  • 14:24 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1040
  • 14:24 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1040
  • 14:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 14:21 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 14:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59345 and previous config saved to /var/cache/conftool/dbconfig/20240403-142142-marostegui.json
  • 14:17 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-eqiad
  • 14:16 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1040.eqiad.wmnet with OS bookworm
  • 14:11 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-eqiad
  • 14:11 jmm@cumin2002: END (PASS) - Cookbook sre.maps.roll-restart-reboot (exit_code=0) rolling restart_daemons on A:maps-replica-codfw
  • 14:09 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 14:09 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 14:08 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'configure' for AS: 2527
  • 14:07 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'configure' for AS: 2527
  • 14:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59344 and previous config saved to /var/cache/conftool/dbconfig/20240403-140634-marostegui.json
  • 14:06 jmm@cumin2002: START - Cookbook sre.maps.roll-restart-reboot rolling restart_daemons on A:maps-replica-codfw
  • 13:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226', diff saved to https://phabricator.wikimedia.org/P59343 and previous config saved to /var/cache/conftool/dbconfig/20240403-135126-marostegui.json
  • 13:41 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T360332)', diff saved to https://phabricator.wikimedia.org/P59342 and previous config saved to /var/cache/conftool/dbconfig/20240403-134136-arnaudb.json
  • 13:41 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:41 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 13:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59341 and previous config saved to /var/cache/conftool/dbconfig/20240403-134044-arnaudb.json
  • 13:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59340 and previous config saved to /var/cache/conftool/dbconfig/20240403-133619-marostegui.json
  • 13:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1226 (T356166)', diff saved to https://phabricator.wikimedia.org/P59339 and previous config saved to /var/cache/conftool/dbconfig/20240403-133200-marostegui.json
  • 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1226.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 13:16 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59336 and previous config saved to /var/cache/conftool/dbconfig/20240403-131606-marostegui.json
  • 13:15 moritzm: installing tiff security updates
  • 13:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59335 and previous config saved to /var/cache/conftool/dbconfig/20240403-131029-arnaudb.json
  • 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214', diff saved to https://phabricator.wikimedia.org/P59334 and previous config saved to /var/cache/conftool/dbconfig/20240403-130058-marostegui.json
  • 12:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59333 and previous config saved to /var/cache/conftool/dbconfig/20240403-125521-arnaudb.json
  • 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59332 and previous config saved to /var/cache/conftool/dbconfig/20240403-124550-marostegui.json
  • 12:42 hashar: Upgrading CI Jenkins # T360759
  • 12:34 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1039.eqiad.wmnet with OS bookworm
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1214 (T356166)', diff saved to https://phabricator.wikimedia.org/P59330 and previous config saved to /var/cache/conftool/dbconfig/20240403-123329-marostegui.json
  • 12:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 12:33 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1214.eqiad.wmnet with reason: Maintenance
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59329 and previous config saved to /var/cache/conftool/dbconfig/20240403-123306-marostegui.json
  • 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T360332)', diff saved to https://phabricator.wikimedia.org/P59328 and previous config saved to /var/cache/conftool/dbconfig/20240403-123156-arnaudb.json
  • 12:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 12:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 12:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59327 and previous config saved to /var/cache/conftool/dbconfig/20240403-123133-arnaudb.json
  • 12:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59326 and previous config saved to /var/cache/conftool/dbconfig/20240403-121759-marostegui.json
  • 12:16 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59325 and previous config saved to /var/cache/conftool/dbconfig/20240403-121626-arnaudb.json
  • 12:11 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
  • 12:08 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1039.eqiad.wmnet with reason: host reimage
  • 12:07 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1039
  • 12:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211', diff saved to https://phabricator.wikimedia.org/P59324 and previous config saved to /var/cache/conftool/dbconfig/20240403-120251-marostegui.json
  • 12:02 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 12:02 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 12:01 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59323 and previous config saved to /var/cache/conftool/dbconfig/20240403-120118-arnaudb.json
  • 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1039
  • 11:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 11:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 11:50 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1039.eqiad.wmnet with OS bookworm
  • 11:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59322 and previous config saved to /var/cache/conftool/dbconfig/20240403-114743-marostegui.json
  • 11:46 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59321 and previous config saved to /var/cache/conftool/dbconfig/20240403-114611-arnaudb.json
  • 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1211 (T356166)', diff saved to https://phabricator.wikimedia.org/P59320 and previous config saved to /var/cache/conftool/dbconfig/20240403-114525-marostegui.json
  • 11:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1211.eqiad.wmnet with reason: Maintenance
  • 11:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59319 and previous config saved to /var/cache/conftool/dbconfig/20240403-114502-marostegui.json
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T360332)', diff saved to https://phabricator.wikimedia.org/P59318 and previous config saved to /var/cache/conftool/dbconfig/20240403-114350-arnaudb.json
  • 11:43 moritzm: installing imagemagick security updates
  • 11:43 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 11:43 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59317 and previous config saved to /var/cache/conftool/dbconfig/20240403-114327-arnaudb.json
  • 11:33 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1038.eqiad.wmnet with OS bookworm
  • 11:30 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/citoid: apply
  • 11:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59315 and previous config saved to /var/cache/conftool/dbconfig/20240403-112955-marostegui.json
  • 11:29 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/citoid: apply
  • 11:28 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/citoid: apply
  • 11:28 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59314 and previous config saved to /var/cache/conftool/dbconfig/20240403-112819-arnaudb.json
  • 11:28 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/citoid: apply
  • 11:27 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/citoid: apply
  • 11:25 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 11:24 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 11:23 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 11:23 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 11:16 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1038
  • 11:16 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1038
  • 11:16 mvolz@deploy1002: helmfile [eqiad] DONE helmfile.d/services/zotero: apply
  • 11:15 mvolz@deploy1002: helmfile [eqiad] START helmfile.d/services/zotero: apply
  • 11:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203', diff saved to https://phabricator.wikimedia.org/P59313 and previous config saved to /var/cache/conftool/dbconfig/20240403-111447-marostegui.json
  • 11:14 mvolz@deploy1002: helmfile [codfw] DONE helmfile.d/services/zotero: apply
  • 11:13 mvolz@deploy1002: helmfile [codfw] START helmfile.d/services/zotero: apply
  • 11:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59312 and previous config saved to /var/cache/conftool/dbconfig/20240403-111312-arnaudb.json
  • 11:11 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:11 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:10 fab@deploy1002: Finished deploy [airflow-dags/research@75163c7]: (no justification provided) (duration: 00m 32s)
  • 11:09 fab@deploy1002: Started deploy [airflow-dags/research@75163c7]: (no justification provided)
  • 11:08 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:08 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:07 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
  • 11:05 mvolz@deploy1002: helmfile [staging] DONE helmfile.d/services/zotero: apply
  • 11:05 mvolz@deploy1002: helmfile [staging] START helmfile.d/services/zotero: apply
  • 11:04 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1038.eqiad.wmnet with reason: host reimage
  • 10:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59311 and previous config saved to /var/cache/conftool/dbconfig/20240403-105940-marostegui.json
  • 10:58 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59310 and previous config saved to /var/cache/conftool/dbconfig/20240403-105804-arnaudb.json
  • 10:57 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1203 (T356166)', diff saved to https://phabricator.wikimedia.org/P59309 and previous config saved to /var/cache/conftool/dbconfig/20240403-105722-marostegui.json
  • 10:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 10:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1203.eqiad.wmnet with reason: Maintenance
  • 10:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59308 and previous config saved to /var/cache/conftool/dbconfig/20240403-105659-marostegui.json
  • 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T360332)', diff saved to https://phabricator.wikimedia.org/P59307 and previous config saved to /var/cache/conftool/dbconfig/20240403-105545-arnaudb.json
  • 10:55 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 10:55 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 10:55 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59306 and previous config saved to /var/cache/conftool/dbconfig/20240403-105522-arnaudb.json
  • 10:47 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:47 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1038.eqiad.wmnet with OS bookworm
  • 10:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59305 and previous config saved to /var/cache/conftool/dbconfig/20240403-104152-marostegui.json
  • 10:40 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59304 and previous config saved to /var/cache/conftool/dbconfig/20240403-104014-arnaudb.json
  • 10:29 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 10:29 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 10:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193', diff saved to https://phabricator.wikimedia.org/P59303 and previous config saved to /var/cache/conftool/dbconfig/20240403-102644-marostegui.json
  • 10:25 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59302 and previous config saved to /var/cache/conftool/dbconfig/20240403-102507-arnaudb.json
  • 10:20 logmsgbot: dreamyjazz Deployed security patch for T361293
  • 10:17 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1037.eqiad.wmnet with OS bookworm
  • 10:14 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 10:14 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 10:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59301 and previous config saved to /var/cache/conftool/dbconfig/20240403-101137-marostegui.json
  • 10:10 marostegui: Restart sanitarium db1154 T361673
  • 10:10 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59300 and previous config saved to /var/cache/conftool/dbconfig/20240403-100959-arnaudb.json
  • 10:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1193 (T356166)', diff saved to https://phabricator.wikimedia.org/P59299 and previous config saved to /var/cache/conftool/dbconfig/20240403-100919-marostegui.json
  • 10:09 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 10:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1193.eqiad.wmnet with reason: Maintenance
  • 10:09 moritzm: installing util-linux security updates
  • 10:08 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59298 and previous config saved to /var/cache/conftool/dbconfig/20240403-100857-marostegui.json
  • 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T360332)', diff saved to https://phabricator.wikimedia.org/P59297 and previous config saved to /var/cache/conftool/dbconfig/20240403-100735-arnaudb.json
  • 10:07 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 10:07 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 10:07 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59296 and previous config saved to /var/cache/conftool/dbconfig/20240403-100712-arnaudb.json
  • 10:06 logmsgbot: dreamyjazz Deployed security patch for T361293
  • 09:53 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59295 and previous config saved to /var/cache/conftool/dbconfig/20240403-095349-marostegui.json
  • 09:52 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59294 and previous config saved to /var/cache/conftool/dbconfig/20240403-095204-arnaudb.json
  • 09:48 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
  • 09:45 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1037.eqiad.wmnet with reason: host reimage
  • 09:45 Dreamy_Jazz: Doing security deploy for T361293
  • 09:43 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1037
  • 09:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59293 and previous config saved to /var/cache/conftool/dbconfig/20240403-094241-root.json
  • 09:42 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1037
  • 09:39 godog: roll-restart prometheus/k8s in codfw/eqiad to apply new retention settings - T360537
  • 09:38 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192', diff saved to https://phabricator.wikimedia.org/P59292 and previous config saved to /var/cache/conftool/dbconfig/20240403-093842-marostegui.json
  • 09:36 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59291 and previous config saved to /var/cache/conftool/dbconfig/20240403-093657-arnaudb.json
  • 09:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:27 marostegui: Restart sanitarium db1155 T361673
  • 09:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59290 and previous config saved to /var/cache/conftool/dbconfig/20240403-092735-root.json
  • 09:27 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1037.eqiad.wmnet with OS bookworm
  • 09:24 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3067.esams.wmnet
  • 09:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59289 and previous config saved to /var/cache/conftool/dbconfig/20240403-092334-marostegui.json
  • 09:21 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59288 and previous config saved to /var/cache/conftool/dbconfig/20240403-092149-arnaudb.json
  • 09:21 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1192 (T356166)', diff saved to https://phabricator.wikimedia.org/P59287 and previous config saved to /var/cache/conftool/dbconfig/20240403-092116-marostegui.json
  • 09:21 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 09:20 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1192.eqiad.wmnet with reason: Maintenance
  • 09:20 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59286 and previous config saved to /var/cache/conftool/dbconfig/20240403-092053-marostegui.json
  • 09:19 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3067.esams.wmnet with OS bullseye
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T360332)', diff saved to https://phabricator.wikimedia.org/P59285 and previous config saved to /var/cache/conftool/dbconfig/20240403-091929-arnaudb.json
  • 09:19 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 09:19 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 09:19 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59284 and previous config saved to /var/cache/conftool/dbconfig/20240403-091906-arnaudb.json
  • 09:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:18 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:12 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59283 and previous config saved to /var/cache/conftool/dbconfig/20240403-091229-root.json
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59282 and previous config saved to /var/cache/conftool/dbconfig/20240403-090545-marostegui.json
  • 09:04 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 09:03 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59281 and previous config saved to /var/cache/conftool/dbconfig/20240403-090358-arnaudb.json
  • 09:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 09:00 slyngs: Upgraded Bitu / idm.wikimedia.org to version 0.0.6-2
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59280 and previous config saved to /var/cache/conftool/dbconfig/20240403-085723-root.json
  • 08:55 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
  • 08:52 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3067.esams.wmnet with reason: host reimage
  • 08:50 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178', diff saved to https://phabricator.wikimedia.org/P59279 and previous config saved to /var/cache/conftool/dbconfig/20240403-085037-marostegui.json
  • 08:48 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59278 and previous config saved to /var/cache/conftool/dbconfig/20240403-084851-arnaudb.json
  • 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59276 and previous config saved to /var/cache/conftool/dbconfig/20240403-084218-root.json
  • 08:36 marostegui: stop sanitarium codfw hosts T361673
  • 08:35 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59275 and previous config saved to /var/cache/conftool/dbconfig/20240403-083530-marostegui.json
  • 08:33 arnaudb@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59274 and previous config saved to /var/cache/conftool/dbconfig/20240403-083343-arnaudb.json
  • 08:33 jnuche@deploy1002: Synchronized php: group1 wikis to 1.42.0-wmf.25 refs T360157 (duration: 13m 00s)
  • 08:33 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1178 (T356166)', diff saved to https://phabricator.wikimedia.org/P59273 and previous config saved to /var/cache/conftool/dbconfig/20240403-083313-marostegui.json
  • 08:33 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.hosts.decommission (exit_code=0) for hosts puppetmaster1002.eqiad.wmnet
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.dns.netbox (exit_code=0)
  • 08:33 jmm@cumin2002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 08:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1178.eqiad.wmnet with reason: Maintenance
  • 08:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59272 and previous config saved to /var/cache/conftool/dbconfig/20240403-083249-marostegui.json
  • 08:31 arnaudb@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T360332)', diff saved to https://phabricator.wikimedia.org/P59271 and previous config saved to /var/cache/conftool/dbconfig/20240403-083123-arnaudb.json
  • 08:31 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:31 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 1 day, 0:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 08:30 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 08:30 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 12:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 08:29 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3067.esams.wmnet with OS bullseye
  • 08:28 jmm@cumin2002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: puppetmaster1002.eqiad.wmnet decommissioned, removing all IPs except the asset tag one - jmm@cumin2002"
  • 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'db2125 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59270 and previous config saved to /var/cache/conftool/dbconfig/20240403-082712-root.json
  • 08:27 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3067.esams.wmnet
  • 08:24 jmm@cumin2002: START - Cookbook sre.dns.netbox
  • 08:24 fabfur: depool cp3067 for reimage (T360430)
  • 08:09 arnaudb@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.dns.netbox: db2100.codfw.wmnet decommissioned, removing all IPs except the asset tag one - arnaudb@cumin1002"
  • 08:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177', diff saved to https://phabricator.wikimedia.org/P59267 and previous config saved to /var/cache/conftool/dbconfig/20240403-080235-marostegui.json
  • 08:01 arnaudb@cumin1002: START - Cookbook sre.dns.netbox
  • 07:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2125.codfw.wmnet with OS bookworm
  • 07:55 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59266 and previous config saved to /var/cache/conftool/dbconfig/20240403-075510-root.json
  • 07:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2202.codfw.wmnet with OS bookworm
  • 07:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59265 and previous config saved to /var/cache/conftool/dbconfig/20240403-074727-marostegui.json
  • 07:45 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1177 (T356166)', diff saved to https://phabricator.wikimedia.org/P59264 and previous config saved to /var/cache/conftool/dbconfig/20240403-074509-marostegui.json
  • 07:45 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 07:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1177.eqiad.wmnet with reason: Maintenance
  • 07:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59263 and previous config saved to /var/cache/conftool/dbconfig/20240403-074446-marostegui.json
  • 07:40 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59262 and previous config saved to /var/cache/conftool/dbconfig/20240403-074004-root.json
  • 07:35 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
  • 07:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
  • 07:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59261 and previous config saved to /var/cache/conftool/dbconfig/20240403-072938-marostegui.json
  • 07:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2125.codfw.wmnet with reason: host reimage
  • 07:27 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2202.codfw.wmnet with reason: host reimage
  • 07:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59260 and previous config saved to /var/cache/conftool/dbconfig/20240403-072459-root.json
  • 07:18 arnaudb@cumin1002: START - Cookbook sre.hosts.decommission for hosts db2100.codfw.wmnet
  • 07:17 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59259 and previous config saved to /var/cache/conftool/dbconfig/20240403-071718-root.json
  • 07:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172', diff saved to https://phabricator.wikimedia.org/P59258 and previous config saved to /var/cache/conftool/dbconfig/20240403-071431-marostegui.json
  • 07:11 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db2202.codfw.wmnet with OS bookworm
  • 07:11 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2125.codfw.wmnet with OS bookworm
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59257 and previous config saved to /var/cache/conftool/dbconfig/20240403-070953-root.json
  • 07:09 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2125 T361543', diff saved to https://phabricator.wikimedia.org/P59256 and previous config saved to /var/cache/conftool/dbconfig/20240403-070946-root.json
  • 07:02 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59255 and previous config saved to /var/cache/conftool/dbconfig/20240403-070212-root.json
  • 06:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59254 and previous config saved to /var/cache/conftool/dbconfig/20240403-065923-marostegui.json
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1172 (T356166)', diff saved to https://phabricator.wikimedia.org/P59253 and previous config saved to /var/cache/conftool/dbconfig/20240403-065706-marostegui.json
  • 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1172.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1171.eqiad.wmnet with reason: Maintenance
  • 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59252 and previous config saved to /var/cache/conftool/dbconfig/20240403-065617-marostegui.json
  • 06:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59251 and previous config saved to /var/cache/conftool/dbconfig/20240403-065447-root.json
  • 06:47 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59250 and previous config saved to /var/cache/conftool/dbconfig/20240403-064704-root.json
  • 06:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59249 and previous config saved to /var/cache/conftool/dbconfig/20240403-064110-marostegui.json
  • 06:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59248 and previous config saved to /var/cache/conftool/dbconfig/20240403-063941-root.json
  • 06:31 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59247 and previous config saved to /var/cache/conftool/dbconfig/20240403-063159-root.json
  • 06:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167', diff saved to https://phabricator.wikimedia.org/P59246 and previous config saved to /var/cache/conftool/dbconfig/20240403-062602-marostegui.json
  • 06:25 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db2148.codfw.wmnet with OS bookworm
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2148 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59245 and previous config saved to /var/cache/conftool/dbconfig/20240403-062436-root.json
  • 06:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 06:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 06:16 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59244 and previous config saved to /var/cache/conftool/dbconfig/20240403-061653-root.json
  • 06:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 06:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 06:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59243 and previous config saved to /var/cache/conftool/dbconfig/20240403-061055-marostegui.json
  • 06:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
  • 06:01 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59242 and previous config saved to /var/cache/conftool/dbconfig/20240403-060147-root.json
  • 06:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db2148.codfw.wmnet with reason: host reimage
  • 05:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 05:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 05:49 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1222.eqiad.wmnet with OS bookworm
  • 05:46 marostegui@cumin1002: dbctl commit (dc=all): 'db1222 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59241 and previous config saved to /var/cache/conftool/dbconfig/20240403-054641-root.json
  • 05:44 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db2148.codfw.wmnet with OS bookworm
  • 05:43 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db2148 T361543', diff saved to https://phabricator.wikimedia.org/P59240 and previous config saved to /var/cache/conftool/dbconfig/20240403-054310-root.json
  • 05:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
  • 05:25 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1222.eqiad.wmnet with reason: host reimage
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1222.eqiad.wmnet with OS bookworm
  • 05:11 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1222 T361543', diff saved to https://phabricator.wikimedia.org/P59239 and previous config saved to /var/cache/conftool/dbconfig/20240403-051149-root.json
  • 05:10 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1167 (T356166)', diff saved to https://phabricator.wikimedia.org/P59238 and previous config saved to /var/cache/conftool/dbconfig/20240403-051029-marostegui.json
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:10 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 05:10 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 05:09 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1167.eqiad.wmnet with reason: Maintenance
  • 01:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:51 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:51 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:40 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2002.codfw.wmnet with OS bookworm
  • 01:19 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
  • 01:15 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2002.codfw.wmnet with reason: host reimage
  • 01:06 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:06 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 01:00 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 01:00 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:46 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
  • 00:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:43 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2002.codfw.wmnet with OS bookworm
  • 00:37 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:25 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2003.codfw.wmnet with OS bookworm
  • 00:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 00:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 00:03 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage

2024-04-02

  • 23:59 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2003.codfw.wmnet with reason: host reimage
  • 23:59 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:59 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:50 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2002.codfw.wmnet with OS bookworm
  • 23:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:30 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
  • 23:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:28 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2003.codfw.wmnet with OS bookworm
  • 23:22 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:22 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 23:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 23:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:57 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:35 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:35 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:33 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:33 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:31 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2003.codfw.wmnet with OS bookworm
  • 22:29 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:29 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:28 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 22:27 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:27 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:25 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:25 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:23 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:23 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:21 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:21 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:19 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:19 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:17 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:17 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:15 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:15 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:13 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:13 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:11 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:11 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:09 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:08 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:08 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:07 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:05 cwhite@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
  • 22:03 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:03 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 22:02 cwhite@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on logging-hd2001.codfw.wmnet with reason: host reimage
  • 22:01 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 22:01 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:58 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:56 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:56 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:54 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:54 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:52 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:52 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:50 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:49 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:48 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:48 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:46 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:44 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:42 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:42 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:40 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:40 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:36 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:34 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:32 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 21:31 cwhite@cumin2002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 21:30 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:30 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:28 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:28 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:26 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:24 logmsgbot: @deploy1002 helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 21:24 logmsgbot: @deploy1002 helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
  • 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
  • 21:21 bking@cumin2002: END (PASS) - Cookbook sre.ganeti.resource-report (exit_code=0)
  • 21:21 bking@cumin2002: START - Cookbook sre.ganeti.resource-report
  • 21:16 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 20:46 mutante: DNS - added new project language 'igl' - Igala is a Yoruboid language, spoken by the Igala ethnic group of Nigeria (800,000 speakers) T361644
  • 20:13 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 16s)
  • 20:13 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 20:07 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 46s)
  • 20:06 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 20:06 sfaci@deploy1002: Finished deploy [airflow-dags/analytics@75163c7]: (no justification provided) (duration: 00m 54s)
  • 20:05 sfaci@deploy1002: Started deploy [airflow-dags/analytics@75163c7]: (no justification provided)
  • 18:48 cstone: civicrm upgraded from ed776060 to 5b7f9e06
  • 17:26 cwhite@cumin2002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host logging-hd2001.codfw.wmnet with OS bookworm
  • 17:24 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1007.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1240.eqiad.wmnet with reason: Maintenance
  • 17:19 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59232 and previous config saved to /var/cache/conftool/dbconfig/20240402-171935-marostegui.json
  • 17:13 Dreamy_Jazz: Creating cu_useragent table on WMF wikis - T359312
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59230 and previous config saved to /var/cache/conftool/dbconfig/20240402-170625-arnaudb.json
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59229 and previous config saved to /var/cache/conftool/dbconfig/20240402-170610-arnaudb.json
  • 17:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59228 and previous config saved to /var/cache/conftool/dbconfig/20240402-170603-arnaudb.json
  • 17:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 100%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59227 and previous config saved to /var/cache/conftool/dbconfig/20240402-170555-arnaudb.json
  • 17:04 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59226 and previous config saved to /var/cache/conftool/dbconfig/20240402-170427-marostegui.json
  • 16:59 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:59 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:56 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59225 and previous config saved to /var/cache/conftool/dbconfig/20240402-165601-arnaudb.json
  • 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59224 and previous config saved to /var/cache/conftool/dbconfig/20240402-165119-arnaudb.json
  • 16:51 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59223 and previous config saved to /var/cache/conftool/dbconfig/20240402-165105-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59222 and previous config saved to /var/cache/conftool/dbconfig/20240402-165058-arnaudb.json
  • 16:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 75%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59221 and previous config saved to /var/cache/conftool/dbconfig/20240402-165049-arnaudb.json
  • 16:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223', diff saved to https://phabricator.wikimedia.org/P59220 and previous config saved to /var/cache/conftool/dbconfig/20240402-164920-marostegui.json
  • 16:45 jgiannelos@deploy1002: helmfile [staging] DONE helmfile.d/services/mobileapps: apply
  • 16:44 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 16:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59219 and previous config saved to /var/cache/conftool/dbconfig/20240402-164055-arnaudb.json
  • 16:40 jgiannelos@deploy1002: helmfile [staging] START helmfile.d/services/mobileapps: apply
  • 16:38 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:38 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59218 and previous config saved to /var/cache/conftool/dbconfig/20240402-163613-arnaudb.json
  • 16:36 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59217 and previous config saved to /var/cache/conftool/dbconfig/20240402-163559-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59216 and previous config saved to /var/cache/conftool/dbconfig/20240402-163552-arnaudb.json
  • 16:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 50%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59215 and previous config saved to /var/cache/conftool/dbconfig/20240402-163544-arnaudb.json
  • 16:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59214 and previous config saved to /var/cache/conftool/dbconfig/20240402-163413-marostegui.json
  • 16:28 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:27 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59213 and previous config saved to /var/cache/conftool/dbconfig/20240402-162550-arnaudb.json
  • 16:22 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1036.eqiad.wmnet with OS bookworm
  • 16:21 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59212 and previous config saved to /var/cache/conftool/dbconfig/20240402-162107-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59211 and previous config saved to /var/cache/conftool/dbconfig/20240402-162053-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59210 and previous config saved to /var/cache/conftool/dbconfig/20240402-162046-arnaudb.json
  • 16:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 25%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59209 and previous config saved to /var/cache/conftool/dbconfig/20240402-162038-arnaudb.json
  • 16:13 vriley@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 16:11 vriley@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1006.eqiad.wmnet with reason: host reimage
  • 16:10 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59208 and previous config saved to /var/cache/conftool/dbconfig/20240402-161044-arnaudb.json
  • 16:06 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59207 and previous config saved to /var/cache/conftool/dbconfig/20240402-160602-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59206 and previous config saved to /var/cache/conftool/dbconfig/20240402-160547-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59205 and previous config saved to /var/cache/conftool/dbconfig/20240402-160540-arnaudb.json
  • 16:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 16%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59204 and previous config saved to /var/cache/conftool/dbconfig/20240402-160532-arnaudb.json
  • 16:04 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 16:04 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 16:02 jnuche@deploy1002: Finished scap: Backport for rest: add default null to nullable typed prop (T361577) (duration: 14m 39s)
  • 16:02 cwhite@cumin2002: START - Cookbook sre.hosts.reimage for host logging-hd2001.codfw.wmnet with OS bookworm
  • 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:57 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 15:55 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1036
  • 15:55 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59203 and previous config saved to /var/cache/conftool/dbconfig/20240402-155538-arnaudb.json
  • 15:55 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1036
  • 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:54 pfischer@deploy1002: helmfile [eqiad] DONE helmfile.d/services/cirrus-streaming-updater: apply
  • 15:54 pfischer@deploy1002: helmfile [eqiad] START helmfile.d/services/cirrus-streaming-updater: apply
  • 15:51 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59202 and previous config saved to /var/cache/conftool/dbconfig/20240402-155056-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59201 and previous config saved to /var/cache/conftool/dbconfig/20240402-155042-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59200 and previous config saved to /var/cache/conftool/dbconfig/20240402-155035-arnaudb.json
  • 15:50 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 8%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59199 and previous config saved to /var/cache/conftool/dbconfig/20240402-155026-arnaudb.json
  • 15:50 jnuche@deploy1002: samtar and jnuche: Continuing with sync
  • 15:50 jnuche@deploy1002: samtar and jnuche: Backport for rest: add default null to nullable typed prop (T361577) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 15:49 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1036.eqiad.wmnet with reason: host reimage
  • 15:47 jnuche@deploy1002: Started scap: Backport for rest: add default null to nullable typed prop (T361577)
  • 15:40 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59198 and previous config saved to /var/cache/conftool/dbconfig/20240402-154033-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59197 and previous config saved to /var/cache/conftool/dbconfig/20240402-153550-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59196 and previous config saved to /var/cache/conftool/dbconfig/20240402-153536-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59195 and previous config saved to /var/cache/conftool/dbconfig/20240402-153529-arnaudb.json
  • 15:35 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 4%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59194 and previous config saved to /var/cache/conftool/dbconfig/20240402-153521-arnaudb.json
  • 15:31 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1036.eqiad.wmnet with OS bookworm
  • 15:25 arnaudb@cumin1002: dbctl commit (dc=all): 'db1230 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59193 and previous config saved to /var/cache/conftool/dbconfig/20240402-152527-arnaudb.json
  • 15:23 jgiannelos@deploy1002: Finished deploy [restbase/deploy@c4d19d7]: (no justification provided) (duration: 00m 16s)
  • 15:23 jgiannelos@deploy1002: Started deploy [restbase/deploy@c4d19d7]: (no justification provided)
  • 15:21 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1230.eqiad.wmnet with OS bookworm
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59192 and previous config saved to /var/cache/conftool/dbconfig/20240402-152044-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59191 and previous config saved to /var/cache/conftool/dbconfig/20240402-152031-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59190 and previous config saved to /var/cache/conftool/dbconfig/20240402-152023-arnaudb.json
  • 15:20 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 2%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59189 and previous config saved to /var/cache/conftool/dbconfig/20240402-152015-arnaudb.json
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1223 (T356166)', diff saved to https://phabricator.wikimedia.org/P59188 and previous config saved to /var/cache/conftool/dbconfig/20240402-151235-marostegui.json
  • 15:12 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1223.eqiad.wmnet with reason: Maintenance
  • 15:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59187 and previous config saved to /var/cache/conftool/dbconfig/20240402-151213-marostegui.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2220 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59186 and previous config saved to /var/cache/conftool/dbconfig/20240402-150538-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2219 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59185 and previous config saved to /var/cache/conftool/dbconfig/20240402-150525-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2215 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59184 and previous config saved to /var/cache/conftool/dbconfig/20240402-150516-arnaudb.json
  • 15:05 arnaudb@cumin1002: dbctl commit (dc=all): 'db2214 (re)pooling @ 1%: Post clone repool (dst)', diff saved to https://phabricator.wikimedia.org/P59183 and previous config saved to /var/cache/conftool/dbconfig/20240402-150509-arnaudb.json
  • 15:01 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
  • 14:57 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: host reimage
  • 14:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59182 and previous config saved to /var/cache/conftool/dbconfig/20240402-145705-marostegui.json
  • 14:56 moritzm: installing mariadb security updates (as packaged in Debian, unrelated to the wmf-mariadb packages)
  • 14:45 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1230.eqiad.wmnet with OS bookworm
  • 14:42 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
  • 14:42 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1230.eqiad.wmnet with reason: Silence for reimaging
  • 14:42 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1230', diff saved to https://phabricator.wikimedia.org/P59181 and previous config saved to /var/cache/conftool/dbconfig/20240402-144221-arnaudb.json
  • 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212', diff saved to https://phabricator.wikimedia.org/P59180 and previous config saved to /var/cache/conftool/dbconfig/20240402-144158-marostegui.json
  • 14:38 fabfur@cumin1002: conftool action : set/pooled=yes; selector: name=cp3066.esams.wmnet
  • 14:38 fabfur: repooling cp3066 after reimage (T360430)
  • 14:29 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 100%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59179 and previous config saved to /var/cache/conftool/dbconfig/20240402-142859-arnaudb.json
  • 14:28 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp3066.esams.wmnet with OS bullseye
  • 14:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59178 and previous config saved to /var/cache/conftool/dbconfig/20240402-142650-marostegui.json
  • 14:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59177 and previous config saved to /var/cache/conftool/dbconfig/20240402-141541-root.json
  • 14:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 75%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59176 and previous config saved to /var/cache/conftool/dbconfig/20240402-141353-arnaudb.json
  • 14:09 moritzm: imported jenkins 2.440.2 to thirdparty/ci for buster-wikimedia T360759
  • 14:05 fabfur@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
  • 14:02 fabfur@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp3066.esams.wmnet with reason: host reimage
  • 14:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59174 and previous config saved to /var/cache/conftool/dbconfig/20240402-140035-root.json
  • 13:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 50%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59173 and previous config saved to /var/cache/conftool/dbconfig/20240402-135847-arnaudb.json
  • 13:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59172 and previous config saved to /var/cache/conftool/dbconfig/20240402-134528-root.json
  • 13:43 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 25%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59171 and previous config saved to /var/cache/conftool/dbconfig/20240402-134342-arnaudb.json
  • 13:38 TheresNoTime: closing UTC afternoon backport window
  • 13:38 fabfur@cumin1002: START - Cookbook sre.hosts.reimage for host cp3066.esams.wmnet with OS bullseye
  • 13:37 samtar@deploy1002: Finished scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) (duration: 16m 26s)
  • 13:33 fabfur@cumin1002: conftool action : set/pooled=no; selector: name=cp3066.esams.wmnet
  • 13:32 fabfur: depool cp3066 for reimage (T360430)
  • 13:30 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59170 and previous config saved to /var/cache/conftool/dbconfig/20240402-133023-root.json
  • 13:28 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 15%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59169 and previous config saved to /var/cache/conftool/dbconfig/20240402-132836-arnaudb.json
  • 13:25 samtar@deploy1002: samtar: Continuing with sync
  • 13:23 samtar@deploy1002: samtar: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:21 samtar@deploy1002: Started scap: Backport for InitialiseSettings: Enable Edit Recovery on all projects (T355548)
  • 13:18 dreamyjazz@deploy1002: Finished scap: Backport for Deploy partial action blocks everywhere (T353496) (duration: 15m 33s)
  • 13:15 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59168 and previous config saved to /var/cache/conftool/dbconfig/20240402-131517-root.json
  • 13:13 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 10%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59167 and previous config saved to /var/cache/conftool/dbconfig/20240402-131330-arnaudb.json
  • 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Continuing with sync
  • 13:05 dreamyjazz@deploy1002: dreamyjazz and tchanders: Backport for Deploy partial action blocks everywhere (T353496) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 13:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1212 (T356166)', diff saved to https://phabricator.wikimedia.org/P59166 and previous config saved to /var/cache/conftool/dbconfig/20240402-130423-marostegui.json
  • 13:04 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:04 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1013,1017,1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1212.eqiad.wmnet with reason: Maintenance
  • 13:03 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59165 and previous config saved to /var/cache/conftool/dbconfig/20240402-130341-marostegui.json
  • 13:02 dreamyjazz@deploy1002: Started scap: Backport for Deploy partial action blocks everywhere (T353496)
  • 13:00 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59164 and previous config saved to /var/cache/conftool/dbconfig/20240402-130012-root.json
  • 12:58 arnaudb@cumin1002: dbctl commit (dc=all): 'db1185 (re)pooling @ 5%: Post reimage repool', diff saved to https://phabricator.wikimedia.org/P59163 and previous config saved to /var/cache/conftool/dbconfig/20240402-125825-arnaudb.json
  • 12:48 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59162 and previous config saved to /var/cache/conftool/dbconfig/20240402-124834-marostegui.json
  • 12:45 marostegui@cumin1002: dbctl commit (dc=all): 'db1188 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59161 and previous config saved to /var/cache/conftool/dbconfig/20240402-124506-root.json
  • 12:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1188.eqiad.wmnet with OS bookworm
  • 12:39 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cloudvirt1035.eqiad.wmnet with OS bookworm
  • 12:33 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198', diff saved to https://phabricator.wikimedia.org/P59160 and previous config saved to /var/cache/conftool/dbconfig/20240402-123326-marostegui.json
  • 12:28 taavi: taavi@deploy1002 ~ $ sudo systemctl kill train-presync.service # T361580
  • 12:22 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
  • 12:20 hnowlan@deploy2002: helmfile [eqiad] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:19 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1188.eqiad.wmnet with reason: host reimage
  • 12:19 hnowlan@deploy2002: helmfile [eqiad] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:18 hnowlan@deploy2002: helmfile [eqiad] [main] START helmfile.d/services/mw-jobrunner : sync
  • 12:18 hnowlan@deploy2002: helmfile [eqiad] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 12:18 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59159 and previous config saved to /var/cache/conftool/dbconfig/20240402-121819-marostegui.json
  • 12:13 moritzm: installing pillow security updates
  • 12:13 hnowlan@deploy2002: helmfile [codfw] [main] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:12 hnowlan@deploy2002: helmfile [codfw] [canary] DONE helmfile.d/services/mw-jobrunner : sync
  • 12:11 hnowlan@deploy2002: helmfile [codfw] [main] START helmfile.d/services/mw-jobrunner : sync
  • 12:11 hnowlan@deploy2002: helmfile [codfw] [canary] START helmfile.d/services/mw-jobrunner : sync
  • 12:09 aborrero@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
  • 12:07 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1188.eqiad.wmnet with OS bookworm
  • 12:06 aborrero@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cloudvirt1035.eqiad.wmnet with reason: host reimage
  • 12:04 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1188 T361543', diff saved to https://phabricator.wikimedia.org/P59158 and previous config saved to /var/cache/conftool/dbconfig/20240402-120455-root.json
  • 11:58 aborrero@cumin1002: END (PASS) - Cookbook sre.network.configure-switch-interfaces (exit_code=0) for host cloudvirt1035
  • 11:58 aborrero@cumin1002: START - Cookbook sre.network.configure-switch-interfaces for host cloudvirt1035
  • 11:49 aborrero@cumin1002: START - Cookbook sre.hosts.reimage for host cloudvirt1035.eqiad.wmnet with OS bookworm
  • 11:33 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/editor-analytics: apply
  • 11:32 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/editor-analytics: apply
  • 11:31 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/editor-analytics: apply
  • 11:31 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/editor-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [eqiad] DONE helmfile.d/services/edit-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [eqiad] START helmfile.d/services/edit-analytics: apply
  • 11:29 sfaci@deploy1002: helmfile [codfw] DONE helmfile.d/services/edit-analytics: apply
  • 11:28 sfaci@deploy1002: helmfile [codfw] START helmfile.d/services/edit-analytics: apply
  • 11:25 sfaci@deploy1002: helmfile [staging] DONE helmfile.d/services/edit-analytics: apply
  • 11:25 sfaci@deploy1002: helmfile [staging] START helmfile.d/services/edit-analytics: apply
  • 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1198 (T356166)', diff saved to https://phabricator.wikimedia.org/P59157 and previous config saved to /var/cache/conftool/dbconfig/20240402-110122-marostegui.json
  • 11:01 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:01 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1198.eqiad.wmnet with reason: Maintenance
  • 11:01 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59156 and previous config saved to /var/cache/conftool/dbconfig/20240402-110100-marostegui.json
  • 10:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59155 and previous config saved to /var/cache/conftool/dbconfig/20240402-104552-marostegui.json
  • 10:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189', diff saved to https://phabricator.wikimedia.org/P59154 and previous config saved to /var/cache/conftool/dbconfig/20240402-103045-marostegui.json
  • 10:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59153 and previous config saved to /var/cache/conftool/dbconfig/20240402-102951-root.json
  • 10:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59152 and previous config saved to /var/cache/conftool/dbconfig/20240402-101538-marostegui.json
  • 10:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59151 and previous config saved to /var/cache/conftool/dbconfig/20240402-101445-root.json
  • 09:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59150 and previous config saved to /var/cache/conftool/dbconfig/20240402-095939-root.json
  • 09:53 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1185.eqiad.wmnet with OS bookworm
  • 09:46 arnaudb@cumin1002: END (PASS) - Cookbook sre.mysql.clone (exit_code=0) Will create a clone of db2115.codfw.wmnet onto db2215.codfw.wmnet
  • 09:44 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59149 and previous config saved to /var/cache/conftool/dbconfig/20240402-094433-root.json
  • 09:32 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
  • 09:29 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59148 and previous config saved to /var/cache/conftool/dbconfig/20240402-092928-root.json
  • 09:28 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: host reimage
  • 09:22 jnuche@deploy1002: rebuilt and synchronized wikiversions files: group0 wikis to 1.42.0-wmf.25 refs T360157
  • 09:15 arnaudb@cumin1002: START - Cookbook sre.hosts.reimage for host db1185.eqiad.wmnet with OS bookworm
  • 09:14 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59147 and previous config saved to /var/cache/conftool/dbconfig/20240402-091422-root.json
  • 09:14 arnaudb@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
  • 09:13 arnaudb@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1185.eqiad.wmnet with reason: Silence for reimaging
  • 09:13 arnaudb@cumin1002: dbctl commit (dc=all): 'Depool to reimage db1185', diff saved to https://phabricator.wikimedia.org/P59146 and previous config saved to /var/cache/conftool/dbconfig/20240402-091303-arnaudb.json
  • 09:02 jnuche@deploy1002: Finished scap: testwikis wikis to 1.42.0-wmf.25 refs T360157 (duration: 51m 03s)
  • 08:59 marostegui@cumin1002: dbctl commit (dc=all): 'db1197 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59145 and previous config saved to /var/cache/conftool/dbconfig/20240402-085917-root.json
  • 08:58 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1189 (T356166)', diff saved to https://phabricator.wikimedia.org/P59144 and previous config saved to /var/cache/conftool/dbconfig/20240402-085814-marostegui.json
  • 08:58 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1189.eqiad.wmnet with reason: Maintenance
  • 08:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59143 and previous config saved to /var/cache/conftool/dbconfig/20240402-085752-marostegui.json
  • 08:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1197.eqiad.wmnet with OS bookworm
  • 08:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59142 and previous config saved to /var/cache/conftool/dbconfig/20240402-084244-marostegui.json
  • 08:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
  • 08:28 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1197.eqiad.wmnet with reason: host reimage
  • 08:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1175', diff saved to https://phabricator.wikimedia.org/P59141 and previous config saved to /var/cache/conftool/dbconfig/20240402-082737-marostegui.json
  • 08:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59137 and previous config saved to /var/cache/conftool/dbconfig/20240402-080642-root.json
  • 08:02 godog: restore SRE business hours routing/escalation after the holidays - T350192
  • 07:51 ayounsi@cumin1002: END (PASS) - Cookbook sre.network.peering (exit_code=0) with action 'email' for AS: 38278
  • 07:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59134 and previous config saved to /var/cache/conftool/dbconfig/20240402-075136-root.json
  • 07:49 ayounsi@cumin1002: START - Cookbook sre.network.peering with action 'email' for AS: 38278
  • 07:47 hashar: UTC morning backport window completed
  • 07:46 hashar@deploy1002: Finished scap: Backport for webrequest: disable canary events. (T314956 T351117) (duration: 34m 03s)
  • 07:39 taavi: update firewall policy on cr-eqiad, cr-codfw T361537
  • 07:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59133 and previous config saved to /var/cache/conftool/dbconfig/20240402-073631-root.json
  • 07:30 hashar@deploy1002: gmodena and hashar: Continuing with sync
  • 07:28 hashar@deploy1002: gmodena and hashar: Backport for webrequest: disable canary events. (T314956 T351117) synced to the testservers (https://wikitech.wikimedia.org/wiki/Mwdebug)
  • 07:21 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59132 and previous config saved to /var/cache/conftool/dbconfig/20240402-072125-root.json
  • 07:12 hashar@deploy1002: Started scap: Backport for webrequest: disable canary events. (T314956 T351117)
  • 07:06 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59131 and previous config saved to /var/cache/conftool/dbconfig/20240402-070619-root.json
  • 07:03 moritzm: installing util-linux security updates
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1175 (T356166)', diff saved to https://phabricator.wikimedia.org/P59130 and previous config saved to /var/cache/conftool/dbconfig/20240402-065751-marostegui.json
  • 06:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 06:57 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1175.eqiad.wmnet with reason: Maintenance
  • 06:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59129 and previous config saved to /var/cache/conftool/dbconfig/20240402-065728-marostegui.json
  • 06:51 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59128 and previous config saved to /var/cache/conftool/dbconfig/20240402-065113-root.json
  • 06:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59127 and previous config saved to /var/cache/conftool/dbconfig/20240402-064221-marostegui.json
  • 06:36 marostegui@cumin1002: dbctl commit (dc=all): 'db1229 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59126 and previous config saved to /var/cache/conftool/dbconfig/20240402-063607-root.json
  • 06:35 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59125 and previous config saved to /var/cache/conftool/dbconfig/20240402-063510-root.json
  • 06:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166', diff saved to https://phabricator.wikimedia.org/P59124 and previous config saved to /var/cache/conftool/dbconfig/20240402-062713-marostegui.json
  • 06:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 100%: Repooling', diff saved to https://phabricator.wikimedia.org/P59123 and previous config saved to /var/cache/conftool/dbconfig/20240402-062427-root.json
  • 06:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host db1229.eqiad.wmnet with OS bookworm
  • 06:20 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59122 and previous config saved to /var/cache/conftool/dbconfig/20240402-062004-root.json
  • 06:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59121 and previous config saved to /var/cache/conftool/dbconfig/20240402-061206-marostegui.json
  • 06:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 75%: Repooling', diff saved to https://phabricator.wikimedia.org/P59120 and previous config saved to /var/cache/conftool/dbconfig/20240402-060922-root.json
  • 06:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59119 and previous config saved to /var/cache/conftool/dbconfig/20240402-060459-root.json
  • 06:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
  • 05:59 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on db1229.eqiad.wmnet with reason: host reimage
  • 05:54 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 50%: Repooling', diff saved to https://phabricator.wikimedia.org/P59118 and previous config saved to /var/cache/conftool/dbconfig/20240402-055416-root.json
  • 05:49 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59117 and previous config saved to /var/cache/conftool/dbconfig/20240402-054953-root.json
  • 05:46 marostegui@cumin1002: START - Cookbook sre.hosts.reimage for host db1229.eqiad.wmnet with OS bookworm
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depool db1229 T361543', diff saved to https://phabricator.wikimedia.org/P59116 and previous config saved to /var/cache/conftool/dbconfig/20240402-054408-root.json
  • 05:39 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 25%: Repooling', diff saved to https://phabricator.wikimedia.org/P59115 and previous config saved to /var/cache/conftool/dbconfig/20240402-053910-root.json
  • 05:34 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59114 and previous config saved to /var/cache/conftool/dbconfig/20240402-053447-root.json
  • 05:24 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 10%: Repooling', diff saved to https://phabricator.wikimedia.org/P59113 and previous config saved to /var/cache/conftool/dbconfig/20240402-052404-root.json
  • 05:19 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59112 and previous config saved to /var/cache/conftool/dbconfig/20240402-051942-root.json
  • 05:09 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 5%: Repooling', diff saved to https://phabricator.wikimedia.org/P59111 and previous config saved to /var/cache/conftool/dbconfig/20240402-050859-root.json
  • 05:04 marostegui@cumin1002: dbctl commit (dc=all): 'es1024 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59110 and previous config saved to /var/cache/conftool/dbconfig/20240402-050436-root.json
  • 04:57 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1166 (T356166)', diff saved to https://phabricator.wikimedia.org/P59109 and previous config saved to /var/cache/conftool/dbconfig/20240402-045716-marostegui.json
  • 04:57 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:56 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1166.eqiad.wmnet with reason: Maintenance
  • 04:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depool es1024 T358746', diff saved to https://phabricator.wikimedia.org/P59108 and previous config saved to /var/cache/conftool/dbconfig/20240402-045559-root.json
  • 04:53 marostegui@cumin1002: dbctl commit (dc=all): 'db2115 (re)pooling @ 1%: Repooling', diff saved to https://phabricator.wikimedia.org/P59107 and previous config saved to /var/cache/conftool/dbconfig/20240402-045353-root.json
  • 04:53 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 04:53 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1150.eqiad.wmnet with reason: Maintenance
  • 03:03 mwpresync@deploy1002: Pruned MediaWiki: 1.42.0-wmf.22 (duration: 03m 20s)

2024-04-01

  • 23:14 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 22:41 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 22:41 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1008.eqiad.wmnet with reason: Maintenance
  • 22:31 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 22:31 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1245.eqiad.wmnet with reason: Maintenance
  • 22:30 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59105 and previous config saved to /var/cache/conftool/dbconfig/20240401-223055-marostegui.json
  • 22:15 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59104 and previous config saved to /var/cache/conftool/dbconfig/20240401-221548-marostegui.json
  • 22:00 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230', diff saved to https://phabricator.wikimedia.org/P59103 and previous config saved to /var/cache/conftool/dbconfig/20240401-220040-marostegui.json
  • 21:53 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 21:45 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59102 and previous config saved to /var/cache/conftool/dbconfig/20240401-214532-marostegui.json
  • 21:38 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1230 (T356166)', diff saved to https://phabricator.wikimedia.org/P59101 and previous config saved to /var/cache/conftool/dbconfig/20240401-213834-marostegui.json
  • 21:38 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 21:38 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1230.eqiad.wmnet with reason: Maintenance
  • 21:28 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 21:27 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1216.eqiad.wmnet with reason: Maintenance
  • 21:27 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59100 and previous config saved to /var/cache/conftool/dbconfig/20240401-212751-marostegui.json
  • 21:12 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59099 and previous config saved to /var/cache/conftool/dbconfig/20240401-211244-marostegui.json
  • 21:03 vriley@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1006.eqiad.wmnet with OS bullseye
  • 20:57 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213', diff saved to https://phabricator.wikimedia.org/P59098 and previous config saved to /var/cache/conftool/dbconfig/20240401-205736-marostegui.json
  • 20:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59097 and previous config saved to /var/cache/conftool/dbconfig/20240401-204229-marostegui.json
  • 20:36 bking@cumin2002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host elastic2088.codfw.wmnet with OS bullseye
  • 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1213 (T356166)', diff saved to https://phabricator.wikimedia.org/P59096 and previous config saved to /var/cache/conftool/dbconfig/20240401-203254-marostegui.json
  • 20:32 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 20:32 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1213.eqiad.wmnet with reason: Maintenance
  • 20:32 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59095 and previous config saved to /var/cache/conftool/dbconfig/20240401-203232-marostegui.json
  • 20:19 bking@cumin2002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
  • 20:17 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59094 and previous config saved to /var/cache/conftool/dbconfig/20240401-201725-marostegui.json
  • 20:17 bking@cumin2002: START - Cookbook sre.hosts.downtime for 2:00:00 on elastic2088.codfw.wmnet with reason: host reimage
  • 20:02 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210', diff saved to https://phabricator.wikimedia.org/P59093 and previous config saved to /var/cache/conftool/dbconfig/20240401-200217-marostegui.json
  • 20:00 bking@cumin2002: START - Cookbook sre.hosts.reimage for host elastic2088.codfw.wmnet with OS bullseye
  • 19:47 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59092 and previous config saved to /var/cache/conftool/dbconfig/20240401-194709-marostegui.json
  • 19:42 vriley@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1006.eqiad.wmnet with OS bullseye
  • 19:37 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1210 (T356166)', diff saved to https://phabricator.wikimedia.org/P59091 and previous config saved to /var/cache/conftool/dbconfig/20240401-193713-marostegui.json
  • 19:37 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1210.eqiad.wmnet with reason: Maintenance
  • 19:36 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59090 and previous config saved to /var/cache/conftool/dbconfig/20240401-193650-marostegui.json
  • 19:21 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59089 and previous config saved to /var/cache/conftool/dbconfig/20240401-192143-marostegui.json
  • 19:06 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200', diff saved to https://phabricator.wikimedia.org/P59088 and previous config saved to /var/cache/conftool/dbconfig/20240401-190635-marostegui.json
  • 18:51 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59087 and previous config saved to /var/cache/conftool/dbconfig/20240401-185128-marostegui.json
  • 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1200 (T356166)', diff saved to https://phabricator.wikimedia.org/P59086 and previous config saved to /var/cache/conftool/dbconfig/20240401-184455-marostegui.json
  • 18:44 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1200.eqiad.wmnet with reason: Maintenance
  • 18:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59085 and previous config saved to /var/cache/conftool/dbconfig/20240401-184432-marostegui.json
  • 18:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59084 and previous config saved to /var/cache/conftool/dbconfig/20240401-182924-marostegui.json
  • 18:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185', diff saved to https://phabricator.wikimedia.org/P59083 and previous config saved to /var/cache/conftool/dbconfig/20240401-181417-marostegui.json
  • 17:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59082 and previous config saved to /var/cache/conftool/dbconfig/20240401-175910-marostegui.json
  • 17:58 mutante: LDAP - removed uid migr from groups nda and wmde (T361266)
  • 17:53 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1185 (T356166)', diff saved to https://phabricator.wikimedia.org/P59081 and previous config saved to /var/cache/conftool/dbconfig/20240401-175300-marostegui.json
  • 17:52 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 17:52 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1185.eqiad.wmnet with reason: Maintenance
  • 17:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59080 and previous config saved to /var/cache/conftool/dbconfig/20240401-175237-marostegui.json
  • 17:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59079 and previous config saved to /var/cache/conftool/dbconfig/20240401-173729-marostegui.json
  • 17:22 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161', diff saved to https://phabricator.wikimedia.org/P59078 and previous config saved to /var/cache/conftool/dbconfig/20240401-172221-marostegui.json
  • 17:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59077 and previous config saved to /var/cache/conftool/dbconfig/20240401-170713-marostegui.json
  • 16:56 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1161 (T356166)', diff saved to https://phabricator.wikimedia.org/P59076 and previous config saved to /var/cache/conftool/dbconfig/20240401-165559-marostegui.json
  • 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1016,1020-1021].eqiad.wmnet,db1154.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 16:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1161.eqiad.wmnet with reason: Maintenance
  • 16:43 eevans@deploy1002: Finished deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided) (duration: 00m 33s)
  • 16:42 eevans@deploy1002: Started deploy [cassandra/logstash-logback-encoder@42653e6]: (no justification provided)
  • 16:07 Dreamy_Jazz: Restarting MediaModeration scanning script - https://wikitech.wikimedia.org/wiki/MediaModeration
  • 16:05 pt1979@cumin2002: END (FAIL) - Cookbook sre.hosts.dhcp (exit_code=99) for host elastic2088.codfw.wmnet
  • 15:50 pt1979@cumin2002: START - Cookbook sre.hosts.dhcp for host elastic2088.codfw.wmnet
  • 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host dbprov1005.eqiad.wmnet with OS bullseye
  • 15:36 jclark@cumin1002: END (PASS) - Cookbook sre.puppet.sync-netbox-hiera (exit_code=0) generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 15:35 jclark@cumin1002: START - Cookbook sre.puppet.sync-netbox-hiera generate netbox hiera data: "Triggered by cookbooks.sre.hosts.reimage: Host reimage - jclark@cumin1002"
  • 15:33 urandom: cassandra (restbase): re-enable blocking read-repair — T354561
  • 15:30 sukhe@puppetmaster1001: conftool action : set/pooled=yes; selector: name=cp4052.ulsfo.wmnet,service=(cdn|ats-be)
  • 15:14 jclark@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
  • 15:13 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.reimage (exit_code=0) for host cp4052.ulsfo.wmnet with OS bullseye
  • 15:11 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:11 jclark@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on dbprov1005.eqiad.wmnet with reason: host reimage
  • 15:11 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on dbstore1009.eqiad.wmnet with reason: Maintenance
  • 15:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59075 and previous config saved to /var/cache/conftool/dbconfig/20240401-151114-marostegui.json
  • 14:58 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
  • 14:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59074 and previous config saved to /var/cache/conftool/dbconfig/20240401-145606-marostegui.json
  • 14:52 sukhe@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:52 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:51 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:49 sukhe@cumin1002: START - Cookbook sre.hosts.downtime for 2:00:00 on cp4052.ulsfo.wmnet with reason: host reimage
  • 14:41 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231', diff saved to https://phabricator.wikimedia.org/P59073 and previous config saved to /var/cache/conftool/dbconfig/20240401-144059-marostegui.json
  • 14:32 jhancock@cumin2002: END (FAIL) - Cookbook sre.hardware.upgrade-firmware (exit_code=99) upgrade firmware for hosts ['elastic2088']
  • 14:32 jhancock@cumin2002: START - Cookbook sre.hardware.upgrade-firmware upgrade firmware for hosts ['elastic2088']
  • 14:28 jclark@cumin1002: END (FAIL) - Cookbook sre.hosts.reimage (exit_code=99) for host dbprov1005.eqiad.wmnet with OS bullseye
  • 14:27 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:26 sukhe@cumin1002: END (ERROR) - Cookbook sre.hosts.reimage (exit_code=97) for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59072 and previous config saved to /var/cache/conftool/dbconfig/20240401-142552-marostegui.json
  • 14:06 sukhe@cumin1002: START - Cookbook sre.hosts.reimage for host cp4052.ulsfo.wmnet with OS bullseye
  • 14:06 sukhe: reimage cp4052 back to bullseye
  • 13:52 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1231 (T356166)', diff saved to https://phabricator.wikimedia.org/P59071 and previous config saved to /var/cache/conftool/dbconfig/20240401-135204-marostegui.json
  • 13:51 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:51 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1231.eqiad.wmnet with reason: Maintenance
  • 13:11 jclark@cumin1002: START - Cookbook sre.hosts.reimage for host dbprov1005.eqiad.wmnet with OS bullseye
  • 12:55 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1225.eqiad.wmnet with reason: Maintenance
  • 12:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59070 and previous config saved to /var/cache/conftool/dbconfig/20240401-125524-marostegui.json
  • 12:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59069 and previous config saved to /var/cache/conftool/dbconfig/20240401-124017-marostegui.json
  • 12:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224', diff saved to https://phabricator.wikimedia.org/P59068 and previous config saved to /var/cache/conftool/dbconfig/20240401-122510-marostegui.json
  • 12:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59067 and previous config saved to /var/cache/conftool/dbconfig/20240401-121002-marostegui.json
  • 11:41 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1224 (T356166)', diff saved to https://phabricator.wikimedia.org/P59066 and previous config saved to /var/cache/conftool/dbconfig/20240401-114105-marostegui.json
  • 11:40 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1224.eqiad.wmnet with reason: Maintenance
  • 11:40 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59065 and previous config saved to /var/cache/conftool/dbconfig/20240401-114043-marostegui.json
  • 11:25 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59064 and previous config saved to /var/cache/conftool/dbconfig/20240401-112536-marostegui.json
  • 11:10 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201', diff saved to https://phabricator.wikimedia.org/P59063 and previous config saved to /var/cache/conftool/dbconfig/20240401-111028-marostegui.json
  • 10:55 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59062 and previous config saved to /var/cache/conftool/dbconfig/20240401-105521-marostegui.json
  • 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1201 (T356166)', diff saved to https://phabricator.wikimedia.org/P59061 and previous config saved to /var/cache/conftool/dbconfig/20240401-102328-marostegui.json
  • 10:23 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1201.eqiad.wmnet with reason: Maintenance
  • 10:23 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59060 and previous config saved to /var/cache/conftool/dbconfig/20240401-102306-marostegui.json
  • 10:07 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59059 and previous config saved to /var/cache/conftool/dbconfig/20240401-100758-marostegui.json
  • 09:52 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187', diff saved to https://phabricator.wikimedia.org/P59058 and previous config saved to /var/cache/conftool/dbconfig/20240401-095251-marostegui.json
  • 09:37 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59057 and previous config saved to /var/cache/conftool/dbconfig/20240401-093744-marostegui.json
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1187 (T356166)', diff saved to https://phabricator.wikimedia.org/P59056 and previous config saved to /var/cache/conftool/dbconfig/20240401-090527-marostegui.json
  • 09:05 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1187.eqiad.wmnet with reason: Maintenance
  • 09:05 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59055 and previous config saved to /var/cache/conftool/dbconfig/20240401-090503-marostegui.json
  • 08:49 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59054 and previous config saved to /var/cache/conftool/dbconfig/20240401-084956-marostegui.json
  • 08:34 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1180', diff saved to https://phabricator.wikimedia.org/P59053 and previous config saved to /var/cache/conftool/dbconfig/20240401-083449-marostegui.json
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1180 (T356166)', diff saved to https://phabricator.wikimedia.org/P59051 and previous config saved to /var/cache/conftool/dbconfig/20240401-074221-marostegui.json
  • 07:42 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1180.eqiad.wmnet with reason: Maintenance
  • 07:42 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59050 and previous config saved to /var/cache/conftool/dbconfig/20240401-074158-marostegui.json
  • 07:26 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59049 and previous config saved to /var/cache/conftool/dbconfig/20240401-072650-marostegui.json
  • 07:11 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168', diff saved to https://phabricator.wikimedia.org/P59048 and previous config saved to /var/cache/conftool/dbconfig/20240401-071143-marostegui.json
  • 06:56 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59047 and previous config saved to /var/cache/conftool/dbconfig/20240401-065635-marostegui.json
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1168 (T356166)', diff saved to https://phabricator.wikimedia.org/P59046 and previous config saved to /var/cache/conftool/dbconfig/20240401-062954-marostegui.json
  • 06:29 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 06:29 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1168.eqiad.wmnet with reason: Maintenance
  • 06:29 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59045 and previous config saved to /var/cache/conftool/dbconfig/20240401-062932-marostegui.json
  • 06:14 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59044 and previous config saved to /var/cache/conftool/dbconfig/20240401-061423-marostegui.json
  • 05:59 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165', diff saved to https://phabricator.wikimedia.org/P59043 and previous config saved to /var/cache/conftool/dbconfig/20240401-055915-marostegui.json
  • 05:44 marostegui@cumin1002: dbctl commit (dc=all): 'Repooling after maintenance db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59042 and previous config saved to /var/cache/conftool/dbconfig/20240401-054408-marostegui.json
  • 05:14 marostegui@cumin1002: dbctl commit (dc=all): 'Depooling db1165 (T356166)', diff saved to https://phabricator.wikimedia.org/P59041 and previous config saved to /var/cache/conftool/dbconfig/20240401-051402-marostegui.json
  • 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 16:00:00 on clouddb[1015,1019,1021].eqiad.wmnet,db1155.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 05:13 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 8:00:00 on db1165.eqiad.wmnet with reason: Maintenance
  • 05:03 marostegui@cumin1002: END (PASS) - Cookbook sre.hosts.downtime (exit_code=0) for 5:00:00 on 9 hosts with reason: Fixing intermediate master
  • 05:03 marostegui@cumin1002: START - Cookbook sre.hosts.downtime for 5:00:00 on 9 hosts with reason: Fixing intermediate master


Other archives

2000s

2010s

2020s

Retrieved from "https://wikitech.wikimedia.org/w/index.php?title=Server_Admin_Log/Archive_78&oldid=2173794"