Page MenuHomePhabricator
Create Task
Maniphest T190607

cp3048 hardware issues
Closed, DuplicatePublic
Actions

Description

cp3048 is out of warranty and has a possibly broken mainboard. It's poweed off and has been used as a "hardware donor" for other OOW hosts in esams.

Decom check list from wikitech:

  • - all system services confirmed offline from production use
  • - set all icinga checks to maint mode/disabled while reclaim/decommmission takes place.
  • - remove system from all lvs/pybal active configuration
  • - any service group puppet/hiera/dsh config removed
  • - remove site.pp (replace with role::spare::system if system isn't shut down immediately during this process.)

START NON-INTERRUPPTABLE STEPS

  • - disable puppet on host
  • - remove all remaining puppet references (include role::spare)
  • - power down host
  • - disable switch port
  • - switch port assignment noted on this task (for later removal)
  • - remove production dns entries
  • - puppet node clean, puppet node deactivate

END NON-INTERRUPPTABLE STEPS

  • - system disks wiped (by onsite)
  • - IF DECOM: system unracked and decommissioned (by onsite), update racktables with result
  • - IF DECOM: switch port configration removed from switch once system is unracked.
  • - IF DECOM: mgmt dns entries removed.
  • - IF RECLAIM: system added back to spares tracking (by onsite)

Details

SubjectRepoBranchLines +/-
Remove cp3048 prod DNS entriesoperations/dnsmaster+0 -4
Remove cp3048 from site.pp/DHCP configoperations/puppetproduction+1 -8
Remove cp3048 from Hiera/conftooloperations/puppetproduction+0 -2
Customize query in gerrit

Related Objects

Event Timeline

BBlack triaged this task as Medium priority.Mar 24 2018, 2:52 AM
BBlack created this task.
Restricted Application added a project: SRE. · View Herald TranscriptMar 24 2018, 2:52 AM
Restricted Application added a subscriber: Aklapper. · View Herald Transcript
BBlack moved this task from Backlog to Hardware on the Traffic board.Mar 26 2018, 7:41 PM
jcrespo subscribed.Apr 12 2018, 1:39 PM

It probably crashed today at 2018-04-12 13:31:20, hardware logs should be checked.

mark moved this task from Backlog to Hardware Failure / Repair on the ops-esams board.Jul 3 2018, 11:59 AM
mark subscribed.Jul 3 2018, 12:55 PM

As Chris confirmed, this is either due to CPU or memory.

Since it's a dual CPU machine, he suggests to swap the two CPUs and see if the error follows the swap.

mark added a comment.Jul 3 2018, 1:02 PM

More recently:

-------------------------------------------------------------------------------
Record:      105
Date/Time:   05/09/2018 22:48:53
Source:      system
Severity:    Critical
Description: The system board PS2 PG Fail voltage is outside of range.
-------------------------------------------------------------------------------
Record:      106
Date/Time:   05/16/2018 10:55:57
Source:      system
Severity:    Critical
Description: CPU 1 has an internal error (IERR).
-------------------------------------------------------------------------------

Which means it's most likely the mainboard that needs replacement.

Either way, this system is out of warranty. I may sacrifice it and use it as a source of parts (SSD, DIMMs) for other broken machines...

mark moved this task from Hardware Failure / Repair to Procurement on the ops-esams board.Jul 3 2018, 1:04 PM
Stashbot subscribed.Jul 4 2018, 11:30 AM

Mentioned in SAL (#wikimedia-traffic) [2018-07-04T11:30:17Z] <ema> shutdown cp3048 and cp3034 (both already depooled) for hardware maintenance T190607 T189305

Stashbot mentioned this in T189305: cp3034: Uncorrectable Memory Error.Jul 4 2018, 11:30 AM
mark added a comment.Jul 4 2018, 12:01 PM

Because cp3048 is out of warranty and it's unlikely we can get it fixed, I've used it as a parts donor for other broken systems:

  • Its second drive (sdb) has been taken for use in cp3043:sdb, the drive carrier is now empty
  • Its B3 DIMM module has been swapped with cp3034 B3, and it may be bad (or may work now after essentially a reseat)
  • CPU1 is still likely having issues, may or may not be due to power supply by the main board.
mark moved this task from Procurement to Hardware Failure / Repair on the ops-esams board.Jul 4 2018, 2:40 PM
MoritzMuehlenhoff updated the task description. (Show Details)Jul 5 2018, 2:33 PM
gerritbot subscribed.Jul 6 2018, 8:31 AM

Change 444169 had a related patch set uploaded (by Ema; owner: Muehlenhoff):
[operations/puppet@production] Remove cp3048 from Hiera/conftool

https://gerrit.wikimedia.org/r/444169

gerritbot added a project: Patch-For-Review.Jul 6 2018, 8:31 AM
gerritbot added a comment.Jul 6 2018, 8:36 AM

Change 444169 merged by Muehlenhoff:
[operations/puppet@production] Remove cp3048 from Hiera/conftool

https://gerrit.wikimedia.org/r/444169

gerritbot added a comment.Jul 6 2018, 8:42 AM

Change 444171 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/puppet@production] Remove cp3048 from site.pp/DHCP config

https://gerrit.wikimedia.org/r/444171

MoritzMuehlenhoff updated the task description. (Show Details)Jul 6 2018, 10:31 AM
gerritbot added a comment.Jul 9 2018, 10:10 AM

Change 444171 merged by Muehlenhoff:
[operations/puppet@production] Remove cp3048 from site.pp/DHCP config

https://gerrit.wikimedia.org/r/444171

MoritzMuehlenhoff updated the task description. (Show Details)Jul 9 2018, 10:13 AM
gerritbot added a comment.Jul 9 2018, 10:22 AM

Change 444560 had a related patch set uploaded (by Muehlenhoff; owner: Muehlenhoff):
[operations/dns@master] Remove cp3048 prod DNS entries

https://gerrit.wikimedia.org/r/444560

MoritzMuehlenhoff updated the task description. (Show Details)Jul 9 2018, 10:24 AM
gerritbot added a comment.Jul 10 2018, 10:55 AM

Change 444560 merged by Muehlenhoff:
[operations/dns@master] Remove cp3048 prod DNS entries

https://gerrit.wikimedia.org/r/444560

MoritzMuehlenhoff updated the task description. (Show Details)Jul 10 2018, 10:55 AM
• Phabricator_maintenance moved this task from Backlog to Acknowledged on the SRE board.Jan 26 2019, 9:52 PM
Ladsgroup removed a project: Patch-For-Review.May 28 2019, 3:37 PM
BBlack closed this task as a duplicate of T236454: decommission cp3030-3049.Oct 25 2019, 4:05 AM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits