In T371426#10174127, @ecarg wrote:

Wow, looks great 🤩! Yes, that sounds like a plan, TY!
How do we route these alerts to trigger our Slack?

Nice! Thank you @hnowlan, resolving as we're done

In T375066#10158541, @Dzahn wrote:

Would it make sense to have clusters named after a team to group all machines owned by a specific subteam?

Or would that go against the purpose of clusters and they should stricly group servers by service / puppet role?

In T374842#10157255, @fnegri wrote:

Reading through the comments I found:

For centrallog we'll do the in place upgrade to Bookworm as part of T353912, I'd say sooner rather than later. Not sure about cloudlb* hosts, thus cc cloud-services-team to see if they do have indeed an ETA for cloudlb on Bookworm or not.

I've created a subtask for the cloudlb bookworm upgrade, it's not in our immediate plans but shouldn't be too hard: T375082: Upgrade cloudlb hosts to bookworm.

In T374842#10153975, @ssingh wrote:

On perhaps a related note, while it is true that many of the things the script is doing are taken care by the self-healing nature of anycast-hc in 0.9 and beyond, I do wonder if there is value in keeping this script around. anycast-hc is a critical part of our infrastructure especially on the DNS boxes because we announce all ns[0-2] routes from there. In a hypothetical case where it is not running, this would mean that all our nameservers are down. This leads me to wonder that in the absence of this script, what kind of an alert will be generated if anycast-hc is not running if it failed to start or is manually stopped for whatever reason?

One such alert can be the BGP one but there are many of them and the SNR is quite terrible. The other alert can be the systemd service not running but that again is shadowed by other such alerts. Does my concern make sense? In which case I am thinking perhaps we can restrict the scope of the script but still keep it around because if we get rid of this, there isn't a single dedicated script to check if anycast-hc is running or and in case it failed or didn't start, we won't probably catch it.

In T374916#10153020, @dcausse wrote:

We could perhaps adapt modules/query_service/files/monitor/prometheus-blazegraph-exporter.py to take care of running this query by possibly re-using the same gauge blazegraph_lastupdated but adapting the query depending on the namespace it's running.

In T374842#10149670, @ssingh wrote:

In T374842#10149337, @fgiunchedi wrote:

Thank you @ssingh, there's also centrallog hosts running bullseye + 0.8 in addition to cloudlb hosts, at any rate I tried installing the bookworm anycast-healthchecker 0.9 package on bullseye on a test host and it seems to work as expected (i.e. a rebuild for bullseye doesn't seem to be needed)

Ah, that's interesting. I mean it's not surprising it worked given the code but I am not sure if we want to extend it like that. Or maybe, it's fine and we do this all the time! (I know of some cases in which we do it but I am not sure if it is frowned upon.)

Doh! You are quite right, the relevant alert is MaxConntrack

Thank you @ssingh, there's also centrallog hosts running bullseye + 0.8 in addition to cloudlb hosts, at any rate I tried installing the bookworm anycast-healthchecker 0.9 package on bullseye on a test host and it seems to work as expected (i.e. a rebuild for bullseye doesn't seem to be needed)

@ssingh from T370068: Upgrade anycast-healthchecker to 0.9.8 (from 0.9.1-1+wmf12u1) I couldn't find any obvious blocker to have 0.9 on bullseye. Do you remember if there were there any obvious blockers for that to happen ? Thank you

Also note that prometheus-postgres-exporter since 0.12.0 has gained support for replication monitoring. This is IMHO the proper solution, though it will require >= trixie unless we chose to backport the package instead: https://github.com/prometheus-community/postgres_exporter/blob/master/CHANGELOG.md#0120--2023-03-21
It is an option that can/should be considered I think since it would mean future-proofing the postgresql monitoring infrastructure.

Resolving since we haven't run into further problems with envoy metrics ingestion

From my investigation so far on IRC:

script is deployed!

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

This is done, I've set the service as non-paging since we're using it for the prometheus web interface (i.e. humans) whereas the http service is paging since that's for automated access

This is done! The only other use of graphite_threshold is 'zuul_gearman_wait_queue' which will be addressed as part of T233089: Export zuul metrics to Prometheus. There are of course graphite-itself alerts left, which will be removed together with graphite.

I have merged the ported mw alerts (thank you @Clement_Goubert !) and changed the referenced dashboard at https://grafana.wikimedia.org/d/000000438/mediawiki-exceptions-alerts to show the prometheus metrics.

Also please note that we're blocked on T374340 before the failover can happen

This is great, thank you all for your help!

With https://gerrit.wikimedia.org/r/c/operations/puppet/+/1060516 merged now corto's profile::corto::active_host hiera setting needs to be changed on failover, @andrea.denisse FYI

I'm optimistically resolving this, please reopen if sth is amiss!

Thank you for the report, the script indeed breaks on renamed users (I112dcae4bf being the cause). I've manually deleted the old user and the script now completes, so you should have access @Southparkfan ! I've filed T374190: grafana-ldap-users-sync breaks on renamed users for followup

In T373980#10121709, @MoritzMuehlenhoff wrote:

In T373980#10121443, @fgiunchedi wrote:

Thank you to all involved so far with the reboots -- much appreciated!

I can confirm the hosts are now reachable from alert2002, except lists1004 and lists2001. On these hosts, unlike the others, there is both nft and iptables, similarly there's /etc/ferm and /etc/nftables which I'm assuming is the cause of the problem (i.e. iptables and nftables together). Does that ring a bell @eoghan ?

Looking at Puppet it seems the host was moved to nftables and then back to iptables/ferm. There's no migration path in Puppet for the latter.

Cleaning up manually the following steps are needed:

• stop nftables.service and following that uninstall the nftables package
• remove /etc/nftables
• reboot into a clean iptables/ferm setup

That should be the core of it, if there's other issues after the reboot, happy to have a closer look.

Thank you to all involved so far with the reboots -- much appreciated!

In T372418#10114221, @Jgreen wrote:

Fundraising servers use nsca exclusively, so we've configured them to send to all four alert[12]00[12] hosts while things are in flux. I'll try to keep an eye on this task but please let us know if when we should remove extraneous hosts from the list.

Thanks for reaching out @Peter ! I'm sure the jenkins server can be polled for metrics by prometheus. As long as the workers run in production they are also able to reach the pushgateway, hope that helps!

In T351927#10072017, @fgiunchedi wrote:

In T351927#10072008, @MatthewVernon wrote:

@fgiunchedi these alerts are still firing; I thought you were going to silence them. Did I misunderstand, and you'd like me to?

Yes please silence the alerts and I'll disable compactions, thank you!

The prometheus bookworm upgrade has been completed as part of T326657: Add prometheus-https load balancer, thank you @andrea.denisse for your help with this

In T326657#10094284, @jhathaway wrote:

In T326657#10090776, @fgiunchedi wrote:

I'm for acking the probefailure alerts for puppetmaster hosts only, given that they are effectively a false positive now. What do you think @jhathaway ?

that sounds fine, thanks for asking

I'm tentatively resolving since the silences are in place, please feel free to reopen as needed!

Thank you for filing the task, I was also looking at the same failed probes due to the recent Prometheus Bookworm upgrade. tl;dr is I think it is safe to ack these alerts and I will do so, see also https://phabricator.wikimedia.org/T326657#10090776

First of all, my apologies for the long lead time on this. I have reviewed upstream documentation, issues and alert logs; the most promising issue I could find that may explain this behavior is https://github.com/prometheus/alertmanager/issues/3808 to which I've inquired further.

The two hosts in Bookworm (prometheus2006 and prometheus1006) work well, the only problem I could find is that probes for puppetmaster https endpoints (not puppetserver!) are failing, this is a long-standing issue due to the fact that said endpoints use certs without SAN. Bookworm prometheus-blackbox-exporter has been compiled with newer golang (>= 1.17) which doesn't allow to ignore certs without SANs anymore.

This has been resolved in the meantime

I tested the bookworm in place upgrade on prometheus2006 and things seem to be working as expected. I did the following:

@bking re: IRC question "output a process list in the alert email body" we don't have facilities for that, however we do have resource utilization per-unit available, that will give you a breakdown of what jupyer users are doing, for example:

Update after team meeting: I'll be starting the in-place Bookworm upgrade since it'll unblock this issue, it is something we have to do anyways, and I have prometheus host in Pontoon running on Bookworm with no obvious problems.

Graphite is going away

As far as I can tell this is done, boldly resolving

I'm boldly declining since it isn't obvious what the advantages are, vs obvious disadvantages (doing the work at the very least)