[00:54:10] <icinga-wm>	 PROBLEM - Check the last execution of monitor_refine_eventlogging_analytics_failure_flags on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_eventlogging_analytics_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[02:51:30] <icinga-wm>	 PROBLEM - Check the last execution of monitor_refine_mediawiki_job_events_failure_flags on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit monitor_refine_mediawiki_job_events_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[06:38:56] <icinga-wm>	 RECOVERY - Check the last execution of monitor_refine_mediawiki_job_events_failure_flags on an-launcher1001 is OK: OK: Status of the systemd unit monitor_refine_mediawiki_job_events_failure_flags https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers
[14:27:09] <wikibugs>	 10Quarry, 10Data-Services: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Marostegui) The query killer is set to 2 hours instead of 4 as we are still troubleshooting the affected server
[14:49:41] <wikibugs>	 10Analytics, 10Analytics-Kanban, 10Analytics-SWAP, 10Product-Analytics, 10User-Elukey: pip not accessible in new SWAP virtual environments - https://phabricator.wikimedia.org/T247752 (10nshahquinn-wmf) >>! In T247752#5975529, @mpopov wrote: > I just have `export PATH=/home/bearloga/venv/bin:${PATH}` in m...
[16:33:16] <wikibugs>	 10Analytics, 10Event-Platform, 10Operations, 10Services (watching): Discovery for Kafka cluster brokers - https://phabricator.wikimedia.org/T213561 (10Aklapper) >>! In T213561#4881255, Joe wrote: > Might I suggest that you use a SRV dns record instead?  >>! In T213561#4882509, Ottomata wrote: > Kafka doesn...
[17:05:38] <wikibugs>	 10Analytics: Decomission notebook hosts - https://phabricator.wikimedia.org/T249752 (10nshahquinn-wmf) I've moved everything off of both notebook servers. Thanks, @elukey!
[20:30:06] <wikibugs>	 10Quarry, 10Data-Services: Quarry: Lost connection to MySQL server during query - https://phabricator.wikimedia.org/T246970 (10Mike_Peel) OK, I think I'm just wasting CPU time by trying to run this query at the moment. I'll pause {{Wikidata Infobox}} deployment on Commons until things are running better. Best...
[21:21:38] <icinga-wm>	 PROBLEM - Check the last execution of camus-mediawiki_job on an-launcher1001 is CRITICAL: CRITICAL: Status of the systemd unit camus-mediawiki_job https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers