[00:08:28] FIRING: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:13:28] RESOLVED: PuppetAgentStaleLastRun: Last Puppet run was over 24 hours ago on instance tf-infra-test in project tf-infra-test - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetAgentStaleLastRun [00:21:28] FIRING: PuppetCertificateAboutToExpire: Puppet CA certificate coibot.linkwatcher.eqiad.wmflabs is about to expire in 27d 23h 58m 37s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [00:31:28] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate coibot.linkwatcher.eqiad.wmflabs is about to expire in 27d 23h 48m 37s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [01:22:29] 10Tools: 'deletion-notification-bot-2' tool uses an unreasonable amount of disk space - https://phabricator.wikimedia.org/T349898#10055872 (10mdaniels5757) 05Open→03Resolved Totally forgot about this task, but I did this a while ago. [05:52:49] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [05:54:43] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29687 bytes in 3.485 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [06:36:39] FIRING: ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [06:41:39] RESOLVED: ProbeDown: Service toolsbeta-test-k8s-haproxy-5:30000 has failed probes (http_this_tool_does_not_exist_beta_toolforge_org_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#toolsbeta-test-k8s-haproxy-5:30000 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/custom&var-module=All - https://prometheus-alerts.wmcloud.org/?q=alertname%3DProbeDown [13:27:10] 10Cloud-VPS (Debian Buster Deprecation), 10Humaniki: Cloud VPS "wikidumpparse" project Buster deprecation - https://phabricator.wikimedia.org/T367561#10056162 (10Maximilianklein) [13:27:52] 10Cloud-VPS (Debian Buster Deprecation), 10Humaniki: Cloud VPS "wikidumpparse" project Buster deprecation - https://phabricator.wikimedia.org/T367561#10056164 (10Maximilianklein) p:05Medium→03Unbreak! [16:06:34] (03CR) 10Legoktm: [C:04-1] config: Index https://gitlab.wikimedia.org/toolforge-repos/* (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1060493 (https://phabricator.wikimedia.org/T371992) (owner: 10BryanDavis) [17:52:18] 10Tool-replag, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change-in-production: Enwiki still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056289 (10GTrang) [17:57:38] 10Tool-replag, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change-in-production: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056290 (10GTrang) [18:12:13] 10Tool-replag, 06Data Products, 06Data-Engineering, 06DBA, 07Schema-change-in-production: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056301 (10RhinosF1) Replication lag is normal during these schema changes. It will naturally... [18:52:51] 10Data-Services: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056313 (10JJMC89) [18:52:59] 10Data-Services: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056315 (10JJMC89) [18:54:04] 10Data-Services: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056317 (10JJMC89) a:05Marostegui→03None [18:54:20] 10Data-Services: enwiki.analytics.db.svc.wikimedia.cloud still on replag for more than a week - https://phabricator.wikimedia.org/T372224#10056308 (10JJMC89) 05Open→03Invalid p:05High→03Triage per above [20:10:51] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [20:12:43] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29687 bytes in 2.065 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:21:57] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [21:22:49] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29687 bytes in 1.385 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [21:36:53] (03CR) 10Urbanecm: [C:03+2] Add .gitreview [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061166 (owner: 10Tacsipacsi) [21:37:15] (03Merged) 10jenkins-bot: Add .gitreview [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061166 (owner: 10Tacsipacsi) [21:37:33] (03CR) 10Urbanecm: [C:03+2] "Thanks for the suggestion! I don't see why not, seems fair to do. Let's ship it!" [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061167 (owner: 10Tacsipacsi) [21:37:52] (03Merged) 10jenkins-bot: Support dark mode in mails [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061167 (owner: 10Tacsipacsi) [21:47:44] (03CR) 10Tacsipacsi: "Thanks for merging and deploying it!" [labs/tools/watch-translations] - 10https://gerrit.wikimedia.org/r/1061167 (owner: 10Tacsipacsi) [23:53:05] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static [23:53:57] RECOVERY - Wikitech-static main page has content on wikitech-static.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 29689 bytes in 2.063 second response time https://wikitech.wikimedia.org/wiki/Wikitech-static [23:57:07] PROBLEM - Wikitech-static main page has content on wikitech-static.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Wikitech-static