[01:11:00] FIRING: [2x] PuppetCertificateAboutToExpire: Puppet CA certificate fullstackd-20210112012248.admin-monitoring.eqiad1.wikimedia.cloud is about to expire in 16d 0h 15m 34s - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/PuppetCertificateAboutToExpire - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPuppetCertificateAboutToExpire [02:45:52] PROBLEM - HTTPS-wikitech-static on wikitech-static.wikimedia.org is CRITICAL: SSL CRITICAL - Certificate status.wikimedia.org valid until 2025-12-28 19:04:50 +0000 (expires in 2 days) https://wikitech.wikimedia.org/wiki/Wikitech-static [02:48:32] PROBLEM - HTTPS-wikitech-static on wikitech-static.wikimedia.org is CRITICAL: SSL CRITICAL - Certificate status.wikimedia.org valid until 2025-12-28 19:04:50 +0000 (expires in 2 days) https://wikitech.wikimedia.org/wiki/Wikitech-static [05:55:40] 10Tool-wsindex, 10Wikisource Reader App: Display extra information from Wikidata in Book Details Screen - https://phabricator.wikimedia.org/T406245#11484784 (10Saiphani02) Merged [06:35:12] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510 (10Hannolans) 03NEW [06:37:21] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11484795 (10Hannolans) [06:37:58] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11484796 (10Hannolans) [08:31:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [08:36:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:08:13] (03approved) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/56 (owner: 10l10n-bot) [09:08:16] (03merge) 10lucaswerkmeister: Localisation updates from https://translatewiki.net. [toolforge-repos/wd-image-positions] - 10https://gitlab.wikimedia.org/toolforge-repos/wd-image-positions/-/merge_requests/56 (owner: 10l10n-bot) [11:11:48] 10Tool-curator: Improve title validation in frontend - https://phabricator.wikimedia.org/T413512 (10DaxServer) 03NEW [11:12:09] 10Tool-curator: Improve title validation in frontend - https://phabricator.wikimedia.org/T413512#11484838 (10DaxServer) p:05Triage→03High [11:12:39] 10Tool-curator: Handle large Mapillary sequences - https://phabricator.wikimedia.org/T412617#11484841 (10DaxServer) p:05Low→03Medium [11:21:29] 06cloud-services-team, 10PAWS: Start of PAWS terminal fails - Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413513 (10M2k_dewiki) 03NEW [12:48:27] 06cloud-services-team, 10PAWS: Start of PAWS terminal fails - Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413513#11484865 (10M2k_dewiki) Also see https://phabricator.wikimedia.org/T413428 https://phabricator.wikimedia.org/T400589 [16:42:18] 10Tool-translatetagger: Add support for adding translate tags for templates - https://phabricator.wikimedia.org/T413520 (10Gauthammohanraj) 03NEW [16:43:20] 10Tool-translatetagger: Add support for adding translate tags for templates - https://phabricator.wikimedia.org/T413520#11484983 (10Gauthammohanraj) a:03Gauthammohanraj I would be working on bringing this in :) [17:31:12] 06cloud-services-team, 10PAWS: Start of PAWS terminal fails - Unable to attach or mount volumes - https://phabricator.wikimedia.org/T413513#11485005 (10Pppery) →14Duplicate dup:03T413510 [17:31:16] 06cloud-services-team, 10PAWS: [bug] server won't launch - https://phabricator.wikimedia.org/T413510#11485007 (10Pppery) [17:38:12] 06cloud-services-team, 10Cloud-VPS: Configure vanity domain for wlmaz - https://phabricator.wikimedia.org/T413467#11485015 (10Aklapper) Hi, what is the assumption based on that the Cloud Services Team can create random country domains? [21:07:06] 10Tools, 05PES1.3.3 WP25 Easter Eggs, 07Software-Licensing: wikipedia25-years-of-wikipedia tool loads and uses non-free JavaScript - https://phabricator.wikimedia.org/T410465#11485107 (10Pppery) 05Open→03Resolved [21:10:51] 10Tools: 360-degree images uploaded after August 12, 2022 cannot be loaded by panoviewer.toolforge.org. - https://phabricator.wikimedia.org/T315153#11485110 (10Pppery) 05Open→03Invalid [23:27:17] FIRING: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:29:26] !log andrew@cloudcumin1001 admin START - Cookbook wmcs.openstack.restart_openstack on deployment codfw1dev for all services [23:32:17] RESOLVED: JobUnavailable: Reduced availability for job rabbitmq in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [23:32:29] !log andrew@cloudcumin1001 admin END (PASS) - Cookbook wmcs.openstack.restart_openstack (exit_code=0) on deployment codfw1dev for all services