[00:04:55] 10Tool-schedule-deployment: schedule-deployment tool doesn't recognize mediawiki-vendor as a "MediaWiki repo" - https://phabricator.wikimedia.org/T411128#11505930 (10bd808) 05Open→03In progress a:03bd808 [00:05:48] FIRING: PuppetFailure: Puppet has failed on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [00:20:04] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Year in Review tool doesn't work for Wikidata and Mediawiki - https://phabricator.wikimedia.org/T381681#11505945 (10Wargo) https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/blob/main/src/utils/backend_helper.js#L239 change to `... [00:27:30] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: it should be a day instead of edits in yearinreview tool - https://phabricator.wikimedia.org/T377669#11505955 (10Wargo) @marrivs Do you still plan to work on it? [00:28:16] (03open) 10bd808: validate_backport: Allow mediawiki/vendor changes [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/21 (https://phabricator.wikimedia.org/T411128) [00:29:15] 10Tool-yearinreview: Page thumbnails work only for en.wikipedia - https://phabricator.wikimedia.org/T414151 (10Wargo) 03NEW [00:34:06] (03merge) 10bd808: validate_backport: Allow mediawiki/vendor changes [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/21 (https://phabricator.wikimedia.org/T411128) [01:03:59] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [01:08:57] FIRING: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:13:57] RESOLVED: ProbeDown: Service labweb-ssl:7443 has failed probes (http_labweb-ssl_ip4) - https://wikitech.wikimedia.org/wiki/Runbook#labweb-ssl:7443 - https://grafana.wikimedia.org/d/O0nHhdhnz/network-probes-overview?var-job=probes/service&var-module=All - https://alerts.wikimedia.org/?q=alertname%3DProbeDown [01:20:20] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11506014 (10bd808) 05Open→03In progress a:03bd808 [01:21:09] (03open) 10bd808: backport: Disable submit button on click [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/22 (https://phabricator.wikimedia.org/T409536) [01:26:50] (03merge) 10bd808: backport: Disable submit button on click [toolforge-repos/schedule-deployment] - 10https://gitlab.wikimedia.org/toolforge-repos/schedule-deployment/-/merge_requests/22 (https://phabricator.wikimedia.org/T409536) [01:32:10] 10Tool-schedule-deployment: schedule-deployment tool doesn't recognize mediawiki-vendor as a "MediaWiki repo" - https://phabricator.wikimedia.org/T411128#11506033 (10bd808) 05In progress→03Resolved [01:32:17] 10Tool-schedule-deployment: Disable "Schedule" button after first click - https://phabricator.wikimedia.org/T409536#11506034 (10bd808) 05In progress→03Resolved [02:27:41] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11506059 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie [03:03:21] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11506069 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie executed with errors: - cloudba... [03:03:52] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11506078 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie [03:10:00] 10Tool-wsindex, 10Wikisource Reader App: Books update option in local library - https://phabricator.wikimedia.org/T408285#11506079 (10Muguro) @Saiphani02 I have opened a PR for this https://github.com/cis-india/Wikisource-Reader/pull/73 [03:45:45] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11506080 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup2003.codfw.wmnet with OS trixie completed: - cloudbackup2003 (*... [04:33:22] 10Tool-logo-test: logo-test.toolforge.org is down - https://phabricator.wikimedia.org/T412361#11506104 (10srishakatux) Tagging @Legoktm. It is still down. [05:03:13] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Year in Review tool doesn't work for Wikidata and Mediawiki - https://phabricator.wikimedia.org/T381681#11506126 (10Gopavasanth) a:03Gopavasanth Commit: https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/commit/ecaa16c3227e1e77... [05:03:59] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [05:24:43] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Year in Review tool doesn't work for Wikidata and Mediawiki - https://phabricator.wikimedia.org/T381681#11506139 (10Gopavasanth) 05Open→03Resolved [05:31:48] FIRING: PuppetFailure: Puppet has failed on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:58:32] 10Tool-curator: Allow updating SDC (safe-merge) when a file already exists - https://phabricator.wikimedia.org/T412958#11506402 (10DaxServer) 05In progress→03Resolved [08:58:44] 10Tool-curator: Integrate with Edit Groups - https://phabricator.wikimedia.org/T413962#11506404 (10DaxServer) 05Open→03In progress [09:03:59] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [09:25:29] 10Tool-yearinreview: Page thumbnails work only for en.wikipedia - https://phabricator.wikimedia.org/T414151#11506493 (10Wargo) a:03Gopavasanth https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/commit/bcc494360fa9f61eafb5956ed70aab511a3558ab [09:28:08] 10Tool-yearinreview: Page thumbnails work only for en.wikipedia - https://phabricator.wikimedia.org/T414151#11506496 (10Wargo) 05Open→03Resolved [09:31:51] (03merge) 10gopavasanth: Update apiresponse.json , fix typo in kannada languagename, addsupport for tcywikitionary and tcywikisource [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/10 (https://phabricator.wikimedia.org/T382850) (owner: 10anzx) [09:32:03] FIRING: PuppetFailure: Puppet has failed on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [09:34:08] 10Tool-yearinreview, 13Patch-For-Review: Home screen: Change language in drop down from kanada to Kannada - https://phabricator.wikimedia.org/T382850#11506520 (10Gopavasanth) 05Open→03Resolved a:03Gopavasanth [09:48:30] 06cloud-services-team, 10Cloud-VPS (Quota-requests), 10WikiCite, 10Wikidata, 10Wikidata-Query-Service: Raise quota on wikiqlever so that an instance with 256 GB RAM and 3 x 4 TB SSD can be launched - https://phabricator.wikimedia.org/T413097#11506547 (10fnegri) > The page says "Cloud VPS users can reques... [09:53:13] (03PS3) 10Ammarpad: write_config: Ignore problematic repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224787 (https://phabricator.wikimedia.org/T413322) [09:54:10] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: Check number of thanks stats in year in review tool - https://phabricator.wikimedia.org/T381413#11506554 (10Gopavasanth) 05Open→03Resolved a:03Gopavasanth @Kuldeepburjbhalaike, thanks for noticing this and reporting. Issue is fi... [09:58:04] 10Tool-yearinreview, 06Indic MediaWiki Developers UG, 06Indic-TechCom: show the name of user i thanked most in yearinreview tool - https://phabricator.wikimedia.org/T377670#11506563 (10Gopavasanth) 05Open→03Resolved a:03Gopavasanth It's resolved now with commit: https://gitlab.wikimedia.org/toolfor... [10:06:09] (03open) 10gopavasanth: Refactor user input layout to support initial values from URL parameters [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/11 (https://phabricator.wikimedia.org/T376371) [10:07:15] (03approved) 10gopavasanth: Refactor user input layout to support initial values from URL parameters [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/11 (https://phabricator.wikimedia.org/T376371) [10:07:28] (03merge) 10gopavasanth: Refactor user input layout to support initial values from URL parameters [toolforge-repos/yearinreview] - 10https://gitlab.wikimedia.org/toolforge-repos/yearinreview/-/merge_requests/11 (https://phabricator.wikimedia.org/T376371) [10:08:04] 10Tool-yearinreview, 13Patch-For-Review: Add Support for URL Parameters to Enable Sharable Links - https://phabricator.wikimedia.org/T376371#11506583 (10Gopavasanth) 05Open→03Resolved a:03Gopavasanth [12:38:41] FIRING: CloudVPSDesignateLeaks: Detected 1 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [13:03:59] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:22:56] 06cloud-services-team, 10Data-Services: Add datetime versions of timestamp fields to Wikireplica databases - https://phabricator.wikimedia.org/T414199 (10Huji) 03NEW [13:23:43] 06cloud-services-team, 10Data-Services: Add datetime versions of timestamp fields to Wikireplica databases - https://phabricator.wikimedia.org/T414199#11507428 (10Huji) [13:30:59] 10Tool-curator: Wait when lockmanager-fail-conflict error occurs - https://phabricator.wikimedia.org/T413915#11507449 (10DaxServer) 05Open→03In progress [13:32:03] FIRING: PuppetFailure: Puppet has failed on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [13:45:08] 10Tool-curator: Wait when lockmanager-fail-conflict error occurs - https://phabricator.wikimedia.org/T413915#11507490 (10DaxServer) Looks like this is in works for Pywikibot 11 https://phabricator.wikimedia.org/T396984 [13:45:21] 10Tool-curator: Wait when lockmanager-fail-conflict error occurs - https://phabricator.wikimedia.org/T413915#11507492 (10DaxServer) 05In progress→03Stalled [13:46:44] 10Tool-curator: Integrate Flickr uploads - https://phabricator.wikimedia.org/T414200 (10DaxServer) 03NEW [13:47:10] 10Tool-curator: Integrate Flickr uploads - https://phabricator.wikimedia.org/T414200#11507514 (10DaxServer) p:05Triage→03High [13:47:28] 10Tool-curator: Integrate Flickr uploads - https://phabricator.wikimedia.org/T414200#11507520 (10DaxServer) 05Open→03In progress [14:19:11] FIRING: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1017 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [14:29:05] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11507610 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie executed with errors: - cloudba... [14:36:02] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11507635 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage was started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie [14:46:18] RESOLVED: PuppetFailure: Puppet has failed on cloudbackup2003:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [15:17:20] 06cloud-services-team, 10Cloud-VPS: Complete upgrading WMCS bare metal hosts to Trixie - https://phabricator.wikimedia.org/T375217#11507763 (10ops-monitoring-bot) Cookbook cookbooks.sre.hosts.reimage started by andrew@cumin2002 for host cloudbackup1003.eqiad.wmnet with OS trixie completed: - cloudbackup1003 (*... [15:19:11] RESOLVED: SmartNotHealthy: Disk not healthy - https://wikitech.wikimedia.org/wiki/SMART#Alerts - https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&var-server=cloudcephosd1017 - https://alerts.wikimedia.org/?q=alertname%3DSmartNotHealthy [15:37:35] 10Tool-curator: Add abort function to Mapillary upload batches in Curator - https://phabricator.wikimedia.org/T414208 (10PantheraLeo1359531) 03NEW [15:45:58] 10Tool-curator: Handle Mapillary API errors instead of passing them to the frontend. - https://phabricator.wikimedia.org/T414210 (10DaxServer) 03NEW [15:46:22] (03CR) 10Ladsgroup: [C:03+2] write_config: Ignore problematic repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224787 (https://phabricator.wikimedia.org/T413322) (owner: 10Ammarpad) [15:47:04] 10Tool-curator: Handle Mapillary API errors instead of passing them to the frontend. - https://phabricator.wikimedia.org/T414210#11507918 (10DaxServer) 05Open→03In progress p:05Triage→03Medium [15:47:46] 10Tool-curator: Handle Mapillary API errors instead of passing them to the frontend. - https://phabricator.wikimedia.org/T414210#11507923 (10DaxServer) [15:47:47] 10Tool-curator: Handle large Mapillary sequences - https://phabricator.wikimedia.org/T412617#11507924 (10DaxServer) [15:47:53] (03Merged) 10jenkins-bot: write_config: Ignore problematic repo [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1224787 (https://phabricator.wikimedia.org/T413322) (owner: 10Ammarpad) [16:55:01] 06cloud-services-team, 06Infrastructure-Foundations: Move https://status.wikimedia.org/ away from rackspace - https://phabricator.wikimedia.org/T414098#11508166 (10tappof) @hnowlan have we already opted to keep the URL accessible? I mean, it’s (let me say) a 'redirect disclosure' from 2022… can we simply remov... [17:11:17] (03update) 10fnegri: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [17:12:08] (03approved) 10fnegri: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro) [17:21:16] (03update) 10fnegri: images: resolve the image every time [repos/cloud/toolforge/jobs-api] - 10https://gitlab.wikimedia.org/repos/cloud/toolforge/jobs-api/-/merge_requests/251 (owner: 10dcaro)