[01:41:51] 10PAWS, 06tools-platform-team: Full storage - https://phabricator.wikimedia.org/T431775 (10MinhTest01) 03NEW [02:16:00] 10PAWS, 06tools-platform-team: Full storage - https://phabricator.wikimedia.org/T431775#12107488 (10JJMC89) →14Duplicate dup:03T429578 [02:16:02] 06cloud-services-team, 10PAWS, 06tools-platform-team: PAWS disk space is running out - https://phabricator.wikimedia.org/T429578#12107490 (10JJMC89) [02:17:26] (03PS9) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [02:41:34] (03merge) 10samwilson: Delete cached file on initial page load [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/15 (https://phabricator.wikimedia.org/T430577) [02:43:19] (03merge) 10samwilson: Use library for CSS parsing when checking for ID selectors [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/14 (https://phabricator.wikimedia.org/T271595 https://phabricator.wikimedia.org/T420835) [02:44:02] (03close) 10samwilson: Resolve T420835 "Bug" [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/8 (owner: 10booksmurf) [02:49:13] (03PS10) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [02:49:48] (03CR) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [02:54:09] (03CR) 10Jsn.sherman: "I now have a local environment up and running, so I'm going to do some more testing on the rate limiter part to make sure it delivers what" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [03:14:12] (03PS11) 10Jsn.sherman: RateLimitSubscriber: anon db traffic + rate limits [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [03:38:34] (03open) 10samwilson: Fix some Phan errors [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/16 (https://phabricator.wikimedia.org/T316310) [05:08:19] 10Tool-wd-act: Clean up and fully flesh out Tool's documentation page - https://phabricator.wikimedia.org/T431058#12107649 (10Tekneos) 05Open→03In progress [05:12:18] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12107651 (10Osuji_pius) @Tiven2240 and @Nokib_Sarkar Please, this is another reminder to complete the mentor's midterm evaluation, which closes later today. Thank you so much [05:13:38] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12107652 (10Nokib_Sarkar) We submitted the evaluation, thanks for reminder [05:15:23] 10Tool-campwiz-nxt, 10Google-Summer-of-Code (2026): GSoC 2026: CampWiz NxT Redesign - https://phabricator.wikimedia.org/T414269#12107653 (10Osuji_pius) >>! In T414269#12107652, @Nokib_Sarkar wrote: > We submitted the evaluation, thanks for reminder Thank you! [05:36:37] 10Tool-wd-act: Determine how to maintain all required open source compliance - https://phabricator.wikimedia.org/T431027#12107675 (10Tekneos) This list is mentioned here - may need to update something when I work on this ticket: https://wikitech.wikimedia.org/wiki/Tool:WikiDataAuthorityControlTool#Technical_impl... [05:48:28] (03close) 10countcount: Fail cleanly when share-ips CLI input files cannot be read [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/3 [06:07:32] (03merge) 10samwilson: Fix some Phan errors [toolforge-repos/svgtranslate] - 10https://gitlab.wikimedia.org/toolforge-repos/svgtranslate/-/merge_requests/16 (https://phabricator.wikimedia.org/T316310) [06:11:48] (03merge) 10countcount: Update dependency typescript to v7 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/20 (owner: 10renovatebot) [06:15:55] (03update) 10renovatebot: Update pnpm to v11.11.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/22 [06:16:37] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [06:21:00] (03open) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/40 [06:21:10] (03open) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/5 [06:22:34] (03close) 10countcount: Update dependency typescript to v7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/38 (owner: 10renovatebot) [06:22:35] (03update) 10countcount: Update dependency typescript to v7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/38 (owner: 10renovatebot) [06:22:45] (03close) 10countcount: Update dependency typescript to v7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/2 (owner: 10renovatebot) [06:22:45] (03update) 10countcount: Update dependency typescript to v7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/2 (owner: 10renovatebot) [06:27:05] (03update) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/40 [06:27:05] (03update) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/5 [06:27:05] (03update) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/40 [06:27:05] (03update) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/5 [06:32:08] (03merge) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/5 [06:32:17] (03merge) 10countcount: Hold TypeScript at 6 until vue-tsc supports TypeScript 7 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/40 [06:33:05] (03update) 10countcount: Update pnpm to v11.11.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/39 (owner: 10renovatebot) [06:33:09] (03merge) 10countcount: Update pnpm to v11.11.0 [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/4 (owner: 10renovatebot) [06:41:03] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [06:41:36] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [06:47:45] (03merge) 10countcount: Update pnpm to v11.11.0 [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/39 (owner: 10renovatebot) [06:48:00] (03merge) 10countcount: Update pnpm to v11.11.0 [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/22 (owner: 10renovatebot) [06:51:03] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [06:51:36] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [07:16:11] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [07:18:08] 10Tool-wd-act: Clean up and fully flesh out Tool's documentation page - https://phabricator.wikimedia.org/T431058#12107745 (10Tekneos) 05In progress→03Resolved a:03Tekneos Done. Everything's fleshed out and lookin' good. Closing. [07:21:54] 10Tool-wd-act: Separate authority control fields in the UI based on category - https://phabricator.wikimedia.org/T431783 (10Tekneos) 03NEW [07:25:44] 10Tool-wd-act: Figure out hosting strategy, likely with ToolForge - https://phabricator.wikimedia.org/T431033#12107763 (10Tekneos) 05Open→03In progress Working on this now [07:38:48] FIRING: PuppetFailure: Puppet has failed on cloudvirt1073:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [07:45:48] 10Tool-wd-act: Figure out hosting strategy, likely with ToolForge - https://phabricator.wikimedia.org/T431033#12107819 (10Tekneos) This section needs some serious review - there's a lot to go through in the terms of use: https://wikitech.wikimedia.org/wiki/Help:Toolforge/Quickstart#Read_and_agree_to_terms_of_use [07:58:48] RESOLVED: PuppetFailure: Puppet has failed on cloudvirt1073:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [08:00:00] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12107905 (10Kavaljeet_Singh) It works now. I was able to complete the Toolforge membership process. Thank you for your help and for fixi... [08:00:28] 06cloud-services-team, 10Striker, 06tools-platform-team: User should be told name of existing Developer account when SUL is already in use - https://phabricator.wikimedia.org/T294767#12107911 (10Kavaljeet_Singh) [08:00:29] 06cloud-services-team, 10Striker, 06tools-platform-team: Cannot join Toolforge: Striker throws IntegrityError while saving user - https://phabricator.wikimedia.org/T429759#12107909 (10Kavaljeet_Singh) 05Open→03Resolved [08:34:56] 06cloud-services-team, 10Cloud-VPS, 06tools-infrastructure-team: Revisit cloudvirt maintenance story - https://phabricator.wikimedia.org/T424802#12108058 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi This is done in the sense that `maintenance` aggregate is no longer a thing. I have updated https... [09:25:21] (03update) 10taavi: Update security groups for metricsinfra VXLAN/Dualstack IPs [repos/cloud/cloud-vps/tofu-infra] - 10https://gitlab.wikimedia.org/repos/cloud/cloud-vps/tofu-infra/-/merge_requests/332 (https://phabricator.wikimedia.org/T401813) [09:57:30] 06cloud-services-team, 10Toolforge, 06tools-platform-team: [lima-kilo] check for k8s health (services running) / ldap service container running prior to executing account injection - https://phabricator.wikimedia.org/T431686#12108342 (10DamianZaremba) I didn't managed to grab the logs. I suspect my connectiv... [09:58:48] FIRING: PuppetFailure: Puppet has failed on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [10:13:48] RESOLVED: PuppetFailure: Puppet has failed on cloudlb1001:9100 - https://puppetboard.wikimedia.org/nodes?status=failed - https://grafana.wikimedia.org/d/yOxVDGvWk/puppet - https://alerts.wikimedia.org/?q=alertname%3DPuppetFailure [11:06:56] 10Toolforge, 06tools-platform-team: Look into spots where we can surface user statistics from toolforge and VPS - https://phabricator.wikimedia.org/T206793#12108526 (10aputhin) [11:08:13] 10Cloud-VPS, 10Toolforge, 06tools-infrastructure-team, 06tools-platform-team: Create phab tag/form/workflow/something for volunteer priv requests - https://phabricator.wikimedia.org/T212012#12108529 (10aputhin) [11:10:57] 06cloud-services-team: Schedule regular failover and restore testing -- design this at least roughly and add to quarterly goals - https://phabricator.wikimedia.org/T227212#12108551 (10aputhin) 05Open→03Invalid Not very clear what this is about. I'll close for now under the same grounds as the parent. [11:13:11] 06cloud-services-team: Create a couple of technical tasks available for new technical contributors and external folks - https://phabricator.wikimedia.org/T220052#12108565 (10aputhin) 05Open→03Invalid [11:14:47] 10Cloud-VPS, 06tools-infrastructure-team: Delegate lists.wmcloud.org domain to be able to add DNS DKIM records - https://phabricator.wikimedia.org/T278358#12108570 (10aputhin) [11:16:59] 06cloud-services-team: Test using phabricator-maintenance-bot to sync wmcs-related boards - https://phabricator.wikimedia.org/T358251#12108601 (10aputhin) 05Open→03Resolved a:03aputhin Done to an extent, and now we're also syncing things within each team's board. [11:20:53] 06cloud-services-team: Test using phabricator-maintenance-bot to sync wmcs-related boards - https://phabricator.wikimedia.org/T358251#12108622 (10aputhin) a:05aputhin→03None [11:22:18] 10Toolforge, 06tools-platform-team: debian packaging: create common guidelines and workflow - https://phabricator.wikimedia.org/T212291#12108628 (10aputhin) Is there more work to be done here? [11:23:18] 06tools-infrastructure-team: Automate cluster reboots with cookbooks - https://phabricator.wikimedia.org/T426727#12108634 (10aputhin) Not sure which project exactly this relates to, so assigning just infra team for now. [11:34:09] 06cloud-services-team: Improve WMCS offboarding process - https://phabricator.wikimedia.org/T398215#12108687 (10aputhin) 05Open→03Resolved Since there are subtasks, I'll close this one. If there's any follow ups needed, we can create more specific tasks again. [11:35:04] 10Cloud-VPS, 06tools-infrastructure-team: Advice for Hybrid Architecture for AI Tool - https://phabricator.wikimedia.org/T416590#12108690 (10aputhin) [11:35:51] 10Cloud-VPS, 06tools-infrastructure-team: SystemdUnitDown and SystemdUnitDownForLong - ditch the latter (?) - https://phabricator.wikimedia.org/T411193#12108692 (10aputhin) [11:41:55] (03CR) 10Thiemo Kreuz (WMDE): Clarify placeholder: regexp not wrapped in `/` (031 comment) [labs/codesearch] - 10https://gerrit.wikimedia.org/r/1306096 (https://phabricator.wikimedia.org/T371191) (owner: 10Simon04) [11:46:53] 10Toolforge, 06tools-platform-team: [lima-kilo] check for k8s health (services running) / ldap service container running prior to executing account injection - https://phabricator.wikimedia.org/T431686#12108705 (10aputhin) [11:56:48] (03open) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/6 [11:57:02] (03open) 10countcount: Trim derivable content from AGENTS.md [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/42 [11:57:09] (03open) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/28 [11:57:37] (03open) 10countcount: Trim boilerplate scaffold sections from AGENTS.md [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/23 [11:58:03] (03open) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/37 [11:58:12] (03open) 10countcount: Trim derivable content from AGENTS.md [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/41 [12:25:04] (03PS1) 10ShivangiU2: Compact depict search results to reduce metadata overlap [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1309665 (https://phabricator.wikimedia.org/T231751) [12:59:32] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [12:59:59] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:04:32] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:04:59] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:08:44] FIRING: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [13:09:03] FIRING: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:09:36] FIRING: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:09:48] (03merge) 10countcount: Trim derivable content from AGENTS.md [toolforge-repos/rangetree] - 10https://gitlab.wikimedia.org/toolforge-repos/rangetree/-/merge_requests/41 [13:09:53] (03merge) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/37 [13:09:56] (03merge) 10countcount: Trim boilerplate scaffold sections from AGENTS.md [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/23 [13:10:06] (03merge) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/28 [13:10:17] FIRING: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [13:10:17] (03merge) 10countcount: Trim derivable content from AGENTS.md [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/42 [13:10:23] (03merge) 10countcount: Trim derivable content from CLAUDE.md [toolforge-repos/checkusertools] - 10https://gitlab.wikimedia.org/toolforge-repos/checkusertools/-/merge_requests/6 [13:13:05] FIRING: HostBGPDown: BGP session for cloudservices2004-dev (2a02:ec80:a100:205::8) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cloudsw1-b1-codfw:9804&var-bgp_group=cloud_host6 - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [13:14:03] RESOLVED: PawsJupyterHubDown: PAWS JupyterHub is down https://wikitech.wikimedia.org/wiki/PAWS/Admin - https://prometheus-alerts.wmcloud.org/?q=alertname%3DPawsJupyterHubDown [13:14:36] RESOLVED: TargetDown: Job jupyterhub is unreachable in project paws instance hub-paws.wmcloud.org:443 - https://prometheus-alerts.wmcloud.org/?q=alertname%3DTargetDown [13:16:09] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/multiuserinfo] - 10https://gitlab.wikimedia.org/toolforge-repos/multiuserinfo/-/merge_requests/28 [13:17:08] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/wiki-mail-verify] - 10https://gitlab.wikimedia.org/toolforge-repos/wiki-mail-verify/-/merge_requests/18 [13:17:49] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewiki-voterightsnotifier] - 10https://gitlab.wikimedia.org/toolforge-repos/dewiki-voterightsnotifier/-/merge_requests/35 [13:18:05] RESOLVED: HostBGPDown: BGP session for cloudservices2004-dev (2a02:ec80:a100:205::8) is down - https://wikitech.wikimedia.org/wiki/Network_monitoring#BGP_status - https://grafana.wikimedia.org/d/ed8da087-4bcb-407d-9596-d158b8145d45/bgp-neighbors-detail?orgId=1&var-site=codfw&var-device=cloudsw1-b1-codfw:9804&var-bgp_group=cloud_host6 - https://alerts.wikimedia.org/?q=alertname%3DHostBGPDown [13:18:16] (03update) 10renovatebot: Lock file maintenance [toolforge-repos/dewikisignbot] - 10https://gitlab.wikimedia.org/toolforge-repos/dewikisignbot/-/merge_requests/26 [13:20:17] RESOLVED: [2x] JobUnavailable: Reduced availability for job pdns in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:08:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [14:11:37] FIRING: NfsAlmostFull: The NFS drive is over 85% capacity (currently 100%) at host paws-nfs-2 in project paws - https://prometheus-alerts.wmcloud.org/?q=alertname%3DNfsAlmostFull [14:32:17] FIRING: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:38:32] FIRING: InstanceDown: Project gitlab-runners instance runner-1035 is down - https://prometheus-alerts.wmcloud.org/?q=alertname%3DInstanceDown [14:42:17] RESOLVED: JobUnavailable: Reduced availability for job openstack in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [14:50:00] 10Cloud-VPS, 06tools-infrastructure-team, 06Infrastructure-Foundations, 10netops, 06SRE: Upgrade cloudsw1-e4-eqiad - https://phabricator.wikimedia.org/T429013#12109624 (10fgiunchedi) >>! In T429013#12109258, @cmooney wrote: > Thanks @fgiunchedi. > > This isn't so urgent we want to cause stress for you g... [15:14:39] 10PAWS, 06tools-platform-team: [bug] Notebook container not spawing for user Soumendrapatra21 - https://phabricator.wikimedia.org/T431744#12109759 (10dcaro) Hi @Soumendrapatra21, you account has been flagged for suspicious activity, can you explain in detail what were you using your account for and how does it... [15:29:55] 10Toolforge (Push-to-Deploy), 06tools-platform-team: jobs-api, jobs-cli: support --webservice option in toolforge jobs run - https://phabricator.wikimedia.org/T431410#12109818 (10JJMC89) →14Duplicate dup:03T428898 [15:29:56] 10Toolforge (Push-to-Deploy), 06tools-platform-team: [jobs-cli, jobs-api] support --webservice option - https://phabricator.wikimedia.org/T428898#12109819 (10JJMC89) [15:58:32] (03PS12) 10Jsn.sherman: RateLimitSubscriber: cut anon session db traffic [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [15:58:32] (03PS1) 10Jsn.sherman: RateLimitSubscriber: limit before replica access [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309690 (https://phabricator.wikimedia.org/T430888) [16:07:38] (03CR) 10Jsn.sherman: "Okay, I ended up doing a deep dive on the rate limiter and IP handling. I pulled the subnet stuff back out of this changelist and opened u" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1307269 (https://phabricator.wikimedia.org/T430888) (owner: 10MusikAnimal) [16:13:44] RESOLVED: MaintainDBUsersManyErrors: Maintain-dbusers is having sustained errors - https://wikitech.wikimedia.org/wiki/Portal:Toolforge/Admin/Runbooks/MaintainDBUsersManyErrors - https://grafana.wikimedia.org/d/ae240a06-c13e-49f3-b12c-58432c551e85/wmcs-maintain-dbusers - https://alerts.wikimedia.org/?q=alertname%3DMaintainDBUsersManyErrors [16:43:14] (03CR) 10Jsn.sherman: "So this patch should fix up the ratelimiter to help it short circuit more replica db calls without making real structural changes. I'm wor" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309690 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [16:46:57] (03CR) 10Jsn.sherman: "hmm, that followup would be less of a clear win since the current constructor arrangement keeps things dry and allows for early validation" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309690 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [16:49:50] 06cloud-services-team, 06collaboration-services, 10Infrastructure Security, 06Infrastructure-Foundations, and 6 others: July 2026 Bookworm/Trixie reboots: Data Platform SRE - https://phabricator.wikimedia.org/T431826#12110010 (10colewhite) [17:21:47] (03CR) 10Jsn.sherman: "okay, I think the next highest value followup is increasing our survivability of issues with meta/dblist. While I'm in there working on th" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309690 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [17:32:09] (03PS2) 10ShivangiU2: Compact depicts search results to reduce metadata overlap [labs/tools/Isa] - 10https://gerrit.wikimedia.org/r/1309665 (https://phabricator.wikimedia.org/T231751) [17:38:41] FIRING: CloudVPSDesignateLeaks: Detected 2 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [19:03:19] 10Tool-wikinewsie, 06WikiNYC Tech Committee: ITN/Wikipulse platform - https://phabricator.wikimedia.org/T427575#12110457 (10Legoktm) [19:12:39] 10Tool-wikinewsie, 06WikiNYC Tech Committee: ITN/Wikipulse platform - https://phabricator.wikimedia.org/T427575#12110504 (10ZhaoFJx) 05Open→03In progress [19:14:09] 10Tool-wikinewsie: Recent Changes page - https://phabricator.wikimedia.org/T431876 (10Pharos) 03NEW [19:15:35] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinews Pulse tool on Toolforge - https://phabricator.wikimedia.org/T427575#12110524 (10Pharos) [19:15:36] 10Tool-wikinewsie, 06WikiNYC Tech Committee: ITN/Wikipulse platform - https://phabricator.wikimedia.org/T427575#12110525 (10Pharos) [19:16:16] 10Tool-wikinewsie, 06WikiNYC Tech Committee: Set up Wikinews Pulse tool on Toolforge - https://phabricator.wikimedia.org/T427575#12110527 (10Pharos) [19:16:46] 10Tool-wikinewsie: Recent Changes page for Wikinewsie - https://phabricator.wikimedia.org/T431876#12110528 (10Pharos) [19:21:32] 10Tool-wikinewsie: Language Selector Fallback for Dash Included Code - https://phabricator.wikimedia.org/T431877 (10ZhaoFJx) 03NEW [19:23:10] 10Tool-wikinewsie: Semi-protection for images on Wikinewsie - https://phabricator.wikimedia.org/T431878 (10Pharos) 03NEW [19:24:04] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12110566 (10ZhaoFJx) [19:31:36] 10Tool-wikinewsie: New articles should be their own event type - https://phabricator.wikimedia.org/T431879 (10Pharos) 03NEW [19:33:12] 10Tool-wikinewsie: New Wikipedia articles should be their own event type on Wikinewsie - https://phabricator.wikimedia.org/T431879#12110596 (10Pharos) [19:37:12] 10Tool-adminscore, 06Moderator-Tools-Team, 10XTools: Redirect XTools adminscore URLs to adminscore.toolforge.org - https://phabricator.wikimedia.org/T431880 (10Legoktm) 03NEW [19:51:45] 10Tool-wikinewsie: Wikinewsie images often off-center - https://phabricator.wikimedia.org/T431881 (10Pharos) 03NEW [20:00:45] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12110665 (10Pharos) Probably not the same issue, but zh and ko both return no results, and jp returns "Query failed". [20:04:25] 10Tool-wikinewsie: Semi-protection for images on Wikinewsie - https://phabricator.wikimedia.org/T431878#12110674 (10Pharos) [20:58:24] (03PS1) 10Jsn.sherman: Repository: 503 + breaker for dead replicas [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309732 (https://phabricator.wikimedia.org/T430888) [20:58:25] (03PS1) 10Jsn.sherman: ProjectRepository: fail over to Sitematrix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) [20:59:02] (03CR) 10CI reject: [V:04-1] Repository: 503 + breaker for dead replicas [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309732 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [20:59:18] (03CR) 10CI reject: [V:04-1] ProjectRepository: fail over to Sitematrix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [21:00:14] (03CR) 10Alien4444: "@jsherman@wikimedia.org Re meta/dblists, do keep in mind I07e30af1e5521487f3950a9bc8fd7951583438f8 (waiting for review), which already dea" [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309690 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman) [21:06:17] FIRING: JobUnavailable: Reduced availability for job blackbox_https in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:11:17] RESOLVED: JobUnavailable: Reduced availability for job blackbox_https in cloud@codfw - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_job_unavailable - https://grafana.wikimedia.org/d/NEJu05xZz/prometheus-targets - https://alerts.wikimedia.org/?q=alertname%3DJobUnavailable [21:11:41] FIRING: PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [21:26:41] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [21:34:40] 10Tool-sentinel: [Security] Stop returning Meta OAuth upstream response bodies to clients - https://phabricator.wikimedia.org/T431884 (10Arian_Ar) 03NEW [21:36:56] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [21:38:56] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [21:47:59] 10Tool-sentinel: Allowlist wiki_url when creating bookmarks - https://phabricator.wikimedia.org/T431886 (10Arian_Ar) 03NEW [21:48:45] 10Tool-sentinel: Allowlist wiki_url on GET /api/feed/diff - https://phabricator.wikimedia.org/T431887 (10Arian_Ar) 03NEW [21:50:34] 10Tool-sentinel: Allowlist wiki_url when creating bookmarks - https://phabricator.wikimedia.org/T431886#12110933 (10Arian_Ar) [21:50:56] 10Tool-sentinel: Allowlist wiki_url on GET /api/feed/diff - https://phabricator.wikimedia.org/T431887#12110934 (10Arian_Ar) [21:51:56] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=codfw%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [21:55:57] 10Tool-sentinel: Feed poller must not use long-lived MediaWiki Redis cache - https://phabricator.wikimedia.org/T431888 (10Arian_Ar) 03NEW [21:56:55] 10Tool-sentinel: Session / WebSocket lifecycle on suspend or reject - https://phabricator.wikimedia.org/T431889 (10Arian_Ar) 03NEW [21:58:13] 10Tool-sentinel: OAuth token storage and encryption fail-open - https://phabricator.wikimedia.org/T431890 (10Arian_Ar) 03NEW [21:58:51] 10Tool-sentinel: Handle rejected account status in frontend router - https://phabricator.wikimedia.org/T431891 (10Arian_Ar) 03NEW [21:59:01] FIRING: OpenstackAPIResponse: Openstack API average response time is too high. - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/OpenstackAPIResponse - https://grafana.wikimedia.org/d/UUmLqqX4k - https://alerts.wikimedia.org/?q=alertname%3DOpenstackAPIResponse [21:59:31] 10Tool-sentinel: Hard-delete parent note with replies fails (500) - https://phabricator.wikimedia.org/T431892 (10Arian_Ar) 03NEW [22:01:20] 10Tool-sentinel: Restrict admin approve/reject to valid status transitions - https://phabricator.wikimedia.org/T431893 (10Arian_Ar) 03NEW [22:01:52] 10Tool-sentinel: Shared collections incomplete for non-owners - https://phabricator.wikimedia.org/T431894 (10Arian_Ar) 03NEW [22:02:30] 10Tool-sentinel: Integration tests for sessions and ownership isolation - https://phabricator.wikimedia.org/T431895 (10Arian_Ar) 03NEW [22:03:00] 10Tool-sentinel: Per-user rate limits for MediaWiki-backed endpoints - https://phabricator.wikimedia.org/T431896 (10Arian_Ar) 03NEW [22:03:30] 10Tool-sentinel: Set Secure flag on session cookie - https://phabricator.wikimedia.org/T431897 (10Arian_Ar) 03NEW [22:03:41] RESOLVED: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:03:43] 10Tool-sentinel: Set Secure flag on session cookie - https://phabricator.wikimedia.org/T431897#12111056 (10Arian_Ar) p:05Triage→03Low [22:04:24] 10Tool-sentinel: Atomic OAuth state consume (GETDEL) - https://phabricator.wikimedia.org/T431898 (10Arian_Ar) 03NEW [22:04:46] 10Tool-sentinel: Do not return Meta OAuth upstream response bodies to clients - https://phabricator.wikimedia.org/T431899 (10Arian_Ar) 03NEW [22:04:46] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:05:25] 10Tool-sentinel: Guard multi-worker webservice against in-process poller/EventStreams - https://phabricator.wikimedia.org/T431900 (10Arian_Ar) 03NEW [22:05:39] (03PS2) 10Jsn.sherman: Repository: 503 + breaker for dead replicas [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309732 (https://phabricator.wikimedia.org/T430888) [22:05:39] (03PS2) 10Jsn.sherman: ProjectRepository: fail over to Sitematrix [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) [22:05:41] 10Tool-sentinel: Audit log when a note is flagged - https://phabricator.wikimedia.org/T431901 (10Arian_Ar) 03NEW [22:06:03] 10Tool-sentinel: Unique constraint on bookmarks - https://phabricator.wikimedia.org/T431902 (10Arian_Ar) 03NEW [22:06:42] 10Tool-sentinel: Refresh username/email on OAuth re-login - https://phabricator.wikimedia.org/T431903 (10Arian_Ar) 03NEW [22:07:12] 10Tool-sentinel: Reduce CPU use in feed WebSocket Redis loop - https://phabricator.wikimedia.org/T431904 (10Arian_Ar) 03NEW [22:07:41] FIRING: CloudVPSDesignateLeaks: Detected 8 stray dns records - https://wikitech.wikimedia.org/wiki/Portal:Cloud_VPS/Admin/Runbooks/Designate_record_leaks - https://grafana.wikimedia.org/d/ebJoA6VWz/wmcs-openstack-eqiad-nova-fullstack - https://alerts.wikimedia.org/?q=alertname%3DCloudVPSDesignateLeaks [22:07:42] 10Tool-sentinel: Remove or wire unused SECRET_KEY setting - https://phabricator.wikimedia.org/T431905 (10Arian_Ar) 03NEW [22:08:25] 10Tool-sentinel: Health endpoint should not expose filesystem path - https://phabricator.wikimedia.org/T431906 (10Arian_Ar) 03NEW [22:08:47] 10Tool-sentinel: Remove redundant local import json in mediawiki.py - https://phabricator.wikimedia.org/T431907 (10Arian_Ar) 03NEW [22:08:49] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12111173 (10ZhaoFJx) a:03ZhaoFJx [22:08:56] 10Tool-wikinewsie: Wikinewsie Language Selector Fallback when Dash Included - https://phabricator.wikimedia.org/T431877#12111176 (10ZhaoFJx) 05Open→03In progress [22:09:18] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908 (10Arian_Ar) 03NEW [22:09:58] 10Tool-sentinel: [Security] Stop returning Meta OAuth upstream response bodies to clients - https://phabricator.wikimedia.org/T431884#12111189 (10Arian_Ar) 05Open→03Invalid [22:10:28] 10Tool-sentinel: Allowlist wiki_url when creating bookmarks - https://phabricator.wikimedia.org/T431886#12111190 (10Arian_Ar) 05Open→03In progress p:05Triage→03High a:03Arian_Ar [22:10:58] 10Tool-sentinel: Allowlist wiki_url on GET /api/feed/diff - https://phabricator.wikimedia.org/T431887#12111193 (10Arian_Ar) p:05Triage→03High a:03Arian_Ar [22:11:37] 10Tool-sentinel: Feed poller must not use long-lived MediaWiki Redis cache - https://phabricator.wikimedia.org/T431888#12111195 (10Arian_Ar) p:05Triage→03High a:03Arian_Ar [22:11:54] 10Tool-sentinel: Session / WebSocket lifecycle on suspend or reject - https://phabricator.wikimedia.org/T431889#12111197 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:14:11] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:14:25] 10Tool-sentinel: OAuth token storage and encryption fail-open - https://phabricator.wikimedia.org/T431890#12111202 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:14:41] 10Tool-sentinel: Handle rejected account status in frontend router - https://phabricator.wikimedia.org/T431891#12111204 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:14:59] 10Tool-sentinel: Hard-delete parent note with replies fails (500) - https://phabricator.wikimedia.org/T431892#12111208 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:15:16] 10Tool-sentinel: Restrict admin approve/reject to valid status transitions - https://phabricator.wikimedia.org/T431893#12111210 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:15:33] 10Tool-sentinel: Shared collections incomplete for non-owners - https://phabricator.wikimedia.org/T431894#12111212 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:15:50] 10Tool-sentinel: Integration tests for sessions and ownership isolation - https://phabricator.wikimedia.org/T431895#12111214 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:16:04] 10Tool-sentinel: Per-user rate limits for MediaWiki-backed endpoints - https://phabricator.wikimedia.org/T431896#12111216 (10Arian_Ar) p:05Triage→03Medium a:03Arian_Ar [22:16:33] 10Tool-sentinel: Atomic OAuth state consume (GETDEL) - https://phabricator.wikimedia.org/T431898#12111219 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:16:50] 10Tool-sentinel: Do not return Meta OAuth upstream response bodies to clients - https://phabricator.wikimedia.org/T431899#12111221 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:17:20] 10Tool-sentinel: Guard multi-worker webservice against in-process poller/EventStreams - https://phabricator.wikimedia.org/T431900#12111224 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:17:29] 10Tool-sentinel: Audit log when a note is flagged - https://phabricator.wikimedia.org/T431901#12111226 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:17:40] 10Tool-sentinel: Unique constraint on bookmarks - https://phabricator.wikimedia.org/T431902#12111229 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:17:50] 10Tool-sentinel: Refresh username/email on OAuth re-login - https://phabricator.wikimedia.org/T431903#12111231 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:18:03] 10Tool-sentinel: Reduce CPU use in feed WebSocket Redis loop - https://phabricator.wikimedia.org/T431904#12111233 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:18:14] 10Tool-sentinel: Remove or wire unused SECRET_KEY setting - https://phabricator.wikimedia.org/T431905#12111235 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:18:27] 10Tool-sentinel: Health endpoint should not expose filesystem path - https://phabricator.wikimedia.org/T431906#12111237 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:18:36] 10Tool-sentinel: Remove redundant local import json in mediawiki.py - https://phabricator.wikimedia.org/T431907#12111239 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:18:47] 10Tool-sentinel: Set last_login on first OAuth user create - https://phabricator.wikimedia.org/T431908#12111241 (10Arian_Ar) p:05Triage→03Low a:03Arian_Ar [22:29:11] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:29:26] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:31:37] (03open) 10arian-ar: Allowlist wiki_url on bookmark create (T431886) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/9 [22:34:11] FIRING: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:37:52] 10Tool-adminscore, 06Moderator-Tools-Team, 10XTools: Redirect XTools adminscore URLs to adminscore.toolforge.org - https://phabricator.wikimedia.org/T431880#12111280 (10MusikAnimal) cc @Alien333 @Matthewrbowker for transparency. We've long believed Admin Score didn't make sense in XTools as it's so enwiki-ce... [22:39:11] RESOLVED: [2x] PrometheusRestarted: Prometheus/cloud restarted: beware monitoring artifacts. - https://wikitech.wikimedia.org/wiki/Prometheus#Prometheus_was_restarted - https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server?var-datasource=eqiad%20prometheus%2Fcloud - https://alerts.wikimedia.org/?q=alertname%3DPrometheusRestarted [22:40:16] (03update) 10arian-ar: Allowlist wiki_url on bookmark create (T431886) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/9 (https://phabricator.wikimedia.org/T431886) [22:42:39] (03update) 10arian-ar: Allowlist wiki_url on bookmark create (T431886) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/9 (https://phabricator.wikimedia.org/T431886) [22:43:02] (03merge) 10arian-ar: Allowlist wiki_url on bookmark create (T431886) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/9 (https://phabricator.wikimedia.org/T431886) [22:54:16] (03open) 10arian-ar: Allowlist wiki_url on GET /api/feed/diff (T431887) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/10 (https://phabricator.wikimedia.org/T431887) [22:55:44] (03merge) 10arian-ar: Allowlist wiki_url on GET /api/feed/diff (T431887) [toolforge-repos/sentinel] - 10https://gitlab.wikimedia.org/toolforge-repos/sentinel/-/merge_requests/10 (https://phabricator.wikimedia.org/T431887) [23:31:55] (03CR) 10Alien4444: ProjectRepository: fail over to Sitematrix (031 comment) [labs/xtools] - 10https://gerrit.wikimedia.org/r/1309733 (https://phabricator.wikimedia.org/T430888) (owner: 10Jsn.sherman)