[00:47:28] 06Release-Engineering-Team (Priority Backlog 📥), 07Essential-Work, 05Release, 05Train Deployments: 1.48.0-wmf.6 deployment blockers - https://phabricator.wikimedia.org/T442224#12590500 (10SecurityPatchBot) [02:23:38] 10Continuous-Integration-Infrastructure, 10Castor, 06Test Platform (Aktau 32): Trial the cache proxy on a sampled non-voting copy of quibble-vendor-mysql-php85 - https://phabricator.wikimedia.org/T445734 (10Mhurd) 03NEW [03:50:13] 10GitLab (Account Approval), 06Release-Engineering-Team: Requesting GitLab account activation for tduk - https://phabricator.wikimedia.org/T445610#12590693 (10Pharos) 05Open→03Resolved [06:32:50] (03PS1) 10Mhurd: castor: skip the save trigger and npm cache verify when CASTOR_DISABLED=1 [integration/config] - 10https://gerrit.wikimedia.org/r/1373007 (https://phabricator.wikimedia.org/T445734) [06:32:52] (03PS1) 10Mhurd: jjb: trial the cache proxy on a sampled copy of quibble-vendor-mysql-php85 [integration/config] - 10https://gerrit.wikimedia.org/r/1373008 (https://phabricator.wikimedia.org/T445734) [07:35:01] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [07:35:16] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [07:35:23] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [07:36:03] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [07:37:01] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [07:48:51] (03update) 10mhurd: Add a dashboard for moving from Castor to the cache proxy [repos/releng/develstats] - 10https://gitlab.wikimedia.org/repos/releng/develstats/-/merge_requests/103 (https://phabricator.wikimedia.org/T445734) [08:26:10] (03update) 10addshore: Draft: Expand existing platform checks for Docker contexts and self-update [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/719 (https://phabricator.wikimedia.org/T384914 https://phabricator.wikimedia.org/T403412) [08:26:31] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [08:47:04] (03update) 10addshore: Draft: Add Wikidata SPARQL query command [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/677 [08:47:10] (03update) 10addshore: Draft: Add Wikimedia service target shortcuts [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/678 [08:47:16] (03update) 10addshore: Draft: Persist development service intent (T314892) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/718 (https://phabricator.wikimedia.org/T314892) [08:47:19] (03update) 10addshore: Draft: docs: specs for translatewiki.net gerrit onboarding (RFC) [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/738 (owner: 10siebrand) [08:47:23] (03update) 10addshore: Draft: Add remote development environments and background sessions [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/792 (https://phabricator.wikimedia.org/T443381) [09:39:43] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [11:41:35] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [12:14:05] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [12:33:04] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [14:20:59] Southparkfan: I wrote some of my choices at https://wikitech.wikimedia.org/wiki/MediaWiki_Engineering/Guides/diagrams.net_conventions [14:21:41] but ultimately there's no substitute for trying and changing things until it is as simple as it can be without being inaccurate [14:22:05] especially where to put boxes and arrows so that there's as few criss-cross lines as possible [14:22:47] I tend to end up with layouts that work well, but are also kind of locked in. nothing can be added or moved without everything having to change by a lot :) [14:36:16] Krinkle: Thanks for sharing that page. Looks like you have provided enough information even I could futher improve your diagrams. :) [14:37:13] As in: slightly update them - styling is still OK. [14:38:02] Tim has gathered some recent changes at https://wikitech.wikimedia.org/wiki/User:Tim_Starling_(WMF)/request_flow [14:38:24] so the infra diagram, could be updated based on that, e.g. to reflect that the app server clusters are now mw-on-k8 server groups. [14:39:48] and removing mathoid/restbase [14:40:27] we could make a list of changes that are needed and then you do the first pass? [14:41:26] That is very useful. And yes, the infrastructure diagram could be adjusted. In File:Wikipedia_webrequest_2022.png, renaming "app servers" to something else ought to be sufficient - listing all relevant server groups (mw-web, mw-api-ext, possibly others?) could clutter the diagram. [14:42:02] yeah, that diagram is intentionally simplified to ignore that the distiction of appserver clusters / servergroups. [14:42:09] Unless the server groups behave very differently, which I think they do not (?), would rather merge them into one container. :) [14:42:23] (in this case, less is more) [14:42:26] same as before. previosuly we had appservers, api_appservers, jobrunners, mwmaint [14:42:32] and now we have mw-web, mw-api-ext, mw-jobrunner, mw-cron [14:42:55] And: yes, let's gather a list of changes. +1. [14:43:04] for the infra diagram, it is shown as a way to show traffic routing [14:43:25] but for the simple "one web request" diagram, it is focussed more on mediawiki and a single pageview request as representative example, not every alternate branch [14:43:46] if you start the wiki page, I'll dump some stuff on it later [14:43:53] Slightly off-topic: I noticed you were working on decommissioning Mathoid. Currently working on removing RESTBase from the Beta Cluster where feasible. Looking forward to your progress! [14:44:10] Aye, I guess that's why https://krinklebot.toolforge.org/mathoid.php stopped working [14:44:12] Yes, I'll create a page. Thanks. [14:44:33] Hm. Mathoid should still be running, though. [14:44:51] It proxies to https://en.wikipedia.beta.wmcloud.org/api/rest_v1/media/math [14:45:00] does POST'ing there still work? [14:45:20] It's fine for MW to be unable to use it, it's more for a bit more testing for a few weeks while community compares and figures things out [14:45:41] not a hard requirement or deadline, but wanted to keep it running if we could. [14:46:15] Let me take a look. I didn't to interfere with your work on decommissioning Mathoid. [14:48:06] I did test a follow-up change for https://gerrit.wikimedia.org/r/c/operations/puppet/+/1182652 yesterday that offloads all /api/rest_v1 requests to other services, including Mathoid *and* including all endpoints that have equivalents in the MediaWiki REST API. For the latter endpoints, I managed to rewrite the URIs to correspond to REST API endpoints, and with the `x-restbase-compat` header set. [14:56:04] https://en.wikipedia.beta.wmcloud.org/api/rest_v1/media/math/check/tex throws a 403. I guess that's https://gerrit.wikimedia.org/r/c/operations/puppet/+/1371862? [14:57:34] On `deployment-restbase05`, `curl -i -X POST -H "Content-Type: application/json" -d '{"q": "E = mc^2"}' http://localhost:7231/en.wikipedia.beta.wmcloud.org/v1/media/math/check/tex` seems to work fine. [15:03:56] I have locally hacked HAProxy to 403 all requests *except* for requests using the MathoidToolforgeBot UA. Your tool seems to work fine now. Let me push this to Gerrit :) [15:19:46] !log Cherry-pick https://gerrit.wikimedia.org/r/c/operations/puppet/+/1373031 on Beta Cluster puppetserver to unbreak https://krinklebot.toolforge.org/mathoid.php - T271001 [15:19:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:19:49] T271001: Transition to client-side MathJax SVG rendering as default - https://phabricator.wikimedia.org/T271001 [15:27:52] Production Mathoid should work too. On deploy2003, https://restbase.discovery.wmnet:7443/en.wikipedia.org/v1/media/math/check/tex or http://localhost:6011/en.wikipedia.org/v1/media/math/check/tex (restbase listener on the services proxy) respond for me, and so does http://localhost:6003/ (mathoid listener on the services proxy). [17:10:55] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump golang.org/x/net from 0.62.0 to 0.63.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/829 [17:11:09] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): bump github.com/charmbracelet/x/ansi from 0.11.9 to 0.11.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/830 [17:11:25] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/modelcontextprotocol/go-sdk from 1.4.0 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/831 [17:11:40] (03open) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/modelcontextprotocol/go-sdk from 1.4.0 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/831 [17:11:45] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/modelcontextprotocol/go-sdk from 1.4.0 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/831 [17:11:46] (03update) 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6: build(deps): [security] bump github.com/modelcontextprotocol/go-sdk from 1.4.0 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/831 [17:14:10] 10Beta-Cluster-Infrastructure, 10Cloud-VPS (Debian Bullseye Deprecation), 13Patch-For-Review: Replace or decommission RESTBase hosts: deployment-restbase(05|-bullseye) (Bullseye deprecation) - https://phabricator.wikimedia.org/T436477#12591106 (10Southparkfan) a:03Southparkfan [17:34:05] !log Applied changeprop config https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/1373039/1 on Beta Cluster [17:34:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:31:05] (03merge) 10addshore: build(deps): [security] bump github.com/modelcontextprotocol/go-sdk from 1.4.0 to 1.4.1 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/831 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [18:45:46] (03update) 10addshore: Draft: Migrate CLI logging to slog and standardize output and errors [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/793 (owner: 10project_16_bot_df05b28c9b5fb77789cdd3cf152eb286) [18:45:56] (03update) 10addshore: build(deps): bump golang.org/x/net from 0.62.0 to 0.63.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/829 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [18:46:05] (03update) 10addshore: build(deps): bump github.com/charmbracelet/x/ansi from 0.11.9 to 0.11.10 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/830 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6) [19:13:59] (03merge) 10addshore: build(deps): bump golang.org/x/net from 0.62.0 to 0.63.0 [repos/releng/cli] - 10https://gitlab.wikimedia.org/repos/releng/cli/-/merge_requests/829 (owner: 10project_16_bot_622d7b6d682aa66907ad1bffedf5b9e6)