[02:38:35] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:49:49] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:53:35] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [02:55:45] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [02:58:32] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:07:37] PROBLEM - Work requests waiting in Zuul Gearman server on contint2002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [400.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [03:13:31] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:18:31] (Queue (Jenkins jobs + Zuul functions) alert) firing: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:33:32] (Queue (Jenkins jobs + Zuul functions) alert) resolved: - https://alerts.wikimedia.org/?q=alertname%3DQueue+%28Jenkins+jobs+%2B+Zuul+functions%29+alert [03:34:11] RECOVERY - Work requests waiting in Zuul Gearman server on contint2002 is OK: OK: Less than 100.00% above the threshold [200.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/d/000000322/zuul-gearman?orgId=1&viewPanel=10 [04:02:13] 10Phabricator, 10Patch-For-Review: Remove custom go-get / go-import code in Diffusion repositories - https://phabricator.wikimedia.org/T352322 (10Aklapper) Note to myself: Also remove corresponding code in rPHEX afterwards [04:12:48] 10Phabricator: Remove unused code in Diffusion to handle closed almanac cluster services - https://phabricator.wikimedia.org/T352530 (10Aklapper) a:03Aklapper [04:13:38] 10Phabricator, 10Patch-For-Review: Remove unused code in Diffusion to handle closed almanac cluster services - https://phabricator.wikimedia.org/T352530 (10CodeReviewBot) aklapper opened https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/26 Revert custom changes to Almanac cluster ser... [04:16:18] 10Differential, 10Phabricator, 10Wikibugs, 10NewFunctionality-Worktype: Create conduit method to query the feed and return records with relevant details populated instead of just a bunch of phids - https://phabricator.wikimedia.org/T123417 (10Aklapper) I stumbled upon rPHAB586aaa547ade5bf97fa02e2c8e11511b0... [06:59:52] 10Phabricator: Production error phabricator form is missing an obvious place to place the RequestID - https://phabricator.wikimedia.org/T352629 (10Novem_Linguae) [09:28:52] 10Beta-Cluster-Infrastructure: Request for another period of interface admin right on zh Beta Cluster - https://phabricator.wikimedia.org/T352641 (10Diskdance) [10:32:14] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow 🦃), 10gitlab-settings, 10collaboration-services, and 2 others: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10CodeReviewBot) jelto merged https://gitlab.wikim... [11:04:18] 10Phabricator, 10translatewiki.net: Phabricator and translatewiki disagree over how to indent JSON files - https://phabricator.wikimedia.org/T349989 (10Nikerabbit) p:05Triage→03Low [11:19:19] 10GitLab (Administration, Settings & Policy), 10Release-Engineering-Team (Quid Pro Crow 🦃), 10gitlab-settings, 10collaboration-services, 10User-brennen: Define "Terms of Service and Privacy Policy" text for GitLab - https://phabricator.wikimedia.org/T285354 (10Jelto) 05Open→03Resolved The Terms of Se... [11:25:27] 10Release-Engineering-Team, 10Dumps-Generation, 10MW-on-K8s, 10serviceops: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650 (10Joe) [11:25:52] 10Release-Engineering-Team, 10Dumps-Generation, 10MW-on-K8s, 10serviceops: Migrate current-generation dumps to run from our containerized images - https://phabricator.wikimedia.org/T352650 (10Joe) p:05Triage→03Medium [11:49:04] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10daniel) 05Resolved→03Open Re-opening, since we still see CI for restbasefai... [11:49:20] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10daniel) [11:56:16] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10Vgutierrez) p:05Unbreak!→03High @daniel what's time outing is `parsoid-exte... [13:08:20] (03PS3) 10Jforrester: Zuul: [mediawiki/extensions/Variables] Re-instate voting CI [integration/config] - 10https://gerrit.wikimedia.org/r/979221 (owner: 10Func) [13:08:23] (03CR) 10Jforrester: [C: 03+2] Zuul: [mediawiki/extensions/Variables] Re-instate voting CI [integration/config] - 10https://gerrit.wikimedia.org/r/979221 (owner: 10Func) [13:09:48] (03Merged) 10jenkins-bot: Zuul: [mediawiki/extensions/Variables] Re-instate voting CI [integration/config] - 10https://gerrit.wikimedia.org/r/979221 (owner: 10Func) [13:47:04] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10Jdforrester-WMF) Looks like https://github.com/wikimedia/restbase/commit/2d9006... [13:48:40] 10Phabricator, 10Wikimedia-production-error: Production error phabricator form is missing an obvious place to place the RequestID - https://phabricator.wikimedia.org/T352629 (10Aklapper) [13:51:38] (03CR) 10Hashar: [C: 03+2] Archive operations/software/varnish/libvmod-netmapper [integration/config] - 10https://gerrit.wikimedia.org/r/979438 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [13:52:41] 10Phabricator: Remove unneeded custom changes in webroot/rsrc/css/phui/phui-property-list-view.css - https://phabricator.wikimedia.org/T352667 (10Aklapper) p:05Triage→03Low [13:53:39] (03Merged) 10jenkins-bot: Archive operations/software/varnish/libvmod-netmapper [integration/config] - 10https://gerrit.wikimedia.org/r/979438 (https://phabricator.wikimedia.org/T347623) (owner: 10BCornwall) [14:04:11] 10Continuous-Integration-Config, 10PHP 8.3 support: Add experimental PHP 8.3 jobs - https://phabricator.wikimedia.org/T339349 (10Jdforrester-WMF) >>! In T339349#9367398, @Jdforrester-WMF wrote: >>>! In T339349#9366041, @Michael wrote: >> I tried it out (`check experimental`) on a Wikibase change and it quickly... [14:27:41] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, 10Security: Add commit-message-validator for gitlab security ci includes merge requests - https://phabricator.wikimedia.org/T352331 (10CodeReviewBot) mmartorana merged https://gitlab.wikimedia.org/repos/secu... [14:29:22] (03PS1) 10AikoChou: ml-services: renamed directory for revert_risk_model [integration/config] - 10https://gerrit.wikimedia.org/r/979974 (https://phabricator.wikimedia.org/T352181) [14:36:48] (03CR) 10Ilias Sarantopoulos: [C: 03+1] "LGTM!" [integration/config] - 10https://gerrit.wikimedia.org/r/979974 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [15:36:59] Hello team releng, I've a Gitlab-ci job failing repeatedly (not always though) with a cmd being killed. Here is an example: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/jobs/171633 failing with `/opt/miniconda/etc/profile.d/conda.sh: line 5: 3704 Killed` Could you help me plz ? [15:43:23] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10Vgutierrez) A quick check on deployment-parsoid12 tells that ferm rules on that... [16:27:05] 10Continuous-Integration-Infrastructure: Add Python 3.10 and 3.11 to Wikimedia CI - https://phabricator.wikimedia.org/T342019 (10Xqt) [16:30:55] 10GitLab (Infrastructure), 10collaboration-services: GitLab replica banners are missing - https://phabricator.wikimedia.org/T352433 (10LSobanski) a:03eoghan [16:31:15] (03PS1) 10Dreamy Jazz: Zuul: Add TorBlock to CheckUser extension dependency list [integration/config] - 10https://gerrit.wikimedia.org/r/979998 (https://phabricator.wikimedia.org/T352681) [16:32:58] 10GitLab (Infrastructure), 10collaboration-services: GitLab replica banners are missing - https://phabricator.wikimedia.org/T352433 (10LSobanski) p:05Triage→03Medium [16:39:57] my local PHPUnit tests now seem to wipe the real wiki database under some circumstances; does that sound like a known issue to anyone? [16:40:06] cc Daimona and kostajh just in case [16:40:22] (I’ll see if I can figure out what’s wrong and file a phab task later, but maybe it’s already known) [16:41:06] It happened to me in the past with SQLite when using paratest, but not when running PHPUnit normally. [16:41:34] I didn't really figure out why though, and whether it could be related. [16:44:14] it happened to me with MariaDB [16:44:18] (and phpunit:entrypoint) [16:45:06] okay, I think I see what happened [16:45:15] I got “Error 1051: Unknown table 'wiki1.unittest_abuse_filter_action'” [16:45:20] in MediaWikiIntegrationTestCase.php:1749 [16:45:31] which is in setupDatabaseWithTestPrefix(), and *before* $db->tablePrefix() [16:45:46] so presumably all of setupDatabaseWithTestPrefix() crashed, due to that unknown table error (where I don’t know why it happened yet) [16:45:55] but then, seemingly, PHPUnit kept running – now targeting the real DB? [16:46:12] which then later resulted in TestUser throwing “RuntimeException: Can't create user on real database” [16:46:28] but by that point the framework had probably wiped my database table a dozen times :S [16:46:34] *tables [16:51:00] whatever it is, I can quite reliably reproduce it at the moment [17:07:43] filed T352695 to track my test database issue [17:07:44] T352695: MediaWiki PHPUnit test suite can result in non-test database being modified - https://phabricator.wikimedia.org/T352695 [18:08:09] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:08:30] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: [EPIC] Application Security Pipeline Components for Gitlab - Phase 2 Work - https://phabricator.wikimedia.org/T342177 (10sbassett) [18:08:54] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) 05Open→03In progress p:05Triage→03Medium [18:09:00] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:10:46] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10SecTeam-Processed, 10Security, 10user-sbassett: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:17:02] 10Gitlab-Application-Security-Pipeline, 10Security Team AppSec, 10Security-Team: Implement a template for Shell scripting testing - https://phabricator.wikimedia.org/T346163 (10Ashikrumon12341234) [18:18:20] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10CodeReviewBot) sbassett opened https://gitlab.wikimedia.org/rep... [18:18:27] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T350085 (10egardner) ##### Risky Patch! 🚂🔥 Removal of Vue.js Compatibility Build * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/773281 * **... [18:18:42] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:20:23] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:20:41] 10Gitlab-Application-Security-Pipeline, 10Security-Team, 10Patch-For-Review, 10SecTeam-Processed, and 2 others: Add explicit default rules: - when: always directive to ALL AppSec pipeline includes - https://phabricator.wikimedia.org/T352702 (10sbassett) [18:20:55] 10GitLab (Project Migration), 10Traffic, 10Patch-For-Review: Migrate Traffic repositories from Gerrit to Gitlab - https://phabricator.wikimedia.org/T347623 (10BCornwall) [18:25:49] 10Release-Engineering-Team (Priority Backlog 📥), 10Release, 10Train Deployments: 1.42.0-wmf.9 deployment blockers - https://phabricator.wikimedia.org/T350085 (10Catrope) >>! In T350085#9380368, @egardner wrote: > * **Change**: https://gerrit.wikimedia.org/r/c/mediawiki/core/+/773281 > * **Revert plan**: Rev... [18:38:52] 10GitLab (Administration, Settings & Policy), 10collaboration-services: Wikimedia GitLab should use a Wikimedia logo - https://phabricator.wikimedia.org/T349167 (10Aklapper) I don't have any strong opinions on favicons. I think anything that allows differentiation in open browser tabs is pretty fine. [19:50:53] 10GitLab (Administration, Settings & Policy), 10collaboration-services: Wikimedia GitLab should use a Wikimedia logo - https://phabricator.wikimedia.org/T349167 (10Dzahn) We could use the black version from https://about.gitlab.com/press/press-kit/ so we don't even modify it but it's also not the orange one. [20:21:15] 10Release-Engineering-Team, 10Fundraising-Backlog, 10MediaWiki-extensions-FundraiserLandingPage, 10donate.wikimedia.org, and 2 others: donate.wikimedia.org landing pages don't have descriptions in search engine results - https://phabricator.wikimedia.org/T254808 (10XenoRyet) [20:45:40] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Data-Engineering: Unblock Dockerfile syntax to build images with Gitlab trusted runner - https://phabricator.wikimedia.org/T351792 (10CodeReviewBot) aqu updated https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/553 Fi... [20:56:29] 10GitLab (CI & Job Runners), 10Release-Engineering-Team, 10Data-Engineering: Unblock Dockerfile syntax to build images with Gitlab trusted runner - https://phabricator.wikimedia.org/T351792 (10CodeReviewBot) milimetric merged https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/5... [20:57:18] I've got a fix to my previous message: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/merge_requests/553 [20:57:18] It's linked with https://phabricator.wikimedia.org/T351792 [21:00:53] it looked good aqu so I merged it and rebased my MR on it, will let yall know if it's still breaking but thanks for the fix [21:18:27] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10egardner) For Codex, we rely on some tools (such as Vite) which have [[ https://vitejs.dev/guide/migration#node-js-support | dropped support for Node 16 ]] as of t... [21:25:31] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10daniel) Ah right, that's {T350353}. I failed to realize that PRs aren't automa... [21:34:53] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10Jdforrester-WMF) >>! In T331180#9381067, @egardner wrote: > For Codex, we rely on some tools (such as Vite) which have [[ https://vitejs.dev/guide/migration#node-j... [21:52:27] 10Phabricator: Revert to using upstream src/infrastructure/daemon/workers/management/PhabricatorWorkerManagementWorkflow.php - https://phabricator.wikimedia.org/T352721 (10Aklapper) p:05Triage→03Low [22:01:25] 10Phabricator (Upstream), 10Upstream: Phabricator uses different monospace fonts for inline code snippets and block code snippets - https://phabricator.wikimedia.org/T141494 (10matmarex) Filed https://we.phorge.it/T15682 [22:12:14] 10Phabricator: Rename request (ClydeFranklin) - https://phabricator.wikimedia.org/T352723 (10ClydeFranklin) [22:27:02] (03CR) 10Hashar: [C: 03+2] ml-services: renamed directory for revert_risk_model [integration/config] - 10https://gerrit.wikimedia.org/r/979974 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [22:28:46] (03Merged) 10jenkins-bot: ml-services: renamed directory for revert_risk_model [integration/config] - 10https://gerrit.wikimedia.org/r/979974 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [22:31:09] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Radar), 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10daniel) [22:31:26] 10Beta-Cluster-Infrastructure, 10Abstract Wikipedia team, 10Traffic, 10WikiLambda, 10Beta-Cluster-reproducible: HTTP 504 connection timeout error accessing MW API on Beta cluster - https://phabricator.wikimedia.org/T351930 (10daniel) 05Open→03Resolved CI works after rebase. Sorry for the noise. [22:36:00] (03CR) 10Hashar: [C: 03+2] "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/979974 (https://phabricator.wikimedia.org/T352181) (owner: 10AikoChou) [22:44:33] 10Continuous-Integration-Config: Upgrade all CI jobs for WMF-deployed projects from Node 16 to Node 18 - https://phabricator.wikimedia.org/T331180 (10egardner) >>! In T331180#9381135, @Jdforrester-WMF wrote: >>>! In T331180#9381067, @egardner wrote: >> For Codex, we rely on some tools (such as Vite) which have [... [22:57:04] 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: Non-sysop translationadmin can remove protected pages from translation but not able to add it back - https://phabricator.wikimedia.org/T352726 (1094rain) [23:11:21] 10Phabricator: Rename request (ClydeFranklin) - https://phabricator.wikimedia.org/T352723 (10taavi) Spaces can't be used in Phabricator user names, how would you like it to be formatted? `Queen_of_Hearts` with underscores?