[03:08:38] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<33.33%) [06:02:44] (03PS1) 10Pwirth: Remove hhvm from BlueSpiceSocial as it's requirements require php7+ [integration/config] - 10https://gerrit.wikimedia.org/r/528622 [06:04:30] (03PS2) 10Pwirth: layout: [BlueSpiceSocial] Disable HHVM tests, this doesn't work there [integration/config] - 10https://gerrit.wikimedia.org/r/528622 [06:58:39] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:29:02] 10Gerrit, 10Release-Engineering-Team-TODO (201908): Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10Marostegui) 05Resolved→03Open I believe this is still broken: Last commit on github: https://github.com/wikimedia/puppet/commit/97c170306b9145a6cc0735e5166832d973... [07:45:00] (03CR) 10Robert Vogel: [C: 03+1] layout: [BlueSpiceSocial] Disable HHVM tests, this doesn't work there [integration/config] - 10https://gerrit.wikimedia.org/r/528622 (owner: 10Pwirth) [09:21:30] 10Phabricator (Upstream), 10Upstream: Unable to view a Phabricator project - https://phabricator.wikimedia.org/T230001 (10Bugreporter) [09:43:56] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Aklapper) [09:44:22] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Aklapper) 05Open→03Stalled Cannot reproduce with the given link. [09:47:29] 10Phabricator: Autofocus on TOTP input field - https://phabricator.wikimedia.org/T229757 (10MarcoAurelio) Can it be upstreamed? [09:54:53] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Bugreporter) Did you find the issue while logged out? [09:56:05] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Bugreporter) Archive: https://archive.is/iHkXH [10:05:08] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Aklapper) @Bugreporter: Does it require being logged out? [10:07:20] 10Phabricator (Upstream), 10Upstream: Cannot view Phab project: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Aklapper) 05Stalled→03Open Ah, I see now, thanks. I can reproduce this when... [10:08:34] 10Phabricator (Upstream), 10Upstream: Error viewing project feed when many recent changes were in access restricted tasks: "Query (of class "PhabricatorFeedQuery") overheated: examined more than 500 raw rows without finding 50 visible objects" - https://phabricator.wikimedia.org/T230001 (10Aklapper) [10:32:25] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [10:39:13] 10Release-Engineering-Team-TODO (201908): Determine CI PoC evaluation criteria - https://phabricator.wikimedia.org/T230006 (10LarsWirzenius) [10:40:17] 10Release-Engineering-Team-TODO (201908): Read Kubernetes book - https://phabricator.wikimedia.org/T230007 (10LarsWirzenius) [10:45:11] 10Release-Engineering-Team-TODO (201908): Read Kubernetes book - https://phabricator.wikimedia.org/T230007 (10LarsWirzenius) p:05Triage→03Normal [10:45:17] 10Release-Engineering-Team-TODO (201908): Determine CI PoC evaluation criteria - https://phabricator.wikimedia.org/T230006 (10LarsWirzenius) p:05Triage→03Normal [11:04:37] PROBLEM - Gerrit Health Check on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:05:13] PROBLEM - Gerrit JSON on gerrit.wikimedia.org is CRITICAL: CRITICAL - Socket timeout after 10 seconds https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [11:06:43] ^^ huh [11:11:12] Project mediawiki-core-doxygen-docker build #8923: 04FAILURE in 0.66 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/8923/ [11:12:23] RECOVERY - Gerrit Health Check on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 865 bytes in 0.076 second response time https://gerrit.wikimedia.org/r/config/server/healthcheck%7Estatus [11:12:28] Project beta-code-update-eqiad build #258261: 04FAILURE in 9 min 28 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/258261/ [11:13:01] RECOVERY - Gerrit JSON on gerrit.wikimedia.org is OK: HTTP OK: HTTP/1.1 200 OK - 26087 bytes in 0.060 second response time https://wikitech.wikimedia.org/wiki/Gerrit%23Monitoring [11:14:23] Yippee, build fixed! [11:14:24] Project beta-code-update-eqiad build #258262: 09FIXED in 1 min 23 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/258262/ [11:16:21] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [11:24:33] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201908), 10serviceops-radar: Gerrit http threads stuck behind sendemail thread - https://phabricator.wikimedia.org/T224448 (10Marostegui) gerrit went unresponsive today again, and I had to restart it. @Paladox lat... [11:29:04] 10Beta-Cluster-Infrastructure, 10Performance-Team: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10MoritzMuehlenhoff) This is blocking the removal of tungsten, what are the remaining blockers/work to do? [11:34:33] hmm https://gerrit.wikimedia.org/r/#/admin/plugins/ replication is missing [11:39:54] 10Gerrit, 10Release-Engineering-Team-TODO (201908): Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10Paladox) The replication plugin seems to have mysteriously disappeared from gerrit. See https://gerrit.wikimedia.org/r/#/admin/plugins/ (where replication is not showi... [11:48:25] 10Gerrit, 10WMDE-Analytics-Engineering, 10Wikidata: Make "analytics/wmde/toolkit-analyzer-build" use git lfs - https://phabricator.wikimedia.org/T230015 (10Ladsgroup) [11:54:10] (03PS1) 10Ladsgroup: Make "analytics/wmde/toolkit-analyzer-build" use git lfs [All-Projects] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528772 (https://phabricator.wikimedia.org/T230015) [12:12:21] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 57.14% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [12:12:37] Yippee, build fixed! [12:12:37] Project mediawiki-core-doxygen-docker build #8924: 09FIXED in 8 min 33 sec: https://integration.wikimedia.org/ci/job/mediawiki-core-doxygen-docker/8924/ [12:20:24] 10Continuous-Integration-Infrastructure, 10Zuul: Stop/Restart tests for zuul - https://phabricator.wikimedia.org/T230019 (10Matthias_Geisler_WMDE) [12:36:31] 10Beta-Cluster-Infrastructure, 10Performance-Team: Move XHGui from tungsten to webperf-002 - https://phabricator.wikimedia.org/T180761 (10Krinkle) From a high-level, four things: 1. Decide on the multi-dc strategy for XHGui (keep SPOF, active-active with replication, active-active without replication). 2. Ver... [12:43:43] 10Continuous-Integration-Infrastructure, 10Zuul: Stop/Restart tests for zuul - https://phabricator.wikimedia.org/T230019 (10Krinkle) To restart the jobs for a patch set, add the comment `recheck` in Gerrit. This will abort the existing jobs and re-queue the task at the end (yielding to other jobs first). The... [13:01:59] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [13:21:32] 10Release-Engineering-Team-TODO, 10Code-Health-Metrics, 10MediaWiki-Core-Testing, 10Code-Health, and 2 others: Unit tests are not being run for extensions under PHPUnit 4.x (HHVM) - https://phabricator.wikimedia.org/T229220 (10Krinkle) [13:25:40] kostajh: btw, the new MediaWikiUnitTestCase is still doing weird things that don't work with regards to $GLOBALS [13:25:47] should we remove this for the time being? [13:43:22] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201907), 10Operations, and 4 others: Cannot connect to vcs@git-ssh.wikimedia.org (since move from phab1001 to phab1003) - https://phabricator.wikimedia.org/T224677 (10MoritzMuehlenhoff) The update has been acce... [13:49:45] PROBLEM - Free space - all mounts on deployment-mediawiki-07 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki-07.diskspace.root.byte_percentfree (<100.00%) [13:52:55] rip [14:04:33] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 42.86% of data above the critical threshold [140.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:17:54] Krinkle: question regarding a possible future wdio update. do you think (gut feeling would be fair enough if there was no hard decision yet) there will be a time where code will have to be compatible with both (to allow gradual adoption), or will this be done as a "big bang" switchover by a task force mending all tests? [14:24:46] Pablo_WMDE: both what? [14:25:31] oh generally a new and older version of wdio [14:26:00] Pablo_WMDE: gradual. per-repo. Not central all at once. Same as for npm-test and composer-test tool chains. [14:26:07] controlled in the local packagejson [14:30:35] got it, standing on the shoulders of https://phabricator.wikimedia.org/T199116 [14:32:00] clear. the question was about code in wdio-mediawiki. e.g. https://gerrit.wikimedia.org/r/plugins/gitiles/mediawiki/core/+/master/tests/selenium/wdio-mediawiki/index.js#18 [14:33:28] seems to be e.g. browser.config.screenshotPath in wdio5 [14:33:37] Krinkle: ^ [14:34:32] Pablo_WMDE: OK. What is the question:) ? [14:34:49] wdio-mediawiki, like wdio is versioned and publsihed on npm. repo decides which version they use. [14:34:59] only exception is mediawiki/core which will naturally use the latest of wdio-mediawiki as it is developed there. [14:36:01] sure. so will we publish a version of wdio-mediawiki that is compatible to 5 only, or will we support 2 versions alongside, for how long? [14:36:45] Pablo_WMDE: semver-major, no compat needed. any repo using 4.x will be using the prev version of the other packages as well [14:36:59] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 30.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:37:53] cool. and is there a plan for when the change to wdio-mediawiki will happen? [14:39:06] Pablo_WMDE: Blocked on feature tests in core being upgraded. I maintain wdio-mediawiki and can help with that, but not with the tests themselves. [14:39:23] you could create a task and list the tests we have that need upgrading and their steward so we can coordinate it. [14:39:36] probably not a high prio though, is there something from 5.x we need? [14:40:06] if you want to upgrade other repos earlier and have priority/resourcing to do so, I'd also accept wdio-mw patches to maintain multi-version compat. [14:40:22] "need" would be a stretch. started a new project and don't want to write new tests that will soon have to be changed again. [14:41:29] these tests run 5 and explode using stuff from wdio-mediawiki which is incompatible with it [14:42:16] current solution is to have our own copy of said methods but i thought we might as well upstream and serve everyone [15:02:34] will do, time allowing. thanks, Krinkle [15:03:15] Pablo_WMDE: if it's only the util method, you may be able to copy that locally for now. E.g. if the other classes work. [15:03:26] it's a pretty simple library. I think one or two repos run their tests without it even [15:03:36] I'll support whichever way you choose [15:06:40] "it's a pretty simple library" indeed so it is even more questionable to have ever so slightly different copies of it. thanks [15:21:59] the current length of *gate-and-submit* is 23 changes 😭 [15:22:03] not to mention test [15:25:13] Lucas_WMDE: make the wikibase tests faster :) [15:25:52] making them less flaky would already help, so they don’t have to go through gate-and-submit three times [15:25:58] we’re working on that [15:26:15] but a lot of these changes are not Wikibase [15:27:08] lots of core and THICC changes, some skins, third-party extensions [15:27:19] there was a big chain of wikibase commits that backed things up for ~2 hours [15:27:29] <3 [15:27:40] but yes we would all like faster CI [15:28:57] running all the tests against HHVM+php7.0+php7.2+php7.3 is not helping that at the moment :/ [15:39:49] 10Continuous-Integration-Infrastructure, 10MediaWiki-Installer, 10Core Platform Team (Needs Cleaning - Security, stability, performance and scalability (TEC1)), 10Core Platform Team Workboards (Clinic Duty Team), and 4 others: install.php --with-extensions silently... - https://phabricator.wikimedia.org/T225512 [15:55:17] Hey folks. Are we still doing service deploys during Wikimania or are they on hold too? [15:55:27] (nothing pressing, just making sure I know what to expect) [16:00:46] !log restarting jenkins for updates [16:00:48] halfak: https://wikitech.wikimedia.org/wiki/Deployments suggests only the train is stopping [16:00:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:01:02] We're still SWAT-ing and there's services windows [16:01:39] Gotcha. Thanks Reedy. Somehow I didn't think to check there. I just heard about the train stopping in SOS [16:04:17] (03PS1) 10MJL: Edit Project Config [extensions/SoftRedirector] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/528884 [16:09:41] Project beta-scap-eqiad build #261377: 15ABORTED in 5 min 13 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/261377/ [16:09:54] ^ me, got going mid-upgrade [16:14:32] oh good. I broke it :\ [16:17:07] Project beta-scap-eqiad build #261378: 15ABORTED in 6 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/261378/ [16:59:57] 10Release-Engineering-Team (Deployment services), 10Security-Team, 10Wikimedia-Site-requests, 10Wikimedia-extension-review-queue: Deploy WebAuthn to Wikimedia Wikis - https://phabricator.wikimedia.org/T227242 (10Reedy) [17:07:10] (03PS1) 10Thcipriani: --no-php-restart missing comma [tools/scap] - 10https://gerrit.wikimedia.org/r/528895 [17:14:25] (03CR) 10Thcipriani: [C: 03+2] --no-php-restart missing comma [tools/scap] - 10https://gerrit.wikimedia.org/r/528895 (owner: 10Thcipriani) [17:17:16] (03Merged) 10jenkins-bot: --no-php-restart missing comma [tools/scap] - 10https://gerrit.wikimedia.org/r/528895 (owner: 10Thcipriani) [17:20:04] (03CR) 10jenkins-bot: --no-php-restart missing comma [tools/scap] - 10https://gerrit.wikimedia.org/r/528895 (owner: 10Thcipriani) [17:57:15] Project beta-scap-eqiad build #261389: 15ABORTED in 15 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/261389/ [18:07:58] (03PS1) 10Thcipriani: PHPRestart: fix attribute error for cmd/job [tools/scap] - 10https://gerrit.wikimedia.org/r/528904 [18:12:00] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [18:26:56] (03CR) 10Thcipriani: [C: 03+2] PHPRestart: fix attribute error for cmd/job [tools/scap] - 10https://gerrit.wikimedia.org/r/528904 (owner: 10Thcipriani) [18:29:03] (03Merged) 10jenkins-bot: PHPRestart: fix attribute error for cmd/job [tools/scap] - 10https://gerrit.wikimedia.org/r/528904 (owner: 10Thcipriani) [18:29:50] (03CR) 10jenkins-bot: PHPRestart: fix attribute error for cmd/job [tools/scap] - 10https://gerrit.wikimedia.org/r/528904 (owner: 10Thcipriani) [18:42:57] Project beta-scap-eqiad build #261394: 15ABORTED in 5 min 12 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/261394/ [19:04:34] 10Continuous-Integration-Config: Make mediawiki-config use swat gate and submit - https://phabricator.wikimedia.org/T230060 (10Reedy) [19:04:36] 10Continuous-Integration-Config: Make mediawiki-config use swat gate and submit - https://phabricator.wikimedia.org/T230060 (10Reedy) p:05Triage→03High [19:06:28] Reedy: <3 :) [19:09:52] https://github.com/wikimedia/integration-config/blob/f7a7c0c9e7135201ed4625c56bac3f96b3358e7a/zuul/layout.yaml#L541-L548 [19:10:06] I'm guessing there's a "repo" or "repository" thing that can be used there as an option [19:12:39] or queue? [19:13:51] 10Continuous-Integration-Config, 10Release-Engineering-Team: Make mediawiki-config use swat gate and submit - https://phabricator.wikimedia.org/T230060 (10Reedy) Basically, mediawiki-config shouldn't be sitting behind numerous other patches (such as these MF ones ) while the swat queue is empty Chances are th... [19:24:03] Reedy: maybe, it's tricky with regards to AND/OR [19:24:31] e.g. we want config & any branch (or branch master at least) and mw & branch wmf [19:30:53] 10Continuous-Integration-Config, 10Release-Engineering-Team: Make mediawiki-config use swat gate and submit - https://phabricator.wikimedia.org/T230060 (10thcipriani) [19:30:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: mediawiki-config (and others?) should ride gate-and-submit-swat not gate-and-submit - https://phabricator.wikimedia.org/T225252 (10thcipriani) [20:29:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: mediawiki-config (and others?) should ride gate-and-submit-swat not gate-and-submit - https://phabricator.wikimedia.org/T225252 (10Krinkle) >>! In T225252#5241899, @thcipriani wrote:... [20:29:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: mediawiki-config (and others?) should ride gate-and-submit-swat not gate-and-submit - https://phabricator.wikimedia.org/T225252 (10Krinkle) p:05Triage→03Low [20:30:44] James_F|Away: the remaining stylelint-config-wikimedia 0.6.0 upgrade will have to be done manually, libup is getting really confused that it's being fixed via npm audit as well (I tried hacking it up last night but it didn't go well at all) [20:44:09] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10Paladox) I think possibly a few mins delay. (It runs on one thread). [20:47:10] (03PS1) 10Thcipriani: PHPRestart: Fix logging, groups, return types [tools/scap] - 10https://gerrit.wikimedia.org/r/528928 [20:47:12] (03PS1) 10Thcipriani: PHPRestart: Create global INSTANCE for pickling [tools/scap] - 10https://gerrit.wikimedia.org/r/528929 [20:58:57] (03CR) 10Thcipriani: [C: 03+2] PHPRestart: Fix logging, groups, return types [tools/scap] - 10https://gerrit.wikimedia.org/r/528928 (owner: 10Thcipriani) [21:01:00] (03Merged) 10jenkins-bot: PHPRestart: Fix logging, groups, return types [tools/scap] - 10https://gerrit.wikimedia.org/r/528928 (owner: 10Thcipriani) [21:01:45] (03CR) 10jenkins-bot: PHPRestart: Fix logging, groups, return types [tools/scap] - 10https://gerrit.wikimedia.org/r/528928 (owner: 10Thcipriani) [21:03:36] (03PS1) 10Ladsgroup: Set cache directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/528933 (https://phabricator.wikimedia.org/T225730) [21:14:58] (03PS1) 10Umherirrender: [LiquidThreads] Add phan dependency [integration/config] - 10https://gerrit.wikimedia.org/r/528937 (https://phabricator.wikimedia.org/T224757) [21:21:12] (03PS1) 10Umherirrender: [LiquidThreads] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/528941 [21:22:13] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10DannyS712) [21:25:22] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T220744 (10mobrovac) [21:26:26] paladox: "fun fact". gerrit-replica puppet run fails if gerrit is down [21:26:30] guess why [21:26:40] Exec[git_pull_All-Avatars] :) [21:26:54] heh [21:26:55] :D [21:27:03] so, i restarted gerrit prod [21:27:03] ssh -p 29418 paladox@gerrit.wikimedia.org gerrit show-queue -w [21:27:08] shows github and gerrit2001 [21:27:09] applied https://gerrit.wikimedia.org/r/c/operations/puppet/+/528769 [21:27:24] https://github.com/wikimedia is updating too! [21:28:03] nice :) [21:29:43] yup! [21:34:36] paladox: also, it's not slow anymore :p [21:34:42] heh [21:34:44] great, github is again a mirror [21:35:50] where is "extdist" Extension Distributor installed .. oh.. mediawiki.org? [21:36:08] but requires ::profile::labs::lvm::srv [21:36:28] i got 2.25 MiB/s for gerrit.w.org [21:36:29] oh.. the part in "labs" sets up a generator for the extdist in prod .. i guess [21:36:39] mediawiki.org [21:36:51] 3 # This class sets up a tarball generator for the Extension Distributor [21:36:54] 4 # extension enabled on mediawiki.org. [21:37:02] so the generator is in labs [21:37:15] and then what it generates is uploaded ? [21:37:58] hehe, there is something in shinken about it [21:38:17] it's accessed at https://extdist.wmflabs.org/dist/extensions/VisualEditor-REL1_33-8c9c37e.tar.gz for example [21:40:40] thcipriani traffic is higher on https://gerrit-replica.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate :) [21:40:50] i see it lower for cobalt https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=httpHitsRate [21:41:45] wowza, there are some spikes on that replica graph for sure. [21:42:23] yup [22:08:38] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Create mirror of Gerrit repositories for consumption by various tools - https://phabricator.wikimedia.org/T226240 (10Dzahn) replication should work again now. there was a syntax issue in the config that has been fixed. [22:08:39] mutante: I added you to a patch I made today re maintenance scripts going to PHP 7 [22:08:53] still WIP though [22:09:14] Asking whether we should briefly enable back logging after migration to see if the script works after that [22:09:21] * hauskatze hides back [22:10:37] hauskatze: i made the table on https://phabricator.wikimedia.org/T195392 with a "logging ? " column that shows how some log and some don't and locations are a bit mixed still, though not bad as it once was [22:11:04] purge_checkuser.pp sends logs to /dev/null currently [22:11:31] I was wondering if we should turn them back for a week or so after migrating the script to see if there are any issues, then disable them again [22:12:01] yea, or just keep logging to /var/log/mediawiki/ like others do [22:12:11] and don't turn it off [22:12:19] as long as we have proper logrotate setup [22:13:52] That, I'm not sure. I'm just making the PHP7.2 migration there. [22:14:14] 10Gerrit, 10Release-Engineering-Team-TODO (201908), 10Patch-For-Review: Gerrit -> GitHub replication not up-to-date - https://phabricator.wikimedia.org/T229945 (10thcipriani) 05Open→03Resolved >>! In T229945#5398555, @Marostegui wrote: > I believe this is still broken: > > Last commit on github: https:... [22:20:02] hauskatze: let's do that one right now? [22:21:49] mutante: I'd prefer to do it tomorrow if you don't mind. I'm heading to bed. [22:22:24] Plus I think I'll send logs back to var/mediawiki [22:22:30] so I'll amend the patch tomorrow [22:22:37] hauskatze: ok!:) [22:22:40] good night then [22:22:54] /var/log/mediawiki I mean [22:23:02] hopefully you'll have logrotate there [22:23:17] I have no idea since I don't have mwmaint access [22:23:39] i am checking that [22:23:59] i remember kind of how i moved stuff myself [22:24:07] because they were all over the place before we used /var/log/mediawiki [22:24:55] at least one file does: [22:24:56] /var/log/mediawiki/mediawiki_job_mediawiki_tor_exit_node/*.log [22:25:49] (03CR) 1020after4: [C: 03+2] Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) (owner: 10markahershberger) [22:26:46] (03Merged) 10jenkins-bot: Tarball creation [tools/release] - 10https://gerrit.wikimedia.org/r/521559 (https://phabricator.wikimedia.org/T217960) (owner: 10markahershberger) [22:37:13] mutante: I'll amend the patch and link it to you tomorrow so you can see it, and I can learn [22:37:16] if that's okay [22:38:34] hauskatze: i think i want to amend it too to change more stuff [22:38:50] like moving the whole thing to use profile::mediawiki::periodic_job [22:39:04] ugh, that isn't totally supported yet iirc [22:39:07] just started doing that after i saw that is how you get logrotate [22:39:13] no script is using that yet right? [22:39:33] one of them does, tor_exit_nodes [22:39:49] that's also why that one has logrotate config and the others dont [22:40:05] it's a free side effect of using the new way [22:41:18] hauskatze: but you know .. i am also fine with the most simple version. i run it manually once and if it works we're good and i merge. the end [22:41:18] it just runs once a week anyways [22:41:46] I'm fine with that too mutante [22:41:58] we can migrate it further later [22:42:04] let me do that and then we can have separate changes for logging stuff [22:42:09] ok, deal [22:42:10] probably a person who understand puppet better than me [22:42:27] (that is: anyone else lol) [22:42:35] nah, come on :) [22:42:47] there is "using puppet" and "using puppet" [22:42:57] srly, I find puppet extremely complicated [22:43:15] somebody wrote the new abstraction but then somebody else just copy/pasted it over and over again [22:43:40] ok, f* sleep, I'm loggin to gerrit [22:43:49] lol [22:43:59] i am just merging that now :p [22:44:12] after i run it manually exactly like the cron does [22:44:19] that is as the same user [22:44:23] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (201908), 10Release, 10Train Deployments: 1.34.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T220741 (10brennen) 05Open→03Resolved [22:44:39] But https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528730/ does send the logs to /dev/null [22:44:48] just in case [22:45:13] if running manually make sure you let the script generate the logs to see if it works as expected [22:45:30] yes, it does [22:45:33] (that was an idiotic remark - I know you will) [22:45:43] i would do that if this was running every 5 min [22:45:48] but it's just once per week [22:46:07] so, to sum up, is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/528730/ okay as it is? [22:46:17] yes it is [22:46:30] :D [22:46:48] running it [22:47:00] logging that too [22:49:11] hauskatze: i am watching it go through all wikis and purge a few rows from recentchanges in each [22:49:18] no errors or anything [22:49:42] great [22:49:49] that script is very important [22:49:56] so it's good to know it runs w/o issues [22:50:16] it could also run every day [22:50:20] and not just every week [22:50:25] would that matter? [22:50:47] seems to me it would just be faster because not as much builds up [22:51:00] we are still at "i" in the alphabet [22:51:01] It just purges the CU data older than 3 months so if there's nothing to delete, or few things to delete it might go faster [22:51:09] yea, ack [22:51:20] but it'd be also a waste or resources maybe [22:51:31] I'm not sure who has to make such call [22:52:04] i dont think it matters either way for resources of mwmaint [22:52:05] https://gerrit.wikimedia.org/r/plugins/gitiles/operations/puppet/+/production/modules/mediawiki/manifests/maintenance/purge_abusefilter.pp runs daily I think [22:52:11] same purpose [22:52:19] i see [22:52:19] perhaps we should make it run daily [22:52:29] now it stopped for a moment at login.wiki [22:52:33] and continued [22:52:41] there was more to do there [22:52:49] loginwiki has tons of records as all accounts get registered there [22:53:20] yep [22:53:29] I think we can make it run daily after the migration patch [22:53:49] and after that, we can think about using the periodic_job stuff [22:53:54] yea, let's try it. and then let's write to a log indeed [22:54:20] ok, all in separate changes is fine with me. usually better that way [22:57:29] zh_min_nab... [22:57:40] zuwiktionary.. and done [22:59:46] Thanks for merging :) [22:59:49] and testing [22:59:59] actually, in reverse: testing and merging [23:00:32] heh, thanks for the patch, i wanted to get another one done today and always distractions before [23:01:47] hauskatze: so the other one you mentioned .. that is for the same thing.. which one was it [23:01:57] ah, purge_abusefilter [23:02:13] yep, that one runs daily [23:02:26] I guess we can make purge_checkuser run daily as well [23:02:43] oh, you know what. it's possible it still used the old PHP version without us noticing [23:03:04] xD [23:03:05] it uses foreachwiki [23:03:16] true [23:03:17] that separates it from those just using mwscript [23:03:27] and some of those scripts hardcoded the "RUNNER" as "php" [23:03:36] that's why i put all those comments in that table [23:03:49] and wanted to check them by group of.. which helper script they use [23:03:57] So after all we made nothing [23:06:56] * paladox wonders if this https://gerrit-review.googlesource.com/c/gerrit/+/233493 will work for us [23:06:57] * paladox tests [23:07:21] hauskatze: not sure, it's a wrapper around another wrapper [23:07:41] oops, yea. there it is again [23:07:43] RUNNER=php [23:08:17] Only initsitestats has the RUNNER=php comment on that table but yup, it looks foreachwiki(indblist) uses that [23:08:24] maybe mwscript as well [23:09:03] mwscript is fine [23:09:06] that's why i did those first [23:09:20] those that just use straight mwscript work like you did it [23:09:27] but those that use foreachwikiindblist won't [23:09:29] foreachwikiindblist looks migrated? https://github.com/wikimedia/puppet/blob/production/modules/scap/files/foreachwikiindblist ? [23:10:02] looking for 'foreachwiki' only [23:10:57] can't find 'foreachwiki' only [23:10:58] that has the =php line still in it [23:11:27] 4.0K -r-xr-xr-x 1 root root 421 Sep 21 2018 foreachwiki [23:11:27] 4.0K -r-xr-xr-x 1 root root 713 Sep 21 2018 foreachwikiindblist [23:11:46] grep RUNNER foreachwiki* [23:11:46] foreachwikiindblist:RUNNER=php [23:12:04] https://github.com/wikimedia/puppet/blob/production/modules/scap/files/foreachwiki [23:12:07] this belonbgs in the other channel btw :p [23:14:49] Next task: make foreachwiki(indblist) not use RUNNER=php [23:15:22] yea [23:15:41] * hauskatze waves good night [23:16:17] good night, Katze [23:18:47] Vielen Dank Herr Zahn [23:29:28] 10Release-Engineering-Team, 10Operations, 10cloud-services-team (Kanban): Requesting access to Puppet for Viztor[S] - https://phabricator.wikimedia.org/T229894 (10colewhite) [23:29:37] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<44.44%) [23:51:45] I'm going to deploy subbu's scandium hack now if there are no objections: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/528591/4