[00:00:40] 10Gerrit, 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10thcipriani) >>! In T243808#5835393, @Dzahn wrote: > @thcipriani ^ This is back to 94% as of right now after ^. And it's been downtime for a month. Is the test instance usable with the current size? > > Als... [00:07:46] (03CR) 10Legoktm: [C: 03+2] Install requirements.txt for pytest if it exists (031 comment) [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569838 (owner: 10Legoktm) [00:10:57] (03Merged) 10jenkins-bot: Install requirements.txt for pytest if it exists [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569838 (owner: 10Legoktm) [00:25:49] 10Continuous-Integration-Config, 10WMCZ-Tracker: Make a tox+gettext Docker image for Wikimedia Tracker - https://phabricator.wikimedia.org/T241685 (10Legoktm) Is everything you need in the `gettext` package? https://packages.debian.org/stretch/gettext If so we can probably just include it in the tox docker im... [00:25:55] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall) [00:26:58] 10Release-Engineering-Team (CI & Testing services), 10Citoid, 10Core Platform Team, 10Release Pipeline: Figure out how to test Citoid with Zotero in the pipeline - https://phabricator.wikimedia.org/T225236 (10dduvall) >>! In T225236#5845104, @thcipriani wrote: > We're expanding the documentation for this c... [00:30:23] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall) @Jrbranaa, would love to talk with you and anyone else in #quality-and-test-engineering-team_qte about t... [00:32:29] (03CR) 10Legoktm: [C: 03+2] "Filed T244314 as follow-up." (032 comments) [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 (owner: 10Legoktm) [00:32:40] (03CR) 10Legoktm: [C: 03+2] Run mypy against this repository [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 (owner: 10Legoktm) [00:32:45] (03CR) 10Legoktm: [C: 03+2] Fix variable name collision with "config" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 (owner: 10Legoktm) [00:32:54] (03CR) 10Legoktm: [C: 03+2] Add descriptions of tools for `tox -av` [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 (owner: 10Legoktm) [00:33:39] (03PS1) 10Legoktm: Document all the settings in TOOLS [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/570171 [00:34:34] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall) [00:36:19] (03CR) 10Volans: [C: 03+1] "LGTM, thanks!" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/570171 (owner: 10Legoktm) [00:36:38] (03Merged) 10jenkins-bot: Add mypy support [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569839 (owner: 10Legoktm) [00:36:41] (03Merged) 10jenkins-bot: Officially support Python 3.8 [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569840 (owner: 10Legoktm) [00:37:37] (03Merged) 10jenkins-bot: Run mypy against this repository [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569841 (owner: 10Legoktm) [00:37:39] (03Merged) 10jenkins-bot: Fix variable name collision with "config" [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569845 (owner: 10Legoktm) [00:37:47] (03Merged) 10jenkins-bot: Add descriptions of tools for `tox -av` [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/569846 (owner: 10Legoktm) [00:39:19] (03CR) 10Legoktm: [C: 03+2] Document all the settings in TOOLS [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/570171 (owner: 10Legoktm) [00:44:00] (03Merged) 10jenkins-bot: Document all the settings in TOOLS [integration/tox-wikimedia] - 10https://gerrit.wikimedia.org/r/570171 (owner: 10Legoktm) [01:18:04] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10Jrbranaa) >>! In T244313#5850608, @dduvall wrote: > @Jrbranaa, would love to talk with you and anyone else in #qu... [01:21:25] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO, 10Code-Stewardship-Reviews: deployment-prep: Code stewardship request - https://phabricator.wikimedia.org/T215217 (10Jrbranaa) >>! In T215217#5850281, @bd808 wrote: > Happy bug birthday {T215217}! Nor... [02:53:45] 10Release-Engineering-Team-TODO, 10DiscussionTools, 10Editing-team, 10Wikimedia-Site-requests, 10OWC2020 (OWC2020 Replying 1.0): Deploy DiscussionTools to Wikimedia production - https://phabricator.wikimedia.org/T240470 (10JTannerWMF) [05:10:19] 10Project-Admins, 10International-Developer-Events: Create Wikimedia Hackathon 2020 Projects - https://phabricator.wikimedia.org/T244328 (10Rfarrand) [05:34:05] 10Continuous-Integration-Config, 10WMCZ-Tracker: Make a tox+gettext Docker image for Wikimedia Tracker - https://phabricator.wikimedia.org/T241685 (10Urbanecm) Yes @Legoktm. [05:49:43] 10Beta-Cluster-Infrastructure, 10Browser-Support-Firefox: en.wikipedia.beta.wmflabs.org not accessible in Firefox, due to something with OCSP - https://phabricator.wikimedia.org/T243881 (10Kaartic) Just wondering, how long would the temporary solution work? I think that's the same as the answer to the question... [06:25:18] 10Phabricator, 10Wikimedia-production-error: Phatality search not working - https://phabricator.wikimedia.org/T244330 (10Tgr) [06:39:30] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10CPT Initiatives (API Integration Tests), 10Code-Health, and 2 others: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10DannyS712) [07:38:37] (03PS1) 10Jeena Huneidi: Remove restbase chart and add restrouter chart [releng/local-charts] - 10https://gerrit.wikimedia.org/r/570236 [07:58:23] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10RESTBase, 10local-charts, and 2 others: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910 (10jeena) Restrouter is no longer in k8s in producti... [08:00:57] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Update parsoid image and helm charts for php - https://phabricator.wikimedia.org/T242259 (10jeena) 05Open→03Resolved Closing due to parsoid soon becoming a library pulled into core. No parsoid image or helm c... [08:02:08] (03PS2) 10Jeena Huneidi: Remove restbase chart and add restrouter chart [releng/local-charts] - 10https://gerrit.wikimedia.org/r/570236 (https://phabricator.wikimedia.org/T228910) [08:05:17] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10local-charts, 10Developer Productivity, 10Patch-For-Review: Update local-charts repository to use restbase chart from deployment-charts repo - https://phabricator.wikimedia.org/T228915 (10jeena) 05Open→... [08:05:19] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10local-charts, 10Developer Productivity, 10Patch-For-Review: Move local-charts helm charts to a chart repository - https://phabricator.wikimedia.org/T224935 (10jeena) [09:35:11] 10MediaWiki-Codesniffer: Set tab-width in the base ruleset file - https://phabricator.wikimedia.org/T243598 (10aaron) I think https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/563289/ is running into this. [10:50:38] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Lucas_Werkmeister_WMDE) I’d say that’s up to the train… manager? person? Anyways, up to you :) The worst-case scenario is that this is a real Media... [11:22:43] 10Phabricator (Upstream), 10Upstream: Make task ids easily copyable in workboard - https://phabricator.wikimedia.org/T190793 (10Sebastian_Berlin-WMSE) 05Open→03Resolved a:03Sebastian_Berlin-WMSE This was (apparently) fixed at some point. Huzzah! [11:29:54] 10MediaWiki-Codesniffer: Set tab-width in the base ruleset file - https://phabricator.wikimedia.org/T243598 (10Daimona) >>! In T243598#5851512, @aaron wrote: > I think https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/563289/ is running into this. Yes, it does. Feel free to disable the rule globally (in the p... [12:34:00] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Ankit-Maity) @Aklapper @Nemo_bis any timeline on when this will be fixed? [12:35:36] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10MBH) If "Problem likely started when FR config was moved to an extension function in summer 2019", why not just... [12:57:48] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Aklapper) >>! In T234743#5851888, @Ankit-Maity wrote: > @Aklapper @Nemo_bis any timeline on when this will be f... [13:01:33] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Ankit-Maity) >>! In T234743#5851935, @Aklapper wrote: >>>! In T234743#5851888, @Ankit-Maity wrote: >> @Aklapper... [13:17:03] 10Phabricator, 10Security-Team, 10Security: Convert #Security to acl*Security_Task_Access - https://phabricator.wikimedia.org/T244165 (10revi) Base, HakanIST, Matanya, Wim_b are stew. [13:21:40] 10Project-Admins, 10International-Developer-Events: Create Wikimedia Hackathon 2020 Projects - https://phabricator.wikimedia.org/T244328 (10Aklapper) 05Open→03Resolved (@Rfarrand: There is no need to assign such tasks to me and make me cookie-lick. Several other people could also resolve this task.) Alrig... [14:00:24] 10Continuous-Integration-Infrastructure: Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 (10Lucas_Werkmeister_WMDE) Now on integration-agent-docker-1006. https://integration.wikimedia.org/ci/job/quibble-vendor-selenium-docker/3677/c... [14:00:55] integration-agent-docker-1006 might be running out of disk space [14:01:12] or not, T243223 has been happening and then resolving itself again a few times now, who knows [14:01:18] T243223: Occasional Selenium build failures related to temporary directories / no space left - https://phabricator.wikimedia.org/T243223 [14:06:14] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:14:38] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [14:43:53] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10zeljkofilipin) p:05Triage→03Unbreak! Train blockers are UBN. Feel free to lower priority if this is not a train blocker, but in that case, remov... [15:17:12] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [15:18:38] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Clean old wmf branches in Gerrit - https://phabricator.wikimedia.org/T244368 (10thcipriani) [15:19:13] beta-scap-eqiad is stuck again https://integration.wikimedia.org/ci/view/Beta/job/beta-scap-eqiad/ [15:19:32] “Waiting for next available executor on deployment-deploy01: … doesn’t have label ‘BetaClusterBastion’” [15:19:42] who knows the magic incantation to fix it? [15:19:51] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Clean old wmf branches in Gerrit - https://phabricator.wikimedia.org/T244368 (10thcipriani) p:05Triage→03Normal I'd like to pair with someone else on #together who is unfamiliar with `s... [15:20:18] Lucas_WMDE: looking [15:20:25] thanks [15:20:53] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins#Hung_beta_code/db_update are the instructions apparently (found via https://phabricator.wikimedia.org/T239145) [15:22:38] oh, I should add my much simpler alternative method :) [15:22:45] Project beta-scap-eqiad build #286816: 04FAILURE in 0.64 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286816/ [15:23:11] oh, I like simpler methods :) [15:23:17] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10zeljkofilipin) [15:23:40] well neat. It builds but fails :) [15:24:02] looks like `$wgLogos` is missing a 1x entry [15:24:29] but the failing code was introduced a month ago already [15:24:31] Project beta-scap-eqiad build #286817: 04STILL FAILING in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286817/ [15:24:55] ah, but config has more recent changes that look related [15:27:33] hrm, yeah, I see a couple of changes about that from this morning [15:28:27] might be time for a Phabricator task? [15:28:29] ^ Jdlrobson seeing "Notice: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186" in beta -- anything related to wgLogo changes in mwconfig today? [15:29:54] Lucas_WMDE: indeed. [15:29:56] * thcipriani files [15:34:26] Project beta-scap-eqiad build #286818: 04STILL FAILING in 0.86 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286818/ [15:36:09] 10Beta-Cluster-Infrastructure, 10MediaWiki-Configuration: Beta: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 - https://phabricator.wikimedia.org/T244370 (10thcipriani) [15:36:41] ^ Lucas_WMDE task filed [15:36:48] thanks [15:38:11] thanks for the heads up about beta-scap-eqiad. [15:38:15] we should alert for that. [15:41:28] 10Beta-Cluster-Infrastructure, 10MediaWiki-Configuration: Beta: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 - https://phabricator.wikimedia.org/T244370 (10Jdforrester-WMF) This'll have been caused by 8cd2e13. Will dig. [15:44:35] Project beta-scap-eqiad build #286819: 04STILL FAILING in 0.8 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286819/ [15:44:54] thcipriani: https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/570365 [15:47:24] James_F: +1'd -- untested, but seems sane to me. [15:54:32] Project beta-scap-eqiad build #286820: 04STILL FAILING in 3.2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286820/ [15:59:03] Oh, good, and CI is resource-starved. [15:59:05] * James_F digs more. [16:04:35] Project beta-scap-eqiad build #286821: 04STILL FAILING in 0.71 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286821/ [16:13:58] James_F: that's...odd. [16:14:07] considering most of the boxen in jenkins are idle [16:14:31] Project beta-scap-eqiad build #286822: 04STILL FAILING in 0.66 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286822/ [16:14:39] Zuul exploded due to too big a stack in puppet and in Wikibase, by the looks of things. [16:15:33] I see zuul-merger working overtime that's for sure [16:16:33] the timing is...not great :) [16:16:58] * urandom says as he awaits a verified [16:18:11] Last time this happened, hashar and I discussed taking away CI rights from WMDE. :-( [16:19:39] also seems to be a hell of a long line of changes for mediawiki-config [16:19:46] Yeah, same issue. [16:20:22] what was the punch line the last time? did you just have to kick zuul? [16:20:39] Yeah. [16:20:52] afaict, zuul-merger is just flailing trying to create the state of the world [16:20:58] we have a deployment window currently blocking on this fwiw [16:21:50] * apergos will not mindlessly submit rechecks, even if tempted [16:23:16] so it seems like zuul-merger has now caught up afaict [16:23:31] which means jobs should start moving, but now we have a massive backlog of jobs [16:24:31] Project beta-scap-eqiad build #286823: 04STILL FAILING in 0.72 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286823/ [16:24:32] omg (just looked at the dashboard) [16:25:07] urandom: which repo are you waiting on? [16:25:19] mediawiki-config [16:25:53] did you +2 your patchset? [16:26:22] maybe just merged? [16:28:00] James_F: let's wait for gate-and-sumit to clear, test-prio as well, make a note of patches to recheck, then kick zuul: sound reasonable? [16:28:33] otherwise we'll be dealing with backlog for the next 5 hours [16:30:56] Yeah. [16:31:36] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Test resvg on beta-cluster - https://phabricator.wikimedia.org/T243893 (10daniel) Ping @fgiunchedi. Does this sound doable? [16:34:31] Project beta-scap-eqiad build #286824: 04STILL FAILING in 2.9 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286824/ [16:34:41] * James_F complained on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Wikibase/+/569332/2 [16:37:43] ok, well, once this core patch merges I'm kicking it [16:39:05] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Test resvg on beta-cluster - https://phabricator.wikimedia.org/T243893 (10daniel) @JoKalliauer you can probably get root access on beta to do this yourself. @greg, is this right? [16:40:47] !log restarting zuul [16:40:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:42:04] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [16:42:51] aww thanks for the recheck :-) [16:44:33] Project beta-scap-eqiad build #286825: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286825/ [16:45:12] * thcipriani doffs cap [16:45:28] (and it's already done ;-) [16:46:51] thcipriani https://gerrit.wikimedia.org/r/#/c/integration/config/+/566915/ wondering if you can merge that please? :) [16:47:16] paladox|UKInEU: sure thing [16:47:22] thanks! [16:47:30] slipped off my radar with all hands :( [16:47:36] thcipriani also 2.16.16 was released so i can get rid of my backport :) [16:48:10] (03PS6) 10Thcipriani: Gerrit: Also install python3 (bazel requires it) [integration/config] - 10https://gerrit.wikimedia.org/r/566915 (owner: 10Paladox) [16:48:19] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Test resvg on beta-cluster - https://phabricator.wikimedia.org/T243893 (10Reedy) >>! In T243893#5852761, @daniel wrote: > @JoKalliauer you can probably get root access on beta to do this yourself. @greg, is t... [16:49:33] 10Release-Engineering-Team, 10MediaWiki-User-management, 10User-DannyS712: User rights validation is sometimes malfunctioning (with FlaggedRevs only?) - https://phabricator.wikimedia.org/T234743 (10Aklapper) >>! In T234743#5851941, @Ankit-Maity wrote: > from what I can see there is no timeline mentioned anyw... [16:50:13] (03CR) 10Thcipriani: [C: 03+2] Gerrit: Also install python3 (bazel requires it) [integration/config] - 10https://gerrit.wikimedia.org/r/566915 (owner: 10Paladox) [16:51:14] (03Merged) 10jenkins-bot: Gerrit: Also install python3 (bazel requires it) [integration/config] - 10https://gerrit.wikimedia.org/r/566915 (owner: 10Paladox) [16:54:30] Project beta-scap-eqiad build #286826: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286826/ [16:56:24] !log Updating docker-pkg files on contint100 to deploy https://gerrit.wikimedia.org/r/566915 [16:56:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:57:09] \o/ [16:57:23] I'll need to deploy the job once the image is built still [16:57:48] ok :) [16:58:47] > INFO:jenkins_jobs.builder:Reconfiguring jenkins job gerrit-docker [16:58:52] paladox|UKInEU: ^ should be good to go [16:58:57] thanks! [16:59:01] * paladox|UKInEU rechecks [16:59:02] thank you :) [16:59:12] (03CR) 10Paladox: "recheck" [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/566890 (owner: 10Paladox) [17:02:15] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Multimedia, and 2 others: Test resvg on beta-cluster - https://phabricator.wikimedia.org/T243893 (10RazrFalcon) >resvg only seems to be packaged in sid though, which is a bigger problem... Yes, this is a problem with Rust projects in... [17:03:16] thcipriani \o/ [17:04:32] Project beta-scap-eqiad build #286827: 04STILL FAILING in 0.69 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286827/ [17:05:31] (03PS3) 10Paladox: Merge branch 'stable-2.16' into wmf/stable-2.16 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/562363 [17:07:15] thcipriani: Aren't regex fixes fun? https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570379/1/wmf-config/InitialiseSettings.php ;-) [17:07:43] thcipriani: https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/570378/1 should fix Beta Cluster, BTW. [17:08:35] thcipriani it built successfully, woo! [17:08:48] (03Abandoned) 10Paladox: Bump Bazel version to 2.0.0 [software/gerrit] (wmf/stable-2.16) - 10https://gerrit.wikimedia.org/r/566890 (owner: 10Paladox) [17:14:37] Project beta-scap-eqiad build #286828: 04STILL FAILING in 3.4 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286828/ [17:17:23] 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall) Thanks, @Jrbranaa. I'll set something up. [17:23:17] Also, T244335 is relevant to our interests. [17:23:18] T244335: Upgrade production kubernetes clusters to a security supported version - https://phabricator.wikimedia.org/T244335 [17:24:28] Project beta-scap-eqiad build #286829: 04STILL FAILING in 0.79 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286829/ [17:27:11] The frontend team for gerrit this week are holding a fixit week. [17:34:36] Project beta-scap-eqiad build #286830: 04STILL FAILING in 0.72 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286830/ [17:35:16] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10Reedy) [17:44:31] Project beta-scap-eqiad build #286831: 04STILL FAILING in 0.65 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286831/ [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:50:14] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10Developer Productivity, 10User-brennen: dev-images: Add XDebug-enabled variant of stretch-php72-fpm-apache2 - https://phabricator.wikimedia.org/T244382 (10brennen) [17:54:32] Project beta-scap-eqiad build #286832: 04STILL FAILING in 3.5 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286832/ [17:57:52] RuntimeError: Scap failed!: Call to mwscript eval.php stderr: Notice: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 [18:00:04] We know :P [18:00:09] I just C+2'd a patch from James to fix it [18:01:09] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Research available third-party Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10dduvall) [18:02:26] marxarelli: ^ Is that going to include kube + argo providers? [18:03:18] Reedy: ah, it should! [18:03:26] :) [18:03:33] just thought I'd (double) check [18:03:41] i don't know of any Argo providers, but we should do more than a cursory look :) [18:04:17] * marxarelli should probably ask the Argo devs about that [18:04:24] heh [18:04:28] Project beta-scap-eqiad build #286833: 04STILL FAILING in 0.67 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286833/ [18:04:40] I mean, if that's the answer - "they don't exist" that's useful to know/have documented [18:05:57] hauskatze: Next CI run should work... [18:06:08] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Research available third-party Argo and Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10dduvall) [18:06:28] Reedy: definitely [18:06:40] thanks for catching that. i completely forgot we'd discussed that option [18:06:53] np :) [18:07:02] Reedy: ok. [18:10:09] (03PS2) 10Daimona Eaytoy: dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+ [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) [18:13:16] (03CR) 10Daimona Eaytoy: "Note, we might want to tweak INI settings, see https://github.com/krakjoe/pcov#notes" [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) (owner: 10Daimona Eaytoy) [18:14:51] New build started. [18:16:35] Gerrit UI gaining support to delete inline messages: https://gerrit-review.googlesource.com/c/gerrit/+/253412 [18:17:29] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10Developer Productivity, 10User-brennen: dev-images: Add XDebug-enabled variant of stretch-php72-fpm-apache2 - https://phabricator.wikimedia.org/T244382 (10thcipriani) p:05Triage→03Normal [18:18:17] oh, actually that's messages in general. [18:20:57] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10User-brennen: dev-images: Add a standalone php-fpm + Apache image for use in running mediawiki code without Blubber dependencies - https://phabricator.wikimedia.org/T240625 (10brennen) These a... [18:20:58] Reedy: I also have a config patch that'd fix it, too. [18:21:11] Indeed [18:21:23] But the core patch is probably needed as it'll break shit elsewhere later :P [18:21:23] Because this shouldn't be hard to get working. And yet. [18:21:35] Sure, but I don't care about other people's installs, I care about mine. ;-) [18:21:54] "Mine" being Wikimedia production. [18:22:26] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Jdforrester-WMF) [18:22:29] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10Jdforrester-WMF) [18:22:40] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10Jdforrester-WMF) p:05Unbreak!→03Triage We don't think this is a train blocker. [18:22:43] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Clarify which of the original New CI candidates have hosted options - https://phabricator.wikimedia.org/T240583 (10thcipriani) 05Open→03Declined [18:23:28] did someoe push around a change that broke wgLogos? [18:23:36] and if so did they push around another one to fix it? [18:23:45] yes [18:23:47] and yes [18:23:49] I'm looking at a cron job from some maintenance things, see... [18:23:52] Yippee, build fixed! [18:23:53] Project beta-scap-eqiad build #286834: 09FIXED in 9 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/286834/ [18:23:58] Whee. [18:24:06] apergos: Yes, and no, but Reedy did. [18:24:46] apergos: What is/was broken more specifically? [18:24:57] Hurrah [18:25:04] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Configuration, 10Patch-For-Review: Beta: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 - https://phabricator.wikimedia.org/T244370 (10Jdforrester-WMF) p:05Triage... [18:25:16] dumps. presumably they'll rerun if the fix went around [18:25:38] Notice: Undefined variable: wgLogos in /srv/mediawiki/wmf-config/CommonSettings.php on line 867 [18:25:43] a bunch of these [18:25:56] Yet opcache is not/was not cleaned James_F / Reedy [18:26:07] hauskatze: That's been broken for a while [18:26:18] I know. Any fix comming up? [18:26:27] Not from me [18:26:30] disclaimer: I don't know what the opcache is [18:26:37] so I'm not sure it really matters [18:27:44] Hmm. So that one is https://github.com/wikimedia/operations-mediawiki-config/commit/5eeb7c5d560efc934038fc543a9904b4c8710757 [18:28:22] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10mmodell) p:05Normal→03High [18:28:45] So... I guess https://gerrit.wikimedia.org/r/#/c/operations/mediawiki-config/+/570378/ is needed to fix that issue [18:29:06] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Code-Review-Workgroup: Define code review metrics - https://phabricator.wikimedia.org/T229510 (10Jdforrester-WMF) p:05High→03Normal [18:30:45] oh amazing, they are *still running* even with the issue... \o/ [18:32:52] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10mmodell) I'm gonna make the call that this isn't serious enough to block the train _yet_ at least not without further evidence that it's affecting p... [18:38:07] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikimedia-Portals: Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 (10Jdforrester-WMF) [18:38:09] 10Release-Engineering-Team, 10Release Pipeline, 10serviceops: Provide the official production base images for Wikimedia use - https://phabricator.wikimedia.org/T238774 (10Jdforrester-WMF) [18:38:35] 10Release-Engineering-Team (Pipeline), 10Release Pipeline, 10Wikimedia-Portals: Migrate www.wikimedia.org (the portal) to be hosted as a service - https://phabricator.wikimedia.org/T238747 (10Jdforrester-WMF) Per Tyler, we need a production-suitable (SRE-blassed) Apache image to host this. [18:38:49] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Release Pipeline, 10serviceops: Provide the official production base images for Wikimedia use - https://phabricator.wikimedia.org/T238774 (10Jdforrester-WMF) [18:39:53] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Operations, 10Security-Team, 10User-greg: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270 (10greg) 05Open→03Resolved a:03greg I think the combin... [18:39:58] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.18 deployment blockers - https://phabricator.wikimedia.org/T233866 (10mmodell) Train status: Unblocked. The train will move forward, at least until there is more evidence for production... [18:47:07] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Configuration, 10Patch-For-Review: Beta: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 - https://phabricator.wikimedia.org/T244370 (10Krinkle) [18:47:48] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Graphite, 10User-greg: Increase retention time of graphite stats for CI - https://phabricator.wikimedia.org/T242826 (10greg) p:05High→03Normal a:03greg [18:48:00] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Configuration, 10Patch-For-Review: Beta: Undefined index: 1x in /srv/mediawiki-staging/php-master/includes/Setup.php on line 186 - https://phabricator.wikimedia.org/T244370 (10Krinkle) 05Open→03Resolved... [18:48:12] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test backup and restore of Gerrit on the ganeti test instance - https://phabricator.wikimedia.org/T243982 (10thcipriani) p:05Triage→03Normal [18:48:24] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Test upgrade of Gerrit to 2.16.13 on the ganeti test instance - https://phabricator.wikimedia.org/T243981 (10thcipriani) p:05Triage→03Normal [18:49:43] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline, 10serviceops-radar: Create initial production MW-land images with blubber - https://phabricator.wikimedia.org/T238773 (10thcipriani) p:05Triage→03Normal [18:50:51] 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Release-Tools: Automate weekly branch cut - https://phabricator.wikimedia.org/T196517 (10thcipriani) a:03mmodell [18:54:36] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Rename Phabricator's "Normal" Priority field value to "Medium" - https://phabricator.wikimedia.org/T228757 (10greg) [18:59:53] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikibugs, 10Patch-For-Review, 10User-greg: Update IRC priority output from "Normal" to "Medium" - https://phabricator.wikimedia.org/T244391 (10greg) a:03greg [19:32:05] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10CPT Initiatives (API Integration Tests), 10Code-Health, and 2 others: In CI, configure MediaWiki to enable API testing - https://phabricator.wikimedia.org/T243978 (10daniel) p:05Triage→03Normal [19:39:04] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10matmarex) This issue affects production as well, but it's not new in this week's release – I found a duplicate report from 23 January: T243490. [19:39:14] 10Beta-Cluster-Infrastructure, 10MediaWiki-User-management: Inconsistent user permissions on Beta English Wikipedia - https://phabricator.wikimedia.org/T244019 (10matmarex) [19:39:52] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10jbond) @Krenair i think the package needed is `puppet-terminus-puppetdb` which is provided by the `puppetdb` source package. I have looked at buil... [19:58:47] 10Gerrit, 10Operations, 10vm-requests: Gerrit VM to test data migration - https://phabricator.wikimedia.org/T239151 (10Dzahn) [19:58:50] 10Gerrit, 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) [19:59:53] 10Gerrit, 10Operations: gerrit1002 running out of space - https://phabricator.wikimedia.org/T243808 (10Dzahn) See T243983. I added a second disk to this VM, it's an additional 10GB and mounted on /srv/dbdump. Hope that does it. [20:01:41] thcipriani https://phabricator.wikimedia.org/T243808#5853647 :) (i guess we can get https://gerrit.wikimedia.org/r/#/c/operations/software/gerrit/+/562363/-1..3 merged, and deploy it to archiva?) [20:32:35] 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-API, 10CPT Initiatives (API Integration Tests), 10Code-Health: Expose flags for controlling API tests from the action API - https://phabricator.wikimedia.org/T243977 (10daniel) @Jdforrester-WMF I don't remember why I put your name on this. Mig... [20:51:41] test [20:53:48] test reply [20:55:28] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikibugs, 10User-greg: Update IRC priority output from "Normal" to "Medium" - https://phabricator.wikimedia.org/T244391 (10valhallasw) 05Open→03Resolved Jobs restarted so this should be working now [20:55:30] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikibugs, 10User-greg: Update IRC priority output from "Normal" to "Medium" - https://phabricator.wikimedia.org/T244391 (10valhallasw) 05Open→03Resolved Jobs restarted so this should be working now [20:55:33] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Rename Phabricator's "Normal" Priority field value to "Medium" - https://phabricator.wikimedia.org/T228757 (10valhallasw) [20:55:36] 10Phabricator, 10Developer-Advocacy (Oct-Dec 2019): Rename Phabricator's "Normal" Priority field value to "Medium" - https://phabricator.wikimedia.org/T228757 (10valhallasw) [21:33:44] 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-API, 10CPT Initiatives (API Integration Tests), 10Code-Health: Expose flags for controlling API tests from the action API - https://phabricator.wikimedia.org/T243977 (10Jdforrester-WMF) a:05Jdforrester-WMF→03None Yeah, happy to help with t... [21:45:29] (03PS3) 10Jforrester: dockerfiles: coverage: add pcov, use it if we're on PHPUnit 8+ [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) (owner: 10Daimona Eaytoy) [21:46:03] (03CR) 10Jforrester: [C: 04-1] "Won't merge, but won't break the world with its version numbering any more. ;-)" [integration/config] - 10https://gerrit.wikimedia.org/r/567938 (https://phabricator.wikimedia.org/T234020) (owner: 10Daimona Eaytoy) [22:00:55] Hey foks. Need anything? [22:00:59] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10serviceops, 10Test-Coverage: Add pcov PHP extension to wikimedia apt so it can be used in Wikimedia CI - https://phabricator.wikimedia.org/T243847 (10Jdforrester-WMF) Alternatively we c... [22:08:56] Nope, other than Dayquil :) [22:15:01] James_F: is hashar leaving or something? He's tagged himself away until 2078 [22:15:54] hauskatze: sigh, no, just a vacation basically. [22:16:04] I don't know when hashar will be back, sorry. [22:16:28] greg-g: ah good [22:16:40] just the date looked weird but I'm glad it's just a pause :D [22:29:40] the date is very close to the day when the epoch overflows the 32-bit integer [22:29:46] I initially thought it was a phab bug of some kind [22:29:59] But instead it's just hashar being hashar? ;-) [22:30:04] indeed. [22:30:25] lolol [22:46:46] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: Set up new personal desktop - https://phabricator.wikimedia.org/T242248 (10brennen) 05Open→03Resolved p:05Triage→03High [23:08:32] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Krenair) ugh, ok [23:31:51] 10Beta-Cluster-Infrastructure, 10Operations: Upgrade puppet in deployment-prep (Puppet agent broken in Beta Cluster) - https://phabricator.wikimedia.org/T243226 (10Krenair) Have put puppetmaster03 back on the old version and created puppetmaster04 [23:39:31] !log Updating dev-images docker-pkg files on contint1001 for https://gerrit.wikimedia.org/r/c/releng/dev-images/+/568140 [23:39:33] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:40:39] (03PS1) 10Jforrester: Follow-up 1d72667: Also drop 'api-testing' generally [integration/config] - 10https://gerrit.wikimedia.org/r/570474 (https://phabricator.wikimedia.org/T242913) [23:46:37] (03CR) 10Jforrester: [C: 03+2] Follow-up 1d72667: Also drop 'api-testing' generally [integration/config] - 10https://gerrit.wikimedia.org/r/570474 (https://phabricator.wikimedia.org/T242913) (owner: 10Jforrester) [23:47:44] (03Merged) 10jenkins-bot: Follow-up 1d72667: Also drop 'api-testing' generally [integration/config] - 10https://gerrit.wikimedia.org/r/570474 (https://phabricator.wikimedia.org/T242913) (owner: 10Jforrester) [23:54:28] 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10CPT Initiatives (API Integration Tests), 10Code-Health, and 2 others: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10Jdforrester-WMF) [23:54:57] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-extensions-OpenStackManager, 10MediaWiki-extensions-WikimediaIncubator, 10CPT Initiatives (API Integration Tests), and 3 others: MW's new API tests are a breaking change which cause OpenStackMan... - https://phabricator.wikimedia.org/T242913 [23:57:51] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10Documentation: dev-images: Add deploy instructions to README.md - https://phabricator.wikimedia.org/T244429 (10brennen) [23:58:06] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10dev-images, 10Documentation, 10User-brennen: dev-images: Add deploy instructions to README.md - https://phabricator.wikimedia.org/T244429 (10brennen)