[00:00:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services): doc1001 permission problems for doc.wikimedia.org deploy - https://phabricator.wikimedia.org/T237707 (10Jdforrester-WMF) 05Open→03Resolved [00:00:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10serviceops, 10Developer-Wishlist (2017), and 3 others: Relocate CI generated docs and coverage reports - https://phabricator.wikimedia.org/T137890 (10Jdforrester-WMF) [00:01:27] greg-g, hey, there's a couple of hosts on https://phabricator.wikimedia.org/T236576 I'm willing to do (puppetmaster and cumin) - if you're happy with that please can I get integration projectadmin? project membership doesn't let me create and configure instances [00:04:21] RECOVERY - Free space - all mounts on deployment-puppetmaster04 is OK: OK: All targets OK [00:04:41] Krenair: nice offer! longma, you and James_F are working on this ^ right? any direction you want to give? [00:05:09] Umm. I hadn't done any work on the puppet/box side. I'd be happy for Krenair to work his magic. :-) [00:05:47] Not sure if longma had? [00:05:56] I haven't [00:06:59] Also I'm not even a member of the integration project, much less an admin, apparently. [00:07:21] greg-g: Please do make Krenair an admin. And maybe me. :-) [00:09:51] PROBLEM - Work requests waiting in Zuul Gearman server on contint1001 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [150.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:18:54] RECOVERY - Work requests waiting in Zuul Gearman server on contint1001 is OK: OK: Less than 100.00% above the threshold [90.0] https://www.mediawiki.org/wiki/Continuous_integration/Zuul https://grafana.wikimedia.org/dashboard/db/zuul-gearman?panelId=10&fullscreen&orgId=1 [00:20:00] oh, forgot to mention here: solved the deployment-puppetmaster04 disk space thing. was an uncommented debug line lurking in /usr/share/puppet/rack/puppet-master/config.ru [00:21:46] Krenair: Ah, so not tracked in git. [00:22:10] * James_F was scouring each non-production patch in puppetmaster to see which it might be. [00:22:11] 10Release-Engineering-Team-TODO, 10Scap: Running "scap pull" on deploy1001 halts for 2 minutes, then reports a php-fpm error - https://phabricator.wikimedia.org/T246959 (10Krinkle) [00:23:57] !log made Krenair a projectadmin in integration [00:23:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:26:05] !log added Jforrester as user and projectadmin to integration [00:26:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:27:54] !log removed cmcmahon, Coren, khorn, manybubbles, mobrovac, laner from users of integration [00:27:55] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:28:13] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds [00:28:49] hm [00:28:56] that's odd. looks fine to me? [00:33:05] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 51918 bytes in 1.020 second response time [00:33:09] * Krenair shrug [00:34:04] looking at commands I had running in cloud vps over the last few minutes actually maybe something was wrong with the network? [00:34:56] some real weird stuff out of puppet, `getaddrinfo: No address associated with hostname` when trying to get node definition. HTTP 500 from puppet-enc on another [00:35:04] but then re-ran and it's fine [00:38:41] Odd. [00:40:26] Looking at the list of instances in integration, every single one is either based off a deprecated stretch or buster image, or off jessie itself. [00:40:43] Oh, wait, no, there's a test image that's up-to-date. [00:50:30] images get deprecated fairly frequently [00:50:54] as long as the instance gets package updates and is replaced when the distro version goes out of support it should be ok [00:52:22] (and gets restarted for kernel security updates etc. I guess) [00:53:11] the 'image' column is about what the instance was originally built with [00:54:08] there have been cases where people have made an instance and then tried to do a distro upgrade in place [01:17:14] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Phabricator-Extensions, 10User-brennen: Related Gerrit Patches: Do not display CR-2 patches as "awaiting review" - https://phabricator.wikimedia.org/T245247 (10mmodell) 05Open→03Resolved [01:17:40] 10Phabricator (next), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Phabricator-Extensions, 10User-brennen: Related Gerrit Patches: Do not display CR-2 patches as "awaiting review" - https://phabricator.wikimedia.org/T245247 (10mmodell) [01:22:55] (03CR) 10Jforrester: [C: 03+2] php-fpm-apache: set upload size limit to 100M [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576983 (https://phabricator.wikimedia.org/T246930) (owner: 10Brennen Bearnes) [01:23:07] (03CR) 10Jforrester: [C: 03+2] "(Feel free to publish.)" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576983 (https://phabricator.wikimedia.org/T246930) (owner: 10Brennen Bearnes) [01:24:30] (03CR) 10Jforrester: [C: 03+2] "This ran fine, and worked. Let's land it and iterate later if we need to." [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) (owner: 10Brennen Bearnes) [01:24:59] (03Merged) 10jenkins-bot: add delete-wmf-branches [tools/release] - 10https://gerrit.wikimedia.org/r/576136 (https://phabricator.wikimedia.org/T244368) (owner: 10Brennen Bearnes) [01:25:50] ^ thx for the pile o' review James_F. [01:27:15] (03CR) 10Brennen Bearnes: [V: 03+2] php-fpm-apache: set upload size limit to 100M [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576983 (https://phabricator.wikimedia.org/T246930) (owner: 10Brennen Bearnes) [01:28:55] !log Updating dev-images docker-pkg files on contint1001 for T246930 [01:28:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:28:57] T246930: Increase Docker image's PHP upload limit from 2 MiB to 100 MiB - https://phabricator.wikimedia.org/T246930 [01:32:41] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10User-brennen: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image - https://phabricator.wikimedia.org/T246942 (10brennen) [03:20:03] 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Wikimedia-Site-requests, 10MW-1.35-notes (1.35.0-wmf.24; 2020-03-17): Investigate whether wmf-config's configuration cache is still needed - https://phabricator.wikimedia.org/T169821 (10Krinkle) The various patches link on this task have al... [04:11:38] tgr: It seems the "Release Engineering" room for wikimedia.riot (relay from here IRC -releng) is bouncing currently. The underlying IRC channel (this one) requires a registered nick. [04:12:34] yeah, I promised to look into that and then didn't, sorry :( [04:13:28] it kicks you out even if your nick is registered, so somehow the bridge bot seems confused about user modes [04:13:52] I'll see if I can find more time this weekend [04:16:14] tgr: ah ok, I don't have my matric account registered with IRC so I asumed that was the reason. thx for looking into it. I don't mind if it works, it was more that clicking around in the new toy led to something didn't work :) [06:16:44] I can read this channel from matrix/riot.im, but I just tried sending from there and it didn'tshow up here [07:00:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10Osnard) We will definitely implement a rate-limit in our tool. Can you point me to that co... [07:07:58] 10Phabricator, 10Project-Admins, 10CommRel-Design, 10WMF-Communications: Archive #CommRel-Design and related Phabricator Form? - https://phabricator.wikimedia.org/T246853 (10Qgil) No additional thoughts on this task. I just want to state that the #community-relations team doesn't offer design support any... [07:08:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10Osnard) >Eventually some of the changes got force merged, which eventually cause Zuul to c... [07:08:50] 10Phabricator, 10Project-Admins, 10CommRel-Design, 10WMF-Communications: Archive #CommRel-Design and related Phabricator Form? - https://phabricator.wikimedia.org/T246853 (10Qgil) [08:24:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10hashar) >>! In T246903#5943952, @Osnard wrote: > We will definitely implement a rate-limit... [08:27:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10hashar) >>! In T246903#5943964, @Osnard wrote: >>Eventually some of the changes got force... [08:47:39] 10Phabricator, 10Operations, 10Traffic: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10akosiaris) >>! In T246923#5943203, @Urbanecm wrote: >>>! In T246923#5943004, @Krenair wrote: >> If it is that I would not expect HTTP 501 responses. > >>>! In T246923... [08:48:06] 10Phabricator, 10Operations, 10Traffic: Phabricator is inaccessible from Egypt: HTTP 501 error - https://phabricator.wikimedia.org/T246923 (10akosiaris) [08:48:10] 10Phabricator, 10Security: if phabricator users are blocked due to IP range blocks send a 4xx rather than 500 - https://phabricator.wikimedia.org/T229620 (10akosiaris) [08:55:56] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10hashar) 05Stalled→03Open @mmod... [09:00:44] (03PS4) 10Kosta Harlan: Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [09:06:06] (03CR) 10DCausse: Add ElasticSearch dev image (032 comments) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) (owner: 10Kosta Harlan) [09:07:43] (03PS5) 10Kosta Harlan: Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [09:09:41] (03PS6) 10Kosta Harlan: Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [09:11:40] (03CR) 10Kosta Harlan: Add ElasticSearch dev image (032 comments) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) (owner: 10Kosta Harlan) [09:15:04] (03CR) 10DCausse: [C: 03+1] Add ElasticSearch dev image (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) (owner: 10Kosta Harlan) [09:36:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) [09:41:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) It seems that Zuul no more receives any events through Gerrit events... [09:45:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10BlueSpice: Investigate Zuul crashing due to BlueSpice REL1_31 backlog - https://phabricator.wikimedia.org/T246903 (10Osnard) > If you have to routinely force merges changes, it is an indication that somethin... [09:47:51] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-DannyS712: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10DannyS712) p:05Triage→03High Was about to report this... [09:51:31] (03CR) 10Daimona Eaytoy: "> Deployed for the --no-update fix, but this doesn't appear to make a" [integration/config] - 10https://gerrit.wikimedia.org/r/575600 (https://phabricator.wikimedia.org/T226117) (owner: 10Daimona Eaytoy) [09:53:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-DannyS712: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) Zuul stack dump: {F31666363} There is apparentl... [10:10:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-DannyS712: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) On Gerrit side I see: ` counterexample,lines=25 [... [10:21:24] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), and 2 others: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) My guess of what happens is roughly: * Ger... [10:44:12] 10Release-Engineering-Team, 10Language-Team, 10MediaWiki-Interface, 10I18n, and 2 others: Interface language using Accept-Language header value instead of $wgLanguageCode - https://phabricator.wikimedia.org/T246071 (10Lucas_Werkmeister_WMDE) Well, I have no idea what the alerting should be… adding an alert... [11:06:45] 10Release-Engineering-Team, 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team), 10Patch-For-Review: Enable phpdbg on mwdebug* servers - https://phabricator.wikimedia.org/T244549 (10hnowlan) 05Open→03Resolved [11:07:13] (03CR) 10Jbond: "> Patch Set 1:" [integration/config] - 10https://gerrit.wikimedia.org/r/575012 (owner: 10Jbond) [11:12:04] 10Phabricator, 10Operations, 10Security-Team, 10Security: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) - https://phabricator.wikimedia.org/T245771 (10jbond) @chasemp can you please crate me a couple of test users so i can ensure the off boarding script is worki... [11:16:56] 10Gerrit, 10Gerrit-Privilege-Requests: Give access to l10n-bot - https://phabricator.wikimedia.org/T246978 (10JoeWalsh) [11:32:38] 10Phabricator, 10Developer-Advocacy (Jan-Mar 2020): Decrease issues with assignee field set for years without progress (aka cookie licking) - https://phabricator.wikimedia.org/T228575 (10Aklapper) Excluding open assigned tasks with #PATCH-FOR-REVIEW isn't as expensive as I had expected. That SQL query for "two... [11:33:46] 10Continuous-Integration-Config: Please add mediawiki-i18n-check-docker to repository mediawiki/services/mobileapps - https://phabricator.wikimedia.org/T246979 (10JoeWalsh) [11:38:08] (03PS7) 10Kosta Harlan: Add ElasticSearch dev image [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) [11:47:09] (03CR) 10Kosta Harlan: Add ElasticSearch dev image (031 comment) [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) (owner: 10Kosta Harlan) [11:47:49] (03PS1) 10Joewalsh: Add mediawiki-i18n-check template to mediawiki/services/mobileapps [integration/config] - 10https://gerrit.wikimedia.org/r/577219 (https://phabricator.wikimedia.org/T246979) [11:50:40] 10Continuous-Integration-Config, 10Patch-For-Review: Please add mediawiki-i18n-check-docker to repository mediawiki/services/mobileapps - https://phabricator.wikimedia.org/T246979 (10JoeWalsh) [11:50:42] 10Gerrit, 10Gerrit-Privilege-Requests: Give access to l10n-bot - https://phabricator.wikimedia.org/T246978 (10JoeWalsh) [12:13:46] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10Xqt) 05Open→03Resolved a:03Xqt I think it is better now. [12:42:46] (03CR) 10Kosta Harlan: "Documentation for usage is here https://www.mediawiki.org/wiki/MediaWiki-Docker#ElasticSearch" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/576327 (https://phabricator.wikimedia.org/T231864) (owner: 10Kosta Harlan) [12:49:27] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), and 2 others: CI / Zuul not processing changes - https://phabricator.wikimedia.org/T246973 (10hashar) 05Open→03Resolved a:03hashar [13:09:04] 10Release-Engineering-Team-TODO, 10Operations: Should 'doc' machines (i.e. doc1001) have contint-roots as a group? - https://phabricator.wikimedia.org/T245691 (10akosiaris) I 'll remove the #sre-access-requests tag for based on the task status. Feel free to readd when it's back to Open so SRE can work on it. [13:38:47] (03CR) 10Hashar: [C: 03+2] "I have updated the job ;)" [integration/config] - 10https://gerrit.wikimedia.org/r/575012 (owner: 10Jbond) [13:39:39] (03Merged) 10jenkins-bot: puppet-compiler: update the host parsing login [integration/config] - 10https://gerrit.wikimedia.org/r/575012 (owner: 10Jbond) [14:09:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10hashar) a:05hashar→03None That i... [14:09:49] 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Quibble: Create an integration test running Quibble with mediawiki/core - https://phabricator.wikimedia.org/T235118 (10hashar) a:05hashar→03None [14:18:50] Hello, can someone requeue gate-and-submit jobs for my patches? https://gerrit.wikimedia.org/r/#/q/reviewedby:%22Thiemo+Kreuz+(WMDE)+%253Cthiemo.kreuz%2540wikimedia.de%253E%22+owner:%22Zoranzoki21+%253Czorandori4444%2540gmail.com%253E%22+status:open [14:19:20] Zoranzoki21: "recheck" might do it [14:19:38] recheck won't requeue gate-and-submit [14:19:52] (for me, I don't have +2 option) [14:20:08] hmm [14:20:12] doesnt bah [14:20:58] https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikEdDiff/+/575773/ fails ;) [14:21:52] Zoranzoki21: I +2ed a few :) [14:22:23] All ok, thank you! [14:25:13] hashar: You forgot https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/BrickipediaExtra/+/576849/ [14:28:51] Zoranzoki21: +2ed :) [14:31:57] Tysm [14:32:41] I'm going now [14:32:43] Bye :) [14:38:10] 10Project-Admins: Lingua Libre Milestone - https://phabricator.wikimedia.org/T246992 (10Eavqwiki) [14:51:29] 10Project-Admins: Lingua Libre Sign It - https://phabricator.wikimedia.org/T246993 (10Eavqwiki) [14:56:03] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen, 10User-zeljkofilipin: 1.35.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T233870 (10LarsWirzenius) [14:59:47] 10Beta-Cluster-Infrastructure, 10MW-1.35-notes (1.35.0-wmf.23; 2020-03-10): Fatal error: Class 'LoggedUpdateMaintenance' not found when using mwscript - https://phabricator.wikimedia.org/T246754 (10Jdforrester-WMF) [15:01:20] 10Project-Admins: Lingua Libre Sign It - https://phabricator.wikimedia.org/T246993 (10Aklapper) Hi, is there any code base already? (Or a location for a [future] code base? Or a web page that provides more information? [15:11:52] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments, 10User-brennen, 10User-zeljkofilipin: 1.35.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T233870 (10Jdforrester-WMF) 05Open→03Resolved Congratulations. [15:23:15] (03CR) 10Jforrester: [C: 03+2] "This is fine, but you'll also need to give l10nbot the C+2 right for the repo in gerrit config for it to work, if you haven't already." [integration/config] - 10https://gerrit.wikimedia.org/r/577219 (https://phabricator.wikimedia.org/T246979) (owner: 10Joewalsh) [15:24:12] (03Merged) 10jenkins-bot: Add mediawiki-i18n-check template to mediawiki/services/mobileapps [integration/config] - 10https://gerrit.wikimedia.org/r/577219 (https://phabricator.wikimedia.org/T246979) (owner: 10Joewalsh) [15:24:38] 10Project-Admins: Lingua Libre Sign It - https://phabricator.wikimedia.org/T246993 (100x010C) @Aklapper Hi, there is already this github repo : https://github.com/lingua-libre/SignIt (issues and so on will be closed there once the project here will be created). Another useful link is the addon page on the mozill... [15:25:52] !log Zuul: Add mediawiki-i18n-check template to mediawiki/services/mobileapps for T246979 [15:25:53] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:25:54] T246979: Please add mediawiki-i18n-check-docker to repository mediawiki/services/mobileapps - https://phabricator.wikimedia.org/T246979 [15:35:04] hm i want to enable a feature on just a small active wiki, haw.wikipedia.org [15:35:12] how do I know what key to use in InitialiseSettings.php [15:35:12] ? [15:36:50] wgImportSources ? [15:37:07] hmm maybe it is hawwiki [15:37:07] ? [15:39:24] James_F: does this look right to you? [15:39:25] https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/577260 [15:39:31] to enable this just on haw.wikipedia.org ? [15:39:42] ottomata: In the staff meeting. Will look later. [15:39:50] Ok me too but you know... :p [15:39:58] ty [16:04:17] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10AlexisJazz) >>! In T99740#5905688, @Jdforrester-WMF wrote: > Deployed to Beta... [16:45:11] 10Phabricator, 10Operations, 10Security-Team, 10Security: Adjust onboarding/offboarding logic to accommodate changes to #security (now acl*security) - https://phabricator.wikimedia.org/T245771 (10chasemp) >>! In T245771#5944399, @jbond wrote: > @chasemp can you please crate me a couple of test users so i c... [17:04:19] 10Release-Engineering-Team-TODO, 10Scap, 10Developer Productivity: Scap's link to Logstash for canary failure is not in sync with the monitoring query - https://phabricator.wikimedia.org/T247005 (10Krinkle) [17:08:58] !log Remove obsolete filters from "scap canary" dashboard, including obsolete "hhvm" query [17:08:59] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [17:09:12] Krinkle: Oh good, I was meaning to do that. [17:09:31] Krinkle: It's also embedded in the link in puppet, right? [17:48:47] ottomata: Hey. Yeah, that looks good to go. Note the output of https://integration.wikimedia.org/ci/job/operations-mw-config-php72-composer-diffConfig-docker/1393/console shows that only hawwiki is effected. [17:54:54] 10Phabricator: Disable Phabricator 2FA for WMFOffice - https://phabricator.wikimedia.org/T247011 (10WMFOffice) [17:55:47] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Disable Phabricator 2FA for WMFOffice - https://phabricator.wikimedia.org/T247011 (10greg) a:03greg Ack. @jrbs and I chatted on IRC as well. [17:56:05] OHHH cool [17:56:06] ok thanks [17:57:00] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Disable Phabricator 2FA for WMFOffice - https://phabricator.wikimedia.org/T247011 (10jrbs) Posting just to sheepishly confirm this task is legit. [17:57:40] 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-greg: Disable Phabricator 2FA for WMFOffice - https://phabricator.wikimedia.org/T247011 (10greg) 05Open→03Resolved ` gjg@phab1001:~$ sudo /srv/phab/phabricator/bin/auth st... [18:07:12] 10Gerrit, 10Cleanup, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog: Archive mediawiki/services/mobileapps/deploy once it's no longer used - https://phabricator.wikimedia.org/T247012 (10Jdforrester-WMF) 05Open→03Stalled [18:07:28] 10Gerrit, 10Cleanup, 10Mobile-Content-Service, 10Product-Infrastructure-Team-Backlog: Archive mediawiki/services/mobileapps/deploy once it's no longer used - https://phabricator.wikimedia.org/T247012 (10Jdforrester-WMF) [18:13:25] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10Dvorapa) 05Resolved→03Open I don't think so, it is similar to the example in description [18:55:59] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [18:55:59] 10Release-Engineering-Team, 10serviceops, 10Core Platform Team Workboards (Clinic Duty Team): Enable phpdbg on mwdebug* servers - https://phabricator.wikimedia.org/T244549 (10EBernhardson) I'm sure this could be further improved, but in general it works! ` ebernhardson@mwdebug1002:~$ cat test.php RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [19:10:07] awww come one [19:10:15] even with the cron job?? [19:11:15] 10Release-Engineering-Team-TODO, 10Scap, 10MediaWiki-Internationalization, 10Performance-Team, 10Patch-For-Review: Use static php array files for l10n cache at WMF (instead of CDB) - https://phabricator.wikimedia.org/T99740 (10ori) >>! In T99740#5945217, @AlexisJazz wrote: >>>! In T99740#5905688, @Jdforr... [19:29:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10mmodell) Deployment repo: https://... [19:35:38] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10hashar) The code to deploy is in `... [21:36:39] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Demonstrate abilities of MediaWiki-Docker with a full TimedMediaHandler stack - https://phabricator.wikimedia.org/T247037 (10Jdforrester-WMF) [21:37:38] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Demonstrate abilities of MediaWiki-Docker with a full TimedMediaHandler stack - https://phabricator.wikimedia.org/T247037 (10Jdforrester-WMF) [21:37:41] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10User-brennen: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image - https://phabricator.wikimedia.org/T246942 (10Jdforrester-WMF) [21:37:45] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Job queue runners for MediaWiki-Docker - https://phabricator.wikimedia.org/T246935 (10Jdforrester-WMF) [21:37:50] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, and 2 others: Increase Docker image's PHP upload limit from 2 MiB to 100 MiB - https://phabricator.wikimedia.org/T246930 (10Jdforrester-WMF) [21:38:41] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, and 2 others: Increase Docker image's PHP upload limit from 2 MiB to 100 MiB - https://phabricator.wikimedia.org/T246930 (10Jdforrester-WMF) Resolved? [21:39:22] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Demonstrate abilities of MediaWiki-Docker with a full TimedMediaHandler stack - https://phabricator.wikimedia.org/T247037 (10brennen) [21:39:48] 10Release-Engineering-Team, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, and 2 others: Increase Docker image's PHP upload limit from 2 MiB to 100 MiB - https://phabricator.wikimedia.org/T246930 (10brennen) 05Open→03Resolved Yep. [21:39:51] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10dev-images, 10User-brennen: Demonstrate abilities of MediaWiki-Docker with a full TimedMediaHandler stack - https://phabricator.wikimedia.org/T247037 (10brennen) [21:39:53] 10Release-Engineering-Team-TODO, 10Wikimedia-General-or-Unknown, 10Wikimedia-Site-requests, 10MW-1.35-notes (1.35.0-wmf.22; 2020-03-03): Try to make wmf-config/wgConf's per-wiki configuration cache redundant - https://phabricator.wikimedia.org/T169821 (10Krinkle) [21:40:52] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10DStrine) hi, I just wanted to check in on the status of this. Is better to use @mmodell or @20after4 ? [21:42:02] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10mmodell) @DStrine: this is my work user account, @mmodell [21:47:53] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10mmodell) If you just need a template task, maybe a custom form is overkill? You can use https://tools.wmflabs.org/phabulous/ to make a custom link with the description pre-filled. [21:48:28] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10MediaWiki-Docker, 10User-brennen: TimedMediaHandler's ffmpeg processes get stuck when using resource limits on Docker image - https://phabricator.wikimedia.org/T246942 (10brion) T206957 is the related ticket. [21:53:47] 10Release-Engineering-Team-TODO, 10Scap, 10Developer Productivity: Scap's link to Logstash for canary failure is not in sync with the monitoring query - https://phabricator.wikimedia.org/T247005 (10Jdforrester-WMF) The URL is specified in puppet at https://gerrit.wikimedia.org/g/operations/puppet/+/HEAD/modu... [21:53:51] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10DStrine) I can try showing the url solution to stakeholders and seeing what they think. I made this one but it doesn't give me the nice formatting I have in my example task. https://phabricator.wikimedia.org/manip... [21:56:26] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10mmodell) https://phabricator.wikimedia.org/maniphest/task/edit/form/1/?description=%3D%20Overview%3A%20%3D%0D%0A%0D%0A**User%20story%3A**%20%20As%20a%20%5Btype%20of%20user%5D%20I%20need%20%5Bfeature%20description%5D%... [22:00:55] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10mmodell) I created a furl short url for that link ^ which can be embedded anywhere by using the short-name like so: `((fundraising-story))` and that turns into this: ((fundraising-story)) [22:06:15] 10Phabricator: Create new custom fundraising form - https://phabricator.wikimedia.org/T243961 (10DStrine) ok thanks. I will run this by the stakeholders. They may still want something in the UI but I'll check. [22:08:21] 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) FYI grafana dash here: https://grafana.wikimedia.org/d/znIuUcsWz/eventstreams-k8s Not totally... [22:45:18] 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Scap, 10Patch-For-Review, 10WorkType-NewFunctionality: Add a --beta option to 'scap update-interwiki-cache' to be able to update the interwiki-labs.php file using Scap - https://phabricator.wikimedia.org/T198844 (10Jdforrester-WMF) 05Open→03Re... [22:45:55] 10Continuous-Integration-Config, 10Pywikibot, 10Pywikibot-tests: Jenkins output for pywikibot job is hard to read - https://phabricator.wikimedia.org/T117570 (10Xqt) a:05Xqt→03None [22:47:21] are people aware of the puppet error on integration-agent-docker-1005.integration.eqiad.wmflabs ? [22:47:49] it's looking for docker-ce=18.06.2~ce~3-0~debian but that's not available in apt, instead we have 19.03.5~3-0~debian-stretch? [22:54:39] Krenair: We use 18.06 everywhere, don't we? 19.03 has a major performance issue. [22:55:00] everywhere apart from agent 1005 it seems [22:55:03] hashar got SRE to roll back I think. [22:55:16] Oh, 1005 ran into issues? Fun. [22:55:23] Variation isn't the plan. :-) [22:55:35] And no to answer your question, I wasn't aware. [23:03:23] James_F, do we have a task about this performance problem? [23:03:43] Krenair: Yes. [23:03:46] * James_F hunts [23:04:04] T236675 [23:04:05] T236675: Investigate Docker slowness between 18.06.2 and 18.09.7 - https://phabricator.wikimedia.org/T236675 [23:04:35] It's on my list of things to talk with hashar about tomorrow. [23:08:47] cool [23:10:31] Well, it is *now*; I meant to, but I'd forgotten it until you mentioned it. :-) Thank you. [23:10:38] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (... [23:11:47] !log T236576 Shut down integration-puppetmaster01 in favour of integration-puppetmaster-02. /root and /home archives can be found in /root on the new system. All integration instances moved to use the new master. Old master to be deleted on 12th March in the absence of objections [23:11:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [23:11:49] T236576: "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 [23:13:37] PROBLEM - Host integration-puppetmaster01 is DOWN: CRITICAL - Host Unreachable (172.16.3.17) [23:14:50] James_F: mwext-node10-rundoc-docker is run on every commit now. Did that change recently? I thought previously it ran on post-merge (just checking i understand correctly) [23:24:30] Jdlrobson: rundoc runs in test and gate; publish runs in post-merge, yes. [23:24:52] Jdlrobson: I don't think that changed recently in general; it may have changed for a few repos as part of clean-up? [23:24:57] Which repo? [23:25:18] (The idea is that doc code doesn't need to be installed in the image that tests your code, and v.v.) [23:34:24] Krenair: So, shinken is now complaining about integration-puppetmaster01 going away. Do you know where that's configured? [23:34:47] James_F: https://gerrit.wikimedia.org/r/576533 [23:34:50] it'll be automatic [23:35:06] Jdlrobson: Oh, that. [23:35:07] im happy to merge it, but i just wanted to make sure if there's an issue with the jsdoc documentation meaning it can't be built it will block merges [23:35:09] config should get cleaned away when the instance is deleted [23:35:17] OK, that works for me. [23:35:22] Jdlrobson: Sure, go ahead. [23:35:26] roger that :) [23:35:37] * James_F is more than a little burnt out on that argument. [23:39:02] 10Phabricator-Bot-Requests: Please create Phabricator bot - QutieBot - https://phabricator.wikimedia.org/T247041 (10Jrbranaa) [23:45:50] 10Release-Engineering-Team (Code Health), 10Quality and Test Engineering Office Hours, 10User-zeljkofilipin: Selenium Test Automation - https://phabricator.wikimedia.org/T228531 (10Jrbranaa) [23:48:51] 10Continuous-Integration-Infrastructure, 10Code-Review-Workgroup, 10Quality-and-Test-Engineering-Team (QTE), 10Quibble: Lightweight preview environment for gerrit changes - https://phabricator.wikimedia.org/T241140 (10Jrbranaa)