[00:00:16] Ah right, so the process isn't defined yet. [00:01:08] It looks like the direction we're headed toward is that keeping beta puppetmaster non-conflicting would be expected to be done by SRE when they merge patches for prod. Which seems reasonable if there is alerting for it in a way that makes it discoverable soon after merging. [00:01:35] given they're most likely to be 1) around and 2) know the area [00:12:11] Krinkle: right. Looks like one conflict is Krenair's patch which already merged in prod [00:12:31] I don't think sre are particularly interested in keeping up with the cherry picks on beta though honestly [00:13:59] 5f853b6 2019-03-23 18:08 Alex Monk o nrpe::monitor_service and monitoring::service: Allow ensure absent without description/nrpe_command [00:14:13] looks like that can be dropped but when I do then a different patch fails to apply [00:14:25] which is 3f02f13 2019-04-01 11:17 Alex Monk o tlsproxy::localssl: No hardcoding of prod webproxy hostname [00:15:00] I dropped the first one [00:15:10] dropping the second one for a moment while I rebase it, will cherry-pick it back [00:19:52] Krinkle, twentyafterfour: it's sorted now [00:20:22] The usual process is that you just fix the conflict. [00:21:00] Krinkle: ok cool! I tried to fix the conflict but rebase was giving me trouble, glad you were able to get it straightened out ;) [00:21:07] Krenair: sorry [00:21:22] * twentyafterfour is tired [00:21:37] thanks for fixing it up [00:35:14] (03PS2) 10Krinkle: zuul: Remove unused "^(mwgate-|)composer-validate$" override [integration/config] - 10https://gerrit.wikimedia.org/r/501813 [00:35:16] (03CR) 10Krinkle: [C: 03+2] zuul: Remove unused "^(mwgate-|)composer-validate$" override [integration/config] - 10https://gerrit.wikimedia.org/r/501813 (owner: 10Krinkle) [00:37:30] (03Merged) 10jenkins-bot: zuul: Remove unused "^(mwgate-|)composer-validate$" override [integration/config] - 10https://gerrit.wikimedia.org/r/501813 (owner: 10Krinkle) [00:52:59] (03PS9) 10Dduvall: WIP Provide configurable pipeline execution [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 [00:53:56] !log Reloading Zuul to deploy https://gerrit.wikimedia.org/r/501813 [00:53:58] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [00:54:01] (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/499918 (owner: 10Dduvall) [01:05:25] 10Phabricator: Possile vandal account on Phabricator - https://phabricator.wikimedia.org/T220253 (10Huji) [01:05:48] (03CR) 10Legoktm: [C: 03+2] [wikihiero] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501786 (owner: 10Umherirrender) [01:06:21] (03CR) 10Legoktm: [C: 03+2] [MapSources] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501714 (owner: 10Umherirrender) [01:06:37] (03CR) 10Legoktm: [C: 03+2] [GoogleNewsSitemap] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501707 (owner: 10Umherirrender) [01:06:50] (03CR) 10Legoktm: [C: 03+2] [NewUserMessage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501782 (owner: 10Umherirrender) [01:07:05] (03CR) 10Legoktm: [C: 03+2] [FundraiserLandingPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501709 (owner: 10Umherirrender) [01:07:20] (03CR) 10Legoktm: [C: 03+2] [timeline] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500111 (owner: 10Umherirrender) [01:07:34] (03CR) 10Legoktm: [C: 03+2] [intersection] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500106 (owner: 10Umherirrender) [01:07:36] (03Merged) 10jenkins-bot: [wikihiero] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501786 (owner: 10Umherirrender) [01:08:02] (03Merged) 10jenkins-bot: [MapSources] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501714 (owner: 10Umherirrender) [01:08:16] (03CR) 10Legoktm: [C: 03+2] [XAnalytics] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500127 (owner: 10Umherirrender) [01:08:18] (03CR) 10Legoktm: [C: 03+2] [TemplateSandbox] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500119 (owner: 10Umherirrender) [01:08:20] (03Merged) 10jenkins-bot: [GoogleNewsSitemap] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501707 (owner: 10Umherirrender) [01:08:23] 10Phabricator: Possile vandal account on Phabricator - https://phabricator.wikimedia.org/T220253 (10Peachey88) That is old gerrit vandalism that has been fixed, There was a problem with [[ https://lists.wikimedia.org/pipermail/wikitech-l/2019-April/091906.html | gerrit/phab replication ]] that is now fixed so th... [01:08:48] (03Merged) 10jenkins-bot: [NewUserMessage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501782 (owner: 10Umherirrender) [01:08:51] (03Merged) 10jenkins-bot: [FundraiserLandingPage] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501709 (owner: 10Umherirrender) [01:09:02] (03Merged) 10jenkins-bot: [timeline] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500111 (owner: 10Umherirrender) [01:09:09] Whee. [01:09:17] (03Merged) 10jenkins-bot: [intersection] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500106 (owner: 10Umherirrender) [01:09:51] (03Merged) 10jenkins-bot: [XAnalytics] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500127 (owner: 10Umherirrender) [01:10:27] (03Merged) 10jenkins-bot: [TemplateSandbox] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/500119 (owner: 10Umherirrender) [01:10:34] !log deploying https://gerrit.wikimedia.org/r/501786 https://gerrit.wikimedia.org/r/501714 https://gerrit.wikimedia.org/r/501707 https://gerrit.wikimedia.org/r/501782 https://gerrit.wikimedia.org/r/501709 https://gerrit.wikimedia.org/r/500111 https://gerrit.wikimedia.org/r/500106 https://gerrit.wikimedia.org/r/500127 https://gerrit.wikimedia.org/r/500119 [01:10:35] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:14:36] (03CR) 10Legoktm: [C: 03+2] [Quiz] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501784 (owner: 10Umherirrender) [01:17:17] (03Merged) 10jenkins-bot: [Quiz] Add phan [integration/config] - 10https://gerrit.wikimedia.org/r/501784 (owner: 10Umherirrender) [01:26:03] 10Phabricator: Possile vandal account on Phabricator - https://phabricator.wikimedia.org/T220253 (10Huji) 05Open→03Declined Got it. I just saw it today so I thought I should make sure it is noted. [01:28:53] 10Release-Engineering-Team, 10MediaWiki-Core-Testing, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure), 10phan: phan 1.2.6 is OOMing on MediaWiki core - https://phabricator.wikimedia.org/T219114 (10Legoktm) OK, I haven't seen any OOMs after deploying that change. I also submitted h... [01:32:00] (03PS3) 10Legoktm: Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 [01:32:34] (03CR) 10Legoktm: [C: 03+2] Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 (owner: 10Legoktm) [01:36:30] (03Merged) 10jenkins-bot: Use custom npm-php job for wikipeg [integration/config] - 10https://gerrit.wikimedia.org/r/501807 (owner: 10Legoktm) [01:47:53] (03PS1) 10Legoktm: npm-php: Install composer [integration/config] - 10https://gerrit.wikimedia.org/r/501847 [01:48:39] (03CR) 10Legoktm: [C: 03+2] npm-php: Install composer [integration/config] - 10https://gerrit.wikimedia.org/r/501847 (owner: 10Legoktm) [01:50:06] (03Merged) 10jenkins-bot: npm-php: Install composer [integration/config] - 10https://gerrit.wikimedia.org/r/501847 (owner: 10Legoktm) [01:52:03] !log rebuilding npm-php image https://gerrit.wikimedia.org/r/501847 [01:52:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:15:28] (03PS1) 10Legoktm: npm-php: Install php-mbstring, drop composer [integration/config] - 10https://gerrit.wikimedia.org/r/501848 [02:15:41] (03CR) 10Legoktm: [C: 03+2] npm-php: Install php-mbstring, drop composer [integration/config] - 10https://gerrit.wikimedia.org/r/501848 (owner: 10Legoktm) [02:17:09] (03Merged) 10jenkins-bot: npm-php: Install php-mbstring, drop composer [integration/config] - 10https://gerrit.wikimedia.org/r/501848 (owner: 10Legoktm) [02:17:28] !log rebuilding npm-php image again https://gerrit.wikimedia.org/r/501848 [02:17:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:17:51] * legoktm -> afk [02:42:02] (03PS1) 10Legoktm: wikipeg job tweaks [integration/config] - 10https://gerrit.wikimedia.org/r/501849 [02:42:36] (03CR) 10Legoktm: [C: 03+2] wikipeg job tweaks [integration/config] - 10https://gerrit.wikimedia.org/r/501849 (owner: 10Legoktm) [02:45:27] (03Merged) 10jenkins-bot: wikipeg job tweaks [integration/config] - 10https://gerrit.wikimedia.org/r/501849 (owner: 10Legoktm) [03:57:46] I wonder what the output of `stty -a` is when a docker image is printing to jenkins output [03:57:51] is there an easy way to test that? [05:57:44] 10Release-Engineering-Team, 10MediaWiki-Core-Testing, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure), 10phan: phan 1.2.6 is OOMing on MediaWiki core - https://phabricator.wikimedia.org/T219114 (10Legoktm) And filed https://github.com/symfony/symfony/issues/30877 with symfony. [06:21:47] interestingly if we were using bash as our shell, $COLUMNS would be already set [06:21:49] but dash doesn't [06:59:49] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:35:57] (03PS1) 10Umherirrender: [MassEditRegex] Add seccheck [integration/config] - 10https://gerrit.wikimedia.org/r/501859 [07:40:30] (03CR) 10Daimona Eaytoy: "Thanks for the comment, I'm abandoning this change and suppressing warnings on the script, then. I think this will also make phan analysis" [integration/config] - 10https://gerrit.wikimedia.org/r/495689 (owner: 10Daimona Eaytoy) [07:40:36] (03Abandoned) 10Daimona Eaytoy: Add dependencies for Echo [integration/config] - 10https://gerrit.wikimedia.org/r/495689 (owner: 10Daimona Eaytoy) [08:18:23] (03PS1) 10Umherirrender: Add phan to some skins [integration/config] - 10https://gerrit.wikimedia.org/r/501864 [08:25:11] (03PS2) 10Umherirrender: Add phan to some skins [integration/config] - 10https://gerrit.wikimedia.org/r/501864 [08:55:20] PROBLEM - SSH on integration-slave-docker-1053 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [09:12:57] (03PS1) 10Umherirrender: run core phan docker image with /cache mounted [integration/config] - 10https://gerrit.wikimedia.org/r/501867 [09:13:24] (03CR) 10Umherirrender: "I have used I1ee4963da2667ad7d9192e6519f760a89d0598ae as template for this change" [integration/config] - 10https://gerrit.wikimedia.org/r/501867 (owner: 10Umherirrender) [09:31:11] 10Phabricator, 10Release-Engineering-Team (Kanban): Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10RhinosF1) @mmodell, What's going on? Can you advise on workarounds? [09:40:10] RECOVERY - SSH on integration-slave-docker-1053 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u8 (protocol 2.0) [10:03:09] 10Phabricator, 10Release-Engineering-Team (Kanban): Unhandled exception error on Phabricator Calendar - https://phabricator.wikimedia.org/T220241 (10Aklapper) >>! In T220241#5090165, @RhinosF1 wrote: > What's going on? [A software bug.](https://en.wikipedia.org/wiki/Software_bug) > Can you advise on workaround... [10:04:10] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10Aklapper) [10:46:52] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10Aklapper) @RhinosF1: I don't know which problem "What's going on?" questions solve, apart from slowing down investigation. If the exact reason... [11:54:10] (03Abandoned) 10Zoranzoki21: Sorted emails at whitelist and added second mail of Zoranzoki21 [integration/config] - 10https://gerrit.wikimedia.org/r/500183 (owner: 10Zoranzoki21) [13:18:31] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:29:59] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) @Aklapper, I was asking as when originally reporting on IRC I was advised to report here even though we were unsure whether this was... [16:11:14] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10Aklapper) @RhinosF1: https://www.mediawiki.org/wiki/How_to_report_a_bug applies as usual. Whether it's an upstream issue is irrelevant; someon... [16:38:28] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) >>! In T220241#5090464, @Aklapper wrote: > @RhinosF1: https://www.mediawiki.org/wiki/How_to_report_a_bug applies as usual. Whether i... [16:43:02] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) This is now fixed. Has anything changed recently? [16:43:25] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) 05Open→03Resolved [16:55:28] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10Aklapper) 05Resolved→03Open Heh. Then I can only ask you to please be more patient here. :) It's not fixed for me. Reopening. [17:03:07] 10Phabricator, 10Release-Engineering-Team (Kanban): Calendar shows "Unhandled exception error: Expected a single result!" - https://phabricator.wikimedia.org/T220241 (10RhinosF1) >>! In T220241#5090486, @Aklapper wrote: > Heh. Then I can only ask you to please be more patient here. :) I will take that advice s... [18:45:47] 10Release-Engineering-Team, 10MediaWiki-Core-Testing, 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure), 10phan: phan 1.2.6 is OOMing on MediaWiki core - https://phabricator.wikimedia.org/T219114 (10Daimona) FYI, phan 1.2.8 was released just today and it includes Lego's fix for --pr... [19:29:05] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Database, 10Quibble, and 2 others: Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10Umherirrender) [19:31:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Database, 10Quibble, and 2 others: Enable MariaDB/MySQL strict mode on CI db hosts - https://phabricator.wikimedia.org/T119371 (10Umherirrender) [19:32:05] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Infrastructure, 10DBA, 10MediaWiki-Database, and 2 others: Enable MariaDB/MySQL's Strict Mode - https://phabricator.wikimedia.org/T108255 (10Umherirrender) [19:57:34] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10MediaWiki-Database, 10Quibble, and 2 others: Enable MariaDB/MySQL strict mode for MW during development and Jenkins jobs - https://phabricator.wikimedia.org/T119371 (10Krinkle) [20:01:08] Krinkle: Your aid on https://gerrit.wikimedia.org/r/c/integration/config/+/501687/ would be wonderful. ;-) [20:03:26] (03CR) 10Krinkle: docker: Provide a -deploy-node10-docker job template (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [20:04:19] James_F: I don't really know the *oid jobs functionality furhter, so I'd recommend letting services/CPT review it further. I think Marko knows them fairly well [20:05:11] It's just the over-ride to point at the different entry point. [20:11:55] 10Diffusion, 10Phabricator: Commits being made in random users names (bad associations) - https://phabricator.wikimedia.org/T77959 (10Aklapper) @mmodell: Should slightly related non-public T195061 also be closed? [20:17:52] 10MediaWiki-Codesniffer: Add/enable rule for spaces before array indices - https://phabricator.wikimedia.org/T154103 (10Krinkle) p:05Triage→03Normal [20:18:43] 10MediaWiki-Codesniffer: Avoid use of isset( ..., ..., [...] ) function construct / syntax - https://phabricator.wikimedia.org/T219139 (10Krinkle) p:05Triage→03Normal [20:34:58] 10Diffusion, 10Phabricator: Commits being made in random users names (bad associations) - https://phabricator.wikimedia.org/T77959 (10mmodell) >>! In T77959#5090682, @Aklapper wrote: > @mmodell: Should slightly related non-public T195061 also be closed? {icon check-square-o color=green} done [22:26:47] 10Release-Engineering-Team (Backlog), 10Zuul, 10Patch-For-Review: Convert zuul to use scap - https://phabricator.wikimedia.org/T215458 (10greg) [22:26:54] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Operations, 10serviceops: Deploy multi-site plugin to cobalt and gerrit2001 - https://phabricator.wikimedia.org/T217174 (10greg) [22:27:08] 10Release-Engineering-Team (Backlog), 10User-Zppix: Jenkins is posting duplicate comments when a depended on change merge fails - https://phabricator.wikimedia.org/T215511 (10greg) [22:27:19] 10Continuous-Integration-Config, 10Gerrit, 10Release-Engineering-Team (Backlog), 10Developer Productivity, 10Release Pipeline: Jenkins Publisher to Comment on Patchset - https://phabricator.wikimedia.org/T215322 (10greg) [22:27:21] 10Diffusion, 10Release-Engineering-Team (Backlog), 10GitHub-Mirrors, 10Scoring-platform-team: LFS objects are not mirroring from Github through Phab to Gerrit consistently - https://phabricator.wikimedia.org/T212818 (10greg) [22:50:57] (03PS2) 10Jforrester: docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 [22:52:00] (03PS3) 10Jforrester: docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 [22:52:08] (03CR) 10Jforrester: docker: Provide a -deploy-node10-docker job template (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [22:52:38] (03PS4) 10Jforrester: docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 [22:53:33] (03CR) 10jerkins-bot: [V: 04-1] docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 (owner: 10Jforrester) [22:56:08] (03PS5) 10Jforrester: docker: Provide a -deploy-node10-docker job template [integration/config] - 10https://gerrit.wikimedia.org/r/501687 [23:00:32] (03PS2) 10Jforrester: [MobileApps/Deploy] Add deploy-node10-docker job as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/501688 [23:00:39] (03PS6) 10Jforrester: [MoblileApps] Move from npm-docker to node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/478762 (https://phabricator.wikimedia.org/T211785) [23:00:45] (03PS2) 10Jforrester: [MobileApps/Deploy] Use node10-docker job, drop node6 ones [integration/config] - 10https://gerrit.wikimedia.org/r/501689 (https://phabricator.wikimedia.org/T211785) [23:02:34] (03CR) 10jerkins-bot: [V: 04-1] [MoblileApps] Move from npm-docker to node10-docker [integration/config] - 10https://gerrit.wikimedia.org/r/478762 (https://phabricator.wikimedia.org/T211785) (owner: 10Jforrester) [23:02:43] (03CR) 10jerkins-bot: [V: 04-1] [MobileApps/Deploy] Add deploy-node10-docker job as experimental [integration/config] - 10https://gerrit.wikimedia.org/r/501688 (owner: 10Jforrester) [23:03:09] (03CR) 10jerkins-bot: [V: 04-1] [MobileApps/Deploy] Use node10-docker job, drop node6 ones [integration/config] - 10https://gerrit.wikimedia.org/r/501689 (https://phabricator.wikimedia.org/T211785) (owner: 10Jforrester)