[04:40:33] 10Release-Engineering-Team (Code Health), 10Release-Engineering-Team-TODO (201908), 10Code-Stewardship-Reviews, 10Graphoid, and 3 others: graphoid: Code stewardship request - https://phabricator.wikimedia.org/T211881 (10Bawolff) >>! In T211881#5607568, @dr0ptp4kt wrote: > @Yair_rand agreed the relatively l... [06:09:39] * bawolff came here to complain about packagist, but i see its already in the topic [06:40:42] 10Phabricator, 10Performance Issue: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10DannyS712) Just looking manually at enabled global rules that target tasks, the following rules appear to all do the same thing for different people (i.e. add the ow... [06:54:35] 10Phabricator, 10User-DannyS712: Combine H32 and H194 - https://phabricator.wikimedia.org/T236647 (10DannyS712) [06:55:24] 10Phabricator, 10User-DannyS712: Combine H32 and H194 - https://phabricator.wikimedia.org/T236647 (10DannyS712) See also {T108586} [06:58:32] 10Phabricator, 10User-DannyS712: Combine H309 and H310 - https://phabricator.wikimedia.org/T236648 (10DannyS712) [06:59:12] 10Phabricator, 10User-DannyS712: Combine H309 and H310 - https://phabricator.wikimedia.org/T236648 (10DannyS712) See also {T108586} [06:59:25] (03PS1) 10Thiemo Kreuz (WMDE): Use severity instead of excludes to allow local overrides [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546392 [07:04:04] (03CR) 10jerkins-bot: [V: 04-1] Use severity instead of excludes to allow local overrides [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546392 (owner: 10Thiemo Kreuz (WMDE)) [08:06:10] 10Project-Admins, 10WMF-Legal: Fix empty project description of #WMF-Legal (and/or remove join/edit restrictions) - https://phabricator.wikimedia.org/T231810 (10Aklapper) [08:47:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs failing with Composer TransportException: 404 Not Found (October... - https://phabricator.wikimedia.org/T236513 [08:48:34] Morning, hashar. [08:49:17] hi [08:49:22] I am jetlagged :-\ [08:52:32] It [08:52:35] Bah. [08:52:42] It'll be worse in two weeks' time. [08:55:58] "stderr: 'fatal: unable to access 'https://gerrit.wikimedia.org/r/mediawiki/extensions/Interwiki/': The requested URL returned error: 502'" [08:56:04] What fun. [09:29:39] James_F: gerrit1001 should have the server side error for that one ? :D [10:02:03] (03PS23) 10Awight: Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [10:02:05] (03PS9) 10Awight: Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 [10:02:07] (03PS5) 10Awight: Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 [10:02:09] (03PS5) 10Awight: [WIP] Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) [10:02:11] (03PS1) 10Awight: Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 [10:02:13] (03PS1) 10Awight: [WIP] Docs for command.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/546435 [10:03:37] (03CR) 10jerkins-bot: [V: 04-1] Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 (owner: 10Awight) [10:03:43] (03CR) 10jerkins-bot: [V: 04-1] Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 (owner: 10Awight) [10:03:48] (03CR) 10jerkins-bot: [V: 04-1] Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [10:03:57] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) (owner: 10Awight) [10:04:02] (03CR) 10jerkins-bot: [V: 04-1] Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 (owner: 10Awight) [10:04:14] (03CR) 10jerkins-bot: [V: 04-1] [WIP] Docs for command.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/546435 (owner: 10Awight) [10:07:36] (03Abandoned) 10Awight: [WIP] Docs for command.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/546435 (owner: 10Awight) [10:14:04] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Global developer for DannyS712 on beta cluster - https://phabricator.wikimedia.org/T235650 (10DannyS712) Going based off of {T228352}, @greg @Jdforrester-WMF can you take a look at this? [10:21:15] (03PS24) 10Awight: Commands can expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [10:22:00] (03PS2) 10Awight: Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 [10:22:28] (03PS10) 10Awight: Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 [10:22:56] (03PS6) 10Awight: Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 [10:23:22] (03PS6) 10Awight: [WIP] Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) [10:35:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MW-1.35-notes (1.35.0-wmf.4; 2019-10-29), 10Patch-For-Review, 10Wikimedia-production-error (Shared Build Failure): Jenkins jobs failing with Composer TransportException: 404 Not Found (October... - https://phabricator.wikimedia.org/T236513 [10:41:52] (03PS7) 10Awight: Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) [10:43:42] (03CR) 10Awight: "> Perhaps as first step, the install step can be parallelised. And" [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) (owner: 10Awight) [10:44:54] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10hashar) [10:45:48] 10Release-Engineering-Team, 10Discovery-Wikidata-Query-Service-Sprint: Give +2 access to the Blazegraph repo to the WDQS dev team - https://phabricator.wikimedia.org/T236670 (10Gehel) [10:46:09] hashar: I'm lost in gerrit permissions ^ [10:46:18] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO, 10Operations, 10Release Pipeline, and 3 others: Migrate production services to kubernetes using the pipeline - https://phabricator.wikimedia.org/T198901 (10Ladsgroup) [10:46:35] and I'm lost enough that I don't really know what question to ask. If you have a few minutes to have a look, I'd be greatful! [10:46:44] grateful [10:46:59] (but I can also be full of greatness if needed) [10:52:19] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10hashar) [10:54:59] 10Release-Engineering-Team, 10Cloud-Services, 10Operations, 10puppet-compiler, 10Puppet: labstore1006.wikimedia.org failing to compile on compiler1003 - https://phabricator.wikimedia.org/T236672 (10jbond) [10:55:54] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Global developer for DannyS712 on beta cluster - https://phabricator.wikimedia.org/T235650 (10Jdforrester-WMF) This is just the Beta Cluster equivalent of `+sysadmin` on production, right? (Why are they named differently?) [10:56:59] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10hashar) Snapshot from https://integration.wikimedia.o... [11:00:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Test-Coverage: mwcore-phpunit-coverage-master times out after 5 hours - https://phabricator.wikimedia.org/T232706 (10hashar) It is constantly faster on the Jessie instanc... [11:04:31] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10serviceops, 10Test-Coverage: Upgrade our php-xdebug package for php7.2 - https://phabricator.wikimedia.org/T234418 (10hashar) That is for the CI Docker containers running php7.2. It seems the production releases... [11:05:03] (03PS1) 10Zfilipin: macOS: minikube has moved from homebrew cask to homebrew [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546446 [11:08:36] (03PS2) 10Zfilipin: macOS: minikube has moved from homebrew cask to homebrew [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546446 [11:09:06] (03PS3) 10Zfilipin: macOS: minikube has moved from homebrew cask to homebrew [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546446 [11:13:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675 (10hashar) [11:14:37] gehel: hi, what are you lost in (perms wise) [11:14:39] 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Cloud-VPS (Debian Jessie Deprecation): "integration" Cloud VPS project jessie deprecation - https://phabricator.wikimedia.org/T236576 (10hashar) [11:21:01] paladox: not sure why David, and myself don't have +2 on wikidata/query/blazegraph, but do on wikidata/query/rdf [11:21:05] (03CR) 1020after4: [C: 03+1] macOS: minikube has moved from homebrew cask to homebrew [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546446 (owner: 10Zfilipin) [11:21:50] paladox: it looks like there is some mess in how we manage group membership in that project, but we should probably have someone who understand the permission system to do the cleanup / strategy [11:23:44] * paladox checks [11:29:27] gehel: it’s because of https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/blazegraph,access . The owner is ‘wikidata-query. The only user in that group is Smalyshev, the group is also owned by the admins. [11:30:35] paladox: I'm probably missing something, but I had the impression that the rdf project is under the same group [11:32:27] Just looked at that one, yeh [11:32:28] https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/rdf,access [11:34:28] That’s wierd [11:34:29] I carn’t find you in either of the groups [11:34:39] So I have no idea how you’ll gaining +2 in rdf [11:36:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201909), 10Cloud-VPS, 10cloud-services-team: Investigate Stretch VM being slower than Jessie VM - https://phabricator.wikimedia.org/T236675 (10hashar) Another job that shows the d... [11:37:09] gehel: paladox eek sorry I have missed the irc notification [11:37:21] I think I recently tweaked permissions for some related project [11:37:45] 10:35 gerrit: added onimisionipe and gehel to wikidata-query-gui groups # T235159 [11:37:45] T235159: Enable write access for Mathew.onipe(onimisionipe) and gehel on wikidata gui repo - https://phabricator.wikimedia.org/T235159 [11:37:46] :D [11:37:46] hashar: I navigated up the project, but cannot find gehel in any of the groups [11:38:11] I did this commit https://gerrit.wikimedia.org/r/plugins/gitiles/wikidata/query/rdf/+/37017fc86f4a7a88ac0f29fd9bd354e4453053c7%5E%21/#F1 [11:38:19] hmm [11:38:23] 10Release-Engineering-Team, 10Discovery-Wikidata-Query-Service-Sprint: Give +2 access to the Blazegraph repo to the WDQS dev team - https://phabricator.wikimedia.org/T236670 (10hashar) Note that previously I have done: 2019-10-10 10:35 gerrit: added onimisionipe and gehel to wikidata-query-gui grou... [11:38:28] maybe that has been lost as part of the Gerrit migration :-\\\ [11:38:52] Hmm [11:39:01] oh [11:39:05] no different group bah [11:39:11] hashar: the ui shows correctly [11:39:33] But some how gehel has +2 in rdf but not in blazegraph [11:40:40] hmm [11:40:55] so blazegraph is owned by the group wikidata-query [11:40:58] which only has Stas [11:41:21] the rdf one is also owned by the 'qa' group https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/rdf,access [11:41:32] and that qa group includes ldap/wmf https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/rdf,access [11:41:53] Oh!! [11:42:03] and gehel is in the wmf ldap group [11:42:09] and thus he is granted access to rdf ;] [11:43:04] Gerrits ui I feel has bugs... as I can see ldap/wmf on projects, but in groups I cannot see it [11:43:52] Yup. [11:43:58] Thought that was known? [11:44:12] hashar: I guess we ca either add qa to blazegraph or add them to the wikidata-query group [11:44:14] 10Release-Engineering-Team, 10Discovery-Wikidata-Query-Service-Sprint: Give +2 access to the Blazegraph repo to the WDQS dev team - https://phabricator.wikimedia.org/T236670 (10hashar) The blazegraph repository is owned by the `wikidata-query` group which only has Stas https://gerrit.wikimedia.org/r/#/admin/gr... [11:44:19] just drop qa [11:44:49] James_F: well showing ldap groups in polygerrit ui is known, I fixed that and backported the fix [11:45:00] Still broken for us, isn't it? [11:45:04] This issue is in GWTUI [11:45:17] It seems to make sense to me that the wdqs team should be in the wikidata-query group [11:45:27] Oh, right. [11:45:29] James_F: you still cannot see the ldap groups? [11:45:35] the problem is that we have projects inheriting from each others [11:45:41] and groups inheritng from each other as well [11:45:54] so there are multiple dimensions permissions inheritance, that makes stuff hards to track / follow :-\\\ [11:46:04] paladox: No, I see them now. Thanks for fixing that. [11:46:17] Ah, ok your welcome :) [11:46:25] and LDFServer is also owned by ldap/wmf bah [11:46:25] https://gerrit.wikimedia.org/r/#/admin/projects/wikidata/query/LDFServer,access [11:47:59] Likley also happens in polygerrit too. If I do not have the perms to see the group on the backend then Gerrits groups simply reject me viewing/adding it. [11:48:08] Where as projects allow me... [11:48:17] We should have a consistency report [11:48:17] That’s some wierd behaviour:P [11:50:51] Maybe we should merge those groups [11:50:56] 10Release-Engineering-Team, 10Discovery-Wikidata-Query-Service-Sprint: Give +2 access to the Blazegraph repo to the WDQS dev team - https://phabricator.wikimedia.org/T236670 (10hashar) All those repositories inherit rights from the meta project `wikidata/query.git`: https://gerrit.wikimedia.org/r/#/admin/proje... [11:50:58] So that only one group is used [11:51:25] there is a task about auditing the gerrit groups and access list [11:51:32] but it is hmm not that trivial [11:51:36] too many layers [11:51:50] Which task? [11:51:52] potentially what we could do is use another tool to generate the rights explicitly [11:52:00] and drop the inheritance magic [11:52:25] 10Phabricator, 10Performance Issue: Herald rules causing delays to task edit saves - getting worse - https://phabricator.wikimedia.org/T108586 (10mmodell) >>! In T108586#5610008, @DannyS712 wrote: > Is there a more efficient way to be doing this? > I could probably build a custom action that's something lik... [11:53:04] paladox: https://phabricator.wikimedia.org/T219006 [11:53:18] I think you have access to it (at least you are subscribing to the task) [11:53:27] Ah! [11:53:31] Yup [11:53:38] and https://phabricator.wikimedia.org/T218805 [11:53:40] for the groups [11:53:58] I get a perms error for that one :( [11:54:07] the thing is that it has grown has a bazaar [11:54:52] paladox: I have added you to https://phabricator.wikimedia.org/T218805 [11:55:23] Thanks! [12:04:58] hashar: https://phabricator.wikimedia.org/T218805#5611083 [12:10:20] (03CR) 10Jforrester: [V: 03+2 C: 03+2] macOS: minikube has moved from homebrew cask to homebrew [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546446 (owner: 10Zfilipin) [12:29:39] 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10local-charts, 10User-zeljkofilipin: `make install` fails with `Error: caskroom/cask was moved. Tap homebrew/cask-cask instead.` - https://phabricator.wikimedia.org/T236393 (10zeljkofilipin) 05Open→03Resolved [12:30:54] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Growth-Team, 10GrowthExperiments, and 3 others: Refactor GrowthExperiments Homepage Selenium test - https://phabricator.wikimedia.org/T236620 (10zeljkofilipin) [12:31:18] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10Growth-Team, 10GrowthExperiments, and 3 others: Refactor GrowthExperiments Homepage Selenium test - https://phabricator.wikimedia.org/T236620 (10zeljkofilipin) p:05Triage→03Normal [12:38:12] (03PS1) 10Jforrester: Start branching DiscussionTools for wmf/ [tools/release] - 10https://gerrit.wikimedia.org/r/546462 (https://phabricator.wikimedia.org/T235923) [12:38:59] (03CR) 10Jforrester: [C: 03+2] Start branching DiscussionTools for wmf/ [tools/release] - 10https://gerrit.wikimedia.org/r/546462 (https://phabricator.wikimedia.org/T235923) (owner: 10Jforrester) [12:39:28] (03Merged) 10jenkins-bot: Start branching DiscussionTools for wmf/ [tools/release] - 10https://gerrit.wikimedia.org/r/546462 (https://phabricator.wikimedia.org/T235923) (owner: 10Jforrester) [12:48:44] 10Continuous-Integration-Config, 10Core Platform Team: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10daniel) [12:49:09] 10Continuous-Integration-Config, 10Core Platform Team, 10Code-Health: Enable API integration tests in CI for MediaWiki core - https://phabricator.wikimedia.org/T236680 (10daniel) [12:58:24] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) The account had been deactivated as pa... [12:58:30] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO (201910), 10Developer-Advocacy, 10wikimedia.biterg.io: biterg.io Gerrit crawling probably stresses the server too much - https://phabricator.wikimedia.org/T234328 (10hashar) p:05Triage→03Normal [13:06:42] (03CR) 10Hashar: Update README.md (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/546157 (owner: 10Hashar) [13:07:49] (03PS2) 10Hashar: Update README.md [integration/config] - 10https://gerrit.wikimedia.org/r/546157 [13:11:40] 10Deployments, 10Release-Engineering-Team, 10VisualEditor, 10Wikimedia-Logstash, and 5 others: Logstash discards messages from MediaWiki if they contain uncommon keys in the $context array - https://phabricator.wikimedia.org/T234564 (10fgiunchedi) 05Open→03Resolved a:03fgiunchedi Resolving as logs ar... [13:11:49] (03PS25) 10Awight: Commands expand recursively [integration/quibble] - 10https://gerrit.wikimedia.org/r/519776 [13:11:50] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Jenkins: Migrate JJB usage --config-xml - https://phabricator.wikimedia.org/T236685 (10hashar) [13:11:53] (03PS3) 10Awight: Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 [13:11:55] (03PS11) 10Awight: Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 [13:11:57] (03PS7) 10Awight: Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 [13:11:59] (03PS8) 10Awight: Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) [13:12:01] (03PS1) 10Awight: Cleanup: require all command parameter keywords [integration/quibble] - 10https://gerrit.wikimedia.org/r/546469 [13:13:30] (03CR) 10jerkins-bot: [V: 04-1] Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 (owner: 10Awight) [13:13:37] (03CR) 10jerkins-bot: [V: 04-1] Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 (owner: 10Awight) [13:13:39] nooo [13:13:48] (03CR) 10jerkins-bot: [V: 04-1] Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 (owner: 10Awight) [13:17:38] (03PS1) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [13:18:19] (03PS2) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [13:21:27] (03PS2) 10Awight: Cleanup: require all command parameter keywords [integration/quibble] - 10https://gerrit.wikimedia.org/r/546469 [13:21:29] (03PS4) 10Awight: Merge utility functions into calling modules [integration/quibble] - 10https://gerrit.wikimedia.org/r/546434 [13:21:33] (03PS12) 10Awight: Clean up logging [integration/quibble] - 10https://gerrit.wikimedia.org/r/546126 [13:21:35] (03PS8) 10Awight: Replace argparse with docopt [integration/quibble] - 10https://gerrit.wikimedia.org/r/546167 [13:21:37] (03PS9) 10Awight: Prepare to run browser suites in parallel [integration/quibble] - 10https://gerrit.wikimedia.org/r/545661 (https://phabricator.wikimedia.org/T226869) [13:24:03] (03CR) 10Hashar: [C: 04-1] "Spams a bunch of messages such as:" [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:34:28] (03PS3) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [13:35:53] (03CR) 10Jforrester: [C: 03+1] "Let's land that in another patch, then?" [integration/config] - 10https://gerrit.wikimedia.org/r/546157 (owner: 10Hashar) [13:36:27] (03CR) 10Hashar: "Yup https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/ and more to follow :]" [integration/config] - 10https://gerrit.wikimedia.org/r/546157 (owner: 10Hashar) [13:37:04] (03CR) 10Jforrester: "(This should add --config-xml to README.md.)" [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:39:08] (03CR) 10jerkins-bot: [V: 04-1] Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:40:50] I will one day have to figure out why jjb is so slow [13:41:06] (03PS1) 10simetrical: Allow @testWith and @doesNotPerformAssertions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546476 [13:42:40] (03PS1) 10Hashar: utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 [13:47:52] (03CR) 10Jforrester: [C: 03+1] "Good to deploy." [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:48:01] :] [13:48:05] updating readme.md [13:50:43] (03CR) 10Hashar: "> (This should add --config-xml to README.md.)" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:51:05] (03PS4) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [13:51:08] (03PS2) 10Hashar: utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 [13:51:09] (03PS1) 10Hashar: Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) [13:52:37] James_F: maybe I should have done everything in a single change :/ [13:52:48] anyway I guess the README.md one can be +2ed https://gerrit.wikimedia.org/r/#/c/integration/config/+/546157/ [13:53:13] then I update integration-config-jjb-diff-docker to use https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/ [13:53:17] which can then be +2ed :] [13:53:51] (03CR) 10jerkins-bot: [V: 04-1] Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:54:27] (03CR) 10jerkins-bot: [V: 04-1] Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:54:37] (03CR) 10Jforrester: [C: 03+2] Update README.md [integration/config] - 10https://gerrit.wikimedia.org/r/546157 (owner: 10Hashar) [13:55:12] Yeah. [13:55:14] Go for it. [13:55:28] Or I can. ;-) [13:55:35] command is ready :D [13:55:40] waiting for the change to merge [13:56:17] (03Merged) 10jenkins-bot: Update README.md [integration/config] - 10https://gerrit.wikimedia.org/r/546157 (owner: 10Hashar) [13:56:36] (03CR) 10Hashar: "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:56:40] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:57:02] (03PS5) 10Jforrester: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [13:57:15] Gerrit is way too pessimistic about merge conflicts. [13:57:20] (03PS6) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [13:57:22] (03PS3) 10Hashar: utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 [13:57:24] (03PS2) 10Hashar: Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) [13:57:33] ;D [13:59:16] (03CR) 10jerkins-bot: [V: 04-1] Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:00:10] James_F: https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/ good for a +2 :] [14:00:39] with https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/6/utils/jjb-diff.sh [14:00:45] having the magic of using diff --recursive [14:00:49] ;D [14:02:09] oh [14:02:13] gotta update zuul/layout as well [14:03:23] (03PS7) 10Hashar: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) [14:04:48] Do you want to merge? ;-) [14:06:24] yeah should be good [14:06:52] that one https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/ [14:06:53] :) [14:08:17] (03CR) 10simetrical: "References:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546476 (owner: 10simetrical) [14:09:48] (03CR) 10Jforrester: [C: 03+2] Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:11:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10hashar) [14:12:35] 10Release-Engineering-Team, 10Cloud-Services, 10Operations, 10puppet-compiler, and 2 others: labstore1006.wikimedia.org failing to compile on compiler1003 - https://phabricator.wikimedia.org/T236672 (10jbond) 05Open→03Resolved Think this is resolved now, please reopen if there are still issues [14:12:40] 10Release-Engineering-Team, 10Operations, 10puppet-compiler, 10Patch-For-Review, 10Puppet: add compiler1003 to jenkins - https://phabricator.wikimedia.org/T236468 (10jbond) [14:12:50] !log Reloading Zuul for https://gerrit.wikimedia.org/r/#/c/integration/config/+/546470/ [14:12:54] (03Merged) 10jenkins-bot: Move jjb/diff.sh to utils/jjb-diff.sh [integration/config] - 10https://gerrit.wikimedia.org/r/546470 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:13:05] (03PS4) 10Hashar: utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 [14:13:06] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:13:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Patch-For-Review: Migrate JJB usage --config-xml - https://phabricator.wikimedia.org/T236685 (10Jdforrester-WMF) [14:13:33] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Patch-For-Review: Migrate JJB usage --config-xml - https://phabricator.wikimedia.org/T236685 (10Jdforrester-WMF) a:03hashar [14:16:20] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [14:21:20] (03CR) 10Jforrester: [C: 03+1] utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 (owner: 10Hashar) [14:21:28] (03PS3) 10Jforrester: Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:22:36] 10Release-Engineering-Team, 10Discovery-Wikidata-Query-Service-Sprint: Give +2 access to the Blazegraph repo to the WDQS dev team - https://phabricator.wikimedia.org/T236670 (10Gehel) 05Open→03Resolved a:03Gehel >>! In T236670#5611037, @hashar wrote: > All those repositories inherit rights from the meta... [14:22:52] hashar, paladox: ^ thanks for cleaning this up! [14:23:15] (03CR) 10jerkins-bot: [V: 04-1] Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:23:20] hashar: https://gerrit.wikimedia.org/r/monitoring?part=graph&graph=activeThreads&period=semaine <3 [14:23:22] hashar did the work! [14:25:12] (03CR) 10Hashar: [C: 03+2] utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 (owner: 10Hashar) [14:25:21] (03PS4) 10Hashar: Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) [14:25:31] 10Project-Admins, 10Research: create tags for Research - https://phabricator.wikimedia.org/T228350 (10Aklapper) a:03Aklapper Alright: * I created `#foundational` as a subproject - @leila: Please add a description to `#foundational` at https://phabricator.wikimedia.org/project/edit/4350/ * I created four quar... [14:25:36] (03CR) 10Hashar: [C: 03+2] "had a flake8 issue" [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:27:02] hashar: Also, did you see https://gerrit.wikimedia.org/r/c/integration/config/+/546264 ? [14:27:47] (03Merged) 10jenkins-bot: utils/jjb-diff.sh do relative diff [integration/config] - 10https://gerrit.wikimedia.org/r/546478 (owner: 10Hashar) [14:27:52] (03Merged) 10jenkins-bot: Convert test to use jjb --config-xml [integration/config] - 10https://gerrit.wikimedia.org/r/546480 (https://phabricator.wikimedia.org/T236685) (owner: 10Hashar) [14:32:28] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Patch-For-Review: Migrate JJB usage --config-xml - https://phabricator.wikimedia.org/T236685 (10hashar) 05Open→03Resolved Thank you James [14:38:08] (03PS8) 10Jforrester: layout: [SmashPig] Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) [14:38:12] (03PS6) 10Jforrester: layout: [integration/docroot] Switch from php56 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/516570 [14:38:14] (03PS10) 10Jforrester: jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [14:38:16] (03PS10) 10Jforrester: dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [14:38:47] (03PS9) 10Jforrester: layout: [SmashPig] Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) [14:38:49] (03PS7) 10Jforrester: layout: [integration/docroot] Switch from php56 to php72 [integration/config] - 10https://gerrit.wikimedia.org/r/516570 [14:38:51] (03PS11) 10Jforrester: jjb: Drop all PHP56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514058 (https://phabricator.wikimedia.org/T224906) [14:38:53] (03PS11) 10Jforrester: dockerfiles: Drop all PHP56 containers [integration/config] - 10https://gerrit.wikimedia.org/r/514059 (https://phabricator.wikimedia.org/T224906) [14:40:12] 10Project-Admins, 10Developer-Advocacy (Oct-Dec 2019): Sort out #Editing-Team vs #VisualEditor Phabricator project tags - https://phabricator.wikimedia.org/T234648 (10Aklapper) [14:40:14] (03CR) 10Hashar: "Quibble 0.0.40 has not been cut :D" [integration/config] - 10https://gerrit.wikimedia.org/r/546264 (https://phabricator.wikimedia.org/T236333) (owner: 10Brennen Bearnes) [14:40:20] James_F: yeah I will deal with it [14:40:22] 10Project-Admins, 10Editing-team, 10Developer-Advocacy (Oct-Dec 2019): Sort out #Editing-Team vs #VisualEditor Phabricator project tags - https://phabricator.wikimedia.org/T234648 (10JTannerWMF) Just adding the Editing Team tag here so we can keep an eye on the progress. [14:41:43] (03PS1) 10Hashar: Fix quibble-stretch-php72 version [integration/config] - 10https://gerrit.wikimedia.org/r/546505 [14:41:52] (03CR) 10Jforrester: [C: 03+2] "FR-tech said they're not running PHP56 for anything any more." [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) (owner: 10Jforrester) [14:42:31] hashar: Also, https://gerrit.wikimedia.org/r/c/integration/config/+/516570 is now the only use of PHP56. :-( We really need to upgrade contint1001. [14:42:36] (03PS2) 10Hashar: mediawiki api testing: bump quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/546264 (https://phabricator.wikimedia.org/T236333) (owner: 10Brennen Bearnes) [14:42:40] yeah :-\ [14:43:02] there is so much infra work needed for the jessie > stretch migration :-\\ [14:43:30] * James_F nods. [14:43:35] Are we not going straight to buster? [14:43:38] and https://gerrit.wikimedia.org/r/#/c/integration/config/+/546505/ fix the container version [14:43:39] (03Merged) 10jenkins-bot: layout: [SmashPig] Drop php56 jobs [integration/config] - 10https://gerrit.wikimedia.org/r/514055 (https://phabricator.wikimedia.org/T224906) (owner: 10Jforrester) [14:43:43] ah yeah jessie > buster [14:43:49] well whatever is needed to phase out jessie :] [14:43:53] Yeah. [14:44:39] there is a even a subproject/milestone in phabricator for it [14:45:00] (03CR) 10Hashar: [C: 03+2] Fix quibble-stretch-php72 version [integration/config] - 10https://gerrit.wikimedia.org/r/546505 (owner: 10Hashar) [14:45:13] !log Zuul: Drop PHP56 jobs for SmashPig T224906 [14:45:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:45:26] T224906: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 [14:46:17] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Drop php56 testing support - https://phabricator.wikimedia.org/T224906 (10Jdforrester-WMF) Everything except T224591 itself is now done. [14:46:41] (03Merged) 10jenkins-bot: Fix quibble-stretch-php72 version [integration/config] - 10https://gerrit.wikimedia.org/r/546505 (owner: 10Hashar) [14:46:52] will do that^ [14:46:57] Kk. [14:47:01] * James_F stands back. [14:47:08] !log contint1001: docker image prune [14:47:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:47:12] I should cron that [14:49:08] !log contint1001 for https://gerrit.wikimedia.org/r/#/c/integration/config/+/546505/ sudo docker tag docker-registry.discovery.wmnet/releng/quibble-stretch-php72:0.0.40 docker-registry.discovery.wmnet/releng/quibble-stretch-php72:0.0.39-s2 [14:49:10] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:49:24] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [14:49:24] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [14:55:38] 10Beta-Cluster-Infrastructure, 10User-DannyS712: Global developer for DannyS712 on beta cluster - https://phabricator.wikimedia.org/T235650 (10DannyS712) >>! In T235650#5610904, @Jdforrester-WMF wrote: > This is just the Beta Cluster equivalent of `+sysadmin` on production, right? (Why are they named different... [14:57:20] !log Updating mediawiki-quibble-api-testing-vendor-mysql-php72-docker mediawiki-quibble-api-testing-vendor-sqlite-php72-docker # T236333 [14:57:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:57:24] T236333: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 [14:57:33] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 2" [integration/config] - 10https://gerrit.wikimedia.org/r/546264 (https://phabricator.wikimedia.org/T236333) (owner: 10Brennen Bearnes) [14:59:51] 14:59:37 Current existing ChromeDriver binary is unavailable, proceding with download and extraction. [14:59:51] 14:59:37 Downloading from file: https://chromedriver.storage.googleapis.com/73.0.3683.20/chromedriver_linux64.zip [14:59:52] ... [15:00:09] (03Merged) 10jenkins-bot: mediawiki api testing: bump quibble-stretch-php72 [integration/config] - 10https://gerrit.wikimedia.org/r/546264 (https://phabricator.wikimedia.org/T236333) (owner: 10Brennen Bearnes) [15:01:24] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Quibble, 10Patch-For-Review: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10hashar) Shoul... [15:01:34] James_F: api-testing to use wikidiff2 [DONE] (poke brennen ) [15:08:16] ugh, morning. [15:08:19] thanks hashar. [15:08:23] :] [15:08:29] to be fair [15:08:43] there is nothing enforcing the dockerfiles image to match the quibble version ;D [15:11:33] (03PS1) 10Pwirth: layout, parameter_functions: Add BlueSpiceUserInfo [integration/config] - 10https://gerrit.wikimedia.org/r/546511 [15:17:57] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10hashar) I went to Fabric website ( https://get.fabric.io/ ) which states: > Fabr... [15:19:59] (03PS1) 10Hashar: Tox env for Fabric + doc [integration/config] - 10https://gerrit.wikimedia.org/r/546513 (https://phabricator.wikimedia.org/T236689) [15:21:07] tldr [15:21:18] we most probably have to transition out of Fabric ;D [15:22:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10hashar) p:05Triage→03Normal [15:32:49] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO, 10Patch-For-Review: Upgrade integration/config to use Fabric 2.x / python3 - https://phabricator.wikimedia.org/T236689 (10Jdforrester-WMF) 😱 [15:33:21] PROBLEM - Free space - all mounts on integration-agent-docker-1005 is CRITICAL: CRITICAL: integration.integration-agent-docker-1005.diskspace._var_lib_docker.byte_percentfree (<11.11%) [15:38:41] hello everybody [15:38:55] it seems that git-sync is broken on deployment-deploy03 [15:39:56] the '[LOCAL HACK] Attempt to secure Puppet DB better' seems causing problems when rebasing [15:40:12] can we remove it ? [15:40:16] no idea who is the owner [15:41:07] Krenair: --^ :) [15:44:54] (03CR) 10Thcipriani: "this one was all me :(" [integration/config] - 10https://gerrit.wikimedia.org/r/546505 (owner: 10Hashar) [15:48:20] RECOVERY - Free space - all mounts on integration-agent-docker-1005 is OK: OK: All targets OK [15:50:33] thcipriani: no worries :] [16:05:09] (03CR) 10Jforrester: [C: 03+2] layout, parameter_functions: Add BlueSpiceUserInfo [integration/config] - 10https://gerrit.wikimedia.org/r/546511 (owner: 10Pwirth) [16:07:16] (03Merged) 10jenkins-bot: layout, parameter_functions: Add BlueSpiceUserInfo [integration/config] - 10https://gerrit.wikimedia.org/r/546511 (owner: 10Pwirth) [16:08:47] 10Continuous-Integration-Config, 10Release-Engineering-Team: quibble-selenium jobs re-downloading npm packages (castor not loading) - https://phabricator.wikimedia.org/T234738 (10Krinkle) I've worked with upstream to make it store the chromedriver binary in `XDG_CACHE_HOME` instead if in a local temp directory... [16:11:27] !log Zuul: Add BlueSpiceUserInfo [16:11:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:48:19] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins: Migrate JJB usage --config-xml - https://phabricator.wikimedia.org/T236685 (10hashar) [16:50:44] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Technical-Debt: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 05Resolved→03Open I have had an outdated fork on m... [16:56:52] (03PS1) 10Hashar: Upgrade jjb to 2.10.1 [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) [16:58:48] (03CR) 10Hashar: [C: 04-1] "Have to review the jjb-diff job output. Locally that changes operations-puppet-doc :" [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:04:36] (03PS2) 10Hashar: Upgrade jjb to 2.10.1 [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) [17:04:39] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Quibble: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10nnikkhoui) Thanks all! Tests are pa... [17:21:42] 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10Developer Productivity, 10local-charts: Ensure that local-charts installs and runs on Debian Buster - https://phabricator.wikimedia.org/T228477 (10brennen) 05Open→03Resolved a:03brennen [17:21:51] (03CR) 10Jforrester: [C: 03+1] Upgrade jjb to 2.10.1 [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:23:49] (03CR) 10Hashar: [C: 03+2] "that is where I have found out that tox does not detect the environment should be rebuild :-\" [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:24:20] (03CR) 10Hashar: [C: 03+2] "INFO:jenkins_jobs.builder:Number of jobs generated: 1" [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:26:16] (03Merged) 10jenkins-bot: Upgrade jjb to 2.10.1 [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:26:18] (03CR) 10Hashar: [C: 03+2] "That does not make the job significantly faster for some reason, but at least it is not broken :]" [integration/config] - 10https://gerrit.wikimedia.org/r/546653 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:30:22] thcipriani: I know this is terrible, but… I created a service user with a typo in the name. I'd now like to either rename or delete so I can recreate it. Is that something you can help with? Former user name is: instance-puppet-used, desired username instance-puppet-user [17:35:48] andrewbogott: I think it might be best to create a new user. I can deactivate the old user, that user won't show up anywhere/be able to login. Does that work for you? [17:36:58] thcipriani: it does as long as that latent user doesn't prevent me from creating the new one [17:37:00] we can try! [17:37:22] (03PS1) 10RLazarus: Add a new project for operations/software/httpbb. [integration/config] - 10https://gerrit.wikimedia.org/r/546661 (https://phabricator.wikimedia.org/T236699) [17:37:42] thcipriani: lmk when to try logging in again [17:38:22] (03CR) 10Giuseppe Lavagetto: [C: 03+1] Add a new project for operations/software/httpbb. [integration/config] - 10https://gerrit.wikimedia.org/r/546661 (https://phabricator.wikimedia.org/T236699) (owner: 10RLazarus) [17:38:23] thcipriani: (I'm assuming you're deactivating the user only in gerrit so that when I log in again it'll pull the record from ldap all over again but with the right name this time… I've fixed it in ldap already) [17:39:48] (03PS1) 10Hashar: Upgrade jjb to 3.0.2 [integration/config] - 10https://gerrit.wikimedia.org/r/546663 (https://phabricator.wikimedia.org/T236036) [17:40:10] (03CR) 10Hashar: "Mostly a noop :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/546663 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [17:40:21] andrewbogott it'll prevent logging in with that account & stop it from showing in the UI [17:41:57] hrm, but it looks like the cn and uid are different: one is correct and the other wasn't, so this might not work :\ [17:43:06] <_joe_> longma: thnaks for the thorough review on the deployment-charts patch [17:45:27] _joe_: Sure! They were easy to miss just scanning through the code, so trying it out on an existing chart helped [17:45:54] thcipriani: I can also just perpetuate that typo in my code, forever, if the alternative is too horrifying to ponder [17:46:09] thcipriani we could rename it All-Users side [17:46:28] Since the user hasen't created patches yet i belive. [17:47:17] (03CR) 10Brennen Bearnes: "I'm likewise still seeing this." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546317 (https://phabricator.wikimedia.org/T236471) (owner: 10Jeena Huneidi) [17:47:23] andrewbogott: heh, it's tricky, honestly. What did you have to update in ldap? the sn? or cn? [17:47:52] PROBLEM - Parsoid on deployment-parsoid09 is CRITICAL: connect to address 172.16.5.63 and port 8000: Connection refused [17:47:52] PROBLEM - Parsoid on deployment-mediawiki-parsoid10 is CRITICAL: connect to address 172.16.0.141 and port 8000: Connection refused [17:48:15] thcipriani: the dn, I'm pretty sure [17:48:26] I renamed the whole record (as opposed to changing the cn /in/ the record) [17:48:56] I take it deleting and starting over is /also/ difficult? [17:53:18] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10mmodell) >>! In T143162#5605053, @Krinkle wrote: > > These were merged patches. I d... [17:55:56] yeah, also not super easy. I'm looking at what needs to happen. the "database" for gerrit is, of course, not a real database: it's a git repo. This requires some form of git ref surgery, so I'm trying to get all my ducks in a row right now. [17:56:54] lol @ ducks :P [17:57:05] quack quack [18:00:07] 10Diffusion, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Reduce task notification noise/frequency of changes to associated open patchsets - https://phabricator.wikimedia.org/T143162 (10mmodell) I realized I probably need to clarify further and make a couple of points e... [18:15:34] andrewbogott: I have a fix, but I've got to jump into a meeting, I'll ping you when I'm out and able to apply the fix ~30mins (sorry :() [18:16:38] thcipriani: I'm going afk for a bit but will read the backscroll when I return [18:16:51] So, no rush! [18:29:08] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:31:19] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:31:56] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:33:43] (03CR) 10Umherirrender: "I have blacklist @testWith because there should be one way for provider and a function is more readable than annotation. Than you does not" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546476 (owner: 10simetrical) [18:34:25] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:36:38] andrewbogott: I think it should be fixed now, if you want to give logging in a try again [18:37:40] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:44:21] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:46:07] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:46:46] (03CR) 10simetrical: "I don't think that there needs to be only one way to do everything, and I don't think a function is always more readable. Compare:" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/546476 (owner: 10simetrical) [18:46:49] 10Release-Engineering-Team (Unit & Int & System Tooling), 10Release-Engineering-Team-TODO (201910), 10MediaWiki-Core-Testing, 10MW-1.35-notes (1.35.0-wmf.1; 2019-10-08), and 3 others: All repositories with Selenium tests should use wdio-mediawiki - https://phabricator.wikimedia.org/T199113 (10zeljkofilipin) [18:50:27] (03PS1) 10Zfilipin: WIP npm ERR! missing script: selenium-daily WIP [integration/config] - 10https://gerrit.wikimedia.org/r/546679 (https://phabricator.wikimedia.org/T199113) [18:59:36] scap for parsoid deploy on beta is failing for arlo ... is there anything known broken there? [19:01:14] (03CR) 10Jforrester: [C: 03+2] Add a new project for operations/software/httpbb. [integration/config] - 10https://gerrit.wikimedia.org/r/546661 (https://phabricator.wikimedia.org/T236699) (owner: 10RLazarus) [19:02:49] (03CR) 10Hashar: [C: 03+2] "Updated:" [integration/config] - 10https://gerrit.wikimedia.org/r/546663 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [19:03:03] (03Merged) 10jenkins-bot: Add a new project for operations/software/httpbb. [integration/config] - 10https://gerrit.wikimedia.org/r/546661 (https://phabricator.wikimedia.org/T236699) (owner: 10RLazarus) [19:04:38] !log Zuul: Add CI for operations/software/httpbb T236699 [19:04:41] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:04:41] T236699: Build a black-box httpd testing framework - https://phabricator.wikimedia.org/T236699 [19:05:21] (03Merged) 10jenkins-bot: Upgrade jjb to 3.0.2 [integration/config] - 10https://gerrit.wikimedia.org/r/546663 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [19:06:13] looks like we are missing a script on the beta parsoid servers which is on production .. restart-php .. [19:38:26] thcipriani, hashar: More missing "merged" changes showing up in gerrit. E.g. https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CodeEditor/+/488213 which landed in February as 5cac43ddd34705cd69af6b19af7935afb19a7c21 [19:43:41] (03PS1) 10Hashar: Have notice dependency changes [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) [19:44:07] (03PS2) 10Hashar: Make tox notice dependency changes [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) [19:44:48] (03CR) 10Jforrester: "Wow. Impressive fix." [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [19:44:59] hashar: I am full of admiration. [19:45:10] haha [19:45:12] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, and 2 others: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) Last thing is that when changing the JJB version in `jenkins... [19:45:31] James_F: I wanted to fix for ages [19:45:48] and had to send an email whenever the git repo got updated [19:45:48] now [19:45:59] one would just send the bump in tox.ini [19:46:06] * James_F nods. [19:46:10] and anyone doing a pull and running tox would have the proper jjb ! [19:46:17] (unless I have missed something) [19:46:33] then I guess I will have to update moare doc on wiki [19:47:15] Yes. [19:47:44] kids story time !!! [19:48:59] for the CodeEditor change [19:49:04] it is there in the git repo [19:49:26] Yeah. [19:49:38] I thought gerrit got fixed up? Did it regress? [19:49:39] $ git -C /srv/gerrit/git/mediawiki/extensions/CodeEditor.git ls-remote . refs/changes/13/488213* [19:49:39] 0f1eb79624d97f14434dd42dfc9feef3c895cd32 refs/changes/13/488213/1 [19:49:39] 98c6bb05dc14bc1341d6e1b302d5860d2b0ed4ab refs/changes/13/488213/meta [19:49:50] though with a different sha1!!? :-\ [19:50:15] Yeah, it's an old patchset of the patch. [19:50:22] It was modified before merging. [19:50:30] $ git log -n1 --decorate --oneline 5cac43ddd34705cd69af6b19af7935afb19a7c21 [19:50:30] 5cac43dd (refs/changes/32/496332/1) build: Update eslint-config-wikimedia to 0.11.0 [19:50:41] and send again as a differnet change [19:51:41] diff -u <(git show 0f1eb796) <(git show 5cac43dd)|head -n12 [19:51:47] - build: Update eslint-config-wikimedia to 0.10.1 [19:51:47] + build: Update eslint-config-wikimedia to 0.11.0 [19:51:47] [19:51:47] Change-Id: I99fd10fbcce5d5e6b87b6e3faeb7b18c4e573ef7 [19:51:50] same Change-Id [19:51:51] Once upon a time there was a badly configured continuous integration plugin... [19:52:00] James_F: so I think it is all good [19:52:06] just that well, there are two change id [19:52:10] Should I abandon the task? [19:52:15] Err. The patch. [19:52:30] no clue how that can happen though [19:52:50] Yeah. [19:53:34] (03CR) 10Jforrester: [C: 03+1] Make tox notice dependency changes [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [20:04:55] (03PS2) 10Jeena Huneidi: Update deployments to be compatible with newer k8s version [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546317 (https://phabricator.wikimedia.org/T236471) [20:07:02] (03CR) 10Hashar: [C: 03+2] "Thanks :)" [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [20:07:16] and it works James_F :] [20:08:26] hmm [20:08:34] I have send an email to QA but it does not show up bah [20:09:30] (03Merged) 10jenkins-bot: Make tox notice dependency changes [integration/config] - 10https://gerrit.wikimedia.org/r/546698 (https://phabricator.wikimedia.org/T236036) (owner: 10Hashar) [20:20:19] 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, and 2 others: Align jenkins-job-builder with upstream - https://phabricator.wikimedia.org/T236036 (10hashar) 05Open→03Resolved Thank you @Jdforrester-WMF for the rev... [20:41:42] James_F: hashar FWIW, that change looked exactly the same on cobalt [20:41:56] Fun. :-( [20:42:33] yeah :-\ [20:42:41] I blame somoeone for doing something crazy at somepoint [20:42:58] Possibly you. ;-) [20:43:20] !! [20:45:00] * hashar profiles JJB [20:45:02] it is slow [20:47:18] it merged as https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CodeEditor/+/496332/ FWIW [20:47:31] I think the Right solution is to delete the other changeset [20:48:22] Delete or just abandon? [20:48:26] I've seen this happen 1 other time: i.e., gerrit making duplicate patchsets. I think it's probably some race condition, but I don't know how to recreate [20:48:32] * James_F nods. [20:49:10] delete in this instance (doesn't seem like anything meaningful happened on the other patchset), duplicate change-ids confuses gerrit. [20:49:12] We depend on gerrit to force pushes with the same change ID to the same branch and project to be the same changeset. Nothing outside gerrit can fix that. [20:49:25] OK. [20:50:15] * thcipriani deletes [20:51:11] fixes 404 on https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/CodeEditor/+/496332/ [20:51:17] ({{done}}) [20:51:27] Hurrah. [20:51:29] evening hack https://review.opendev.org/#/c/691720/2/jenkins_jobs/registry.py :D [20:51:42] makes jjb for our config go down from 28 secs to 17 secs on my machine [20:52:37] thcipriani: James_F: I guess a change can be written to the git repository, fails to index for some reason and thus the UI does not show it up? [20:52:44] then later someone resend the same patch [20:52:52] thcipriani: everything works! Thank you [20:52:54] but it is not in the index, so Gerrit happilly accepts it [20:52:59] Apparently upstream hits that race frequently thcipriani [20:53:07] unless it's something to do with there data center [20:53:35] hashar: that was my theory. Essentially, we restart before the change is reindexed, it hits the disk, but not the index, repush creates new change [20:53:38] * paladox hit the create change and end up with duplicate changes issue upstream frequenty [20:53:56] following reindex: two patchsets with the same changeid [20:54:14] andrewbogott: great! [20:54:23] hashar: Mayve. [20:58:35] (03CR) 10Brennen Bearnes: [C: 03+2] "Tested, working." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546317 (https://phabricator.wikimedia.org/T236471) (owner: 10Jeena Huneidi) [20:59:56] python3 -Ximporttime [21:00:05] to trace duration of imports! [21:02:14] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Quibble: Install wikidiff2 into mediawiki-quibble-api-testing-vendor-mysql-php72-docker - https://phabricator.wikimedia.org/T236333 (10brennen) 05Open→03Resolved [21:03:32] and for a visualization of profiling https://jiffyclub.github.io/snakeviz/ /:D [21:14:57] (03PS1) 10Umherirrender: [WikimediaMaintenance] Run phan job [integration/config] - 10https://gerrit.wikimedia.org/r/546736 (https://phabricator.wikimedia.org/T224770) [21:15:23] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] Update deployments to be compatible with newer k8s version [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546317 (https://phabricator.wikimedia.org/T236471) (owner: 10Jeena Huneidi) [21:17:52] (03CR) 10Jeena Huneidi: "I know this could use some improvement but I am not sure what to add." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546319 (https://phabricator.wikimedia.org/T236587) (owner: 10Jeena Huneidi) [21:22:18] (03CR) 10Brennen Bearnes: [V: 03+2 C: 03+2] "Tested, working." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546315 (owner: 10Jeena Huneidi) [21:26:04] (03CR) 10Brennen Bearnes: "Should we go into detail on building dev base images, etc.?" [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546319 (https://phabricator.wikimedia.org/T236587) (owner: 10Jeena Huneidi) [21:38:24] 10RelEng-Archive-FY201718-Q1, 10Scap, 10Patch-For-Review: Scap: keyholder Too many authentication failures - https://phabricator.wikimedia.org/T172333 (10thcipriani) >>! In T172333#5607532, @mmodell wrote: > This is happening to me on phab1001 now and phab's scap::target includes the requisite part: > > `la... [21:38:48] (03PS1) 10Brennen Bearnes: debian_prereq.sh: bump helm version to 2.15.1 [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546750 (https://phabricator.wikimedia.org/T233960) [21:43:39] (03CR) 10Jeena Huneidi: [C: 03+2] debian_prereq.sh: bump helm version to 2.15.1 [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546750 (https://phabricator.wikimedia.org/T233960) (owner: 10Brennen Bearnes) [21:44:58] moauaha [21:45:05] down to 4 seconds to generate our JJB config :] [21:51:53] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Reimage cobalt and gerrit2001 as buster - https://phabricator.wikimedia.org/T176774 (10Dzahn) gerrit prod has switched from cobalt to gerrit1001, it will be decom'ed but we need to create the ticket... [21:54:25] (03CR) 10Brennen Bearnes: [V: 03+2] "Tested before submission on Debian Buster." [releng/local-charts] - 10https://gerrit.wikimedia.org/r/546750 (https://phabricator.wikimedia.org/T233960) (owner: 10Brennen Bearnes) [22:10:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) [22:10:38] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Upstream: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) p:05Triage→03Normal [22:12:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (201910), 10Jenkins, 10Upstream: JJB is too slow - https://phabricator.wikimedia.org/T236746 (10hashar) | Patch | Description | Before | After |--|--|--|-- | https://review.opendev... [22:17:14] 10Gerrit, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO, 10Operations: Reimage cobalt and gerrit2001 as buster - https://phabricator.wikimedia.org/T176774 (10Dzahn) [22:46:03] 10RelEng-Archive-FY201718-Q1, 10Scap, 10Patch-For-Review: Scap: keyholder Too many authentication failures - https://phabricator.wikimedia.org/T172333 (10mmodell) 05Open→03Resolved So I'm dumb. I had keyholder_key set in puppet but not in the scap.cfg in the deployment repo. False alarm, sorry! (Tested... [22:46:05] 10Gerrit, 10RelEng-Archive-FY201718-Q2, 10Scap: Deploy gerrit with scap3 - https://phabricator.wikimedia.org/T157414 (10mmodell) [23:13:08] Group Not Found: user%3Amobrovac i wonder what repo uses that... [23:13:23] that's singleusergroup syntax, we uninstalled that plugin a long while ago