[00:58:24] (03CR) 10Legoktm: [C: 032] Add CheckUser as dependency to AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/453419 (owner: 10Umherirrender) [00:59:37] (03CR) 10Legoktm: [C: 032] Archive the PdfExport extension [integration/config] - 10https://gerrit.wikimedia.org/r/453884 (https://phabricator.wikimedia.org/T202220) (owner: 10MarcoAurelio) [00:59:55] (03Merged) 10jenkins-bot: Add CheckUser as dependency to AbuseFilter [integration/config] - 10https://gerrit.wikimedia.org/r/453419 (owner: 10Umherirrender) [01:01:39] (03Merged) 10jenkins-bot: Archive the PdfExport extension [integration/config] - 10https://gerrit.wikimedia.org/r/453884 (https://phabricator.wikimedia.org/T202220) (owner: 10MarcoAurelio) [01:02:22] !log deployed https://gerrit.wikimedia.org/r/453419 https://gerrit.wikimedia.org/r/453884 [01:02:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:05:31] (03CR) 10Legoktm: [C: 032] Use extension-quibble-composer for Mailgun extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/452580 (owner: 10Phantom42) [01:05:37] (03CR) 10Legoktm: [C: 032] Revert "Drop Echo dependency for PageTriage, causes tests to fail" [integration/config] - 10https://gerrit.wikimedia.org/r/452617 (https://phabricator.wikimedia.org/T201812) (owner: 10Umherirrender) [01:05:49] (03CR) 10Legoktm: [C: 032] Add test dependency for BlueSpice extension [integration/config] - 10https://gerrit.wikimedia.org/r/447758 (owner: 10Pboehm) [01:06:21] (03CR) 10Legoktm: [C: 032] "Can't hurt..." [integration/config] - 10https://gerrit.wikimedia.org/r/449654 (owner: 10Legoktm) [01:06:57] (03Merged) 10jenkins-bot: Use extension-quibble-composer for Mailgun extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/452580 (owner: 10Phantom42) [01:06:59] (03Merged) 10jenkins-bot: Revert "Drop Echo dependency for PageTriage, causes tests to fail" [integration/config] - 10https://gerrit.wikimedia.org/r/452617 (https://phabricator.wikimedia.org/T201812) (owner: 10Umherirrender) [01:07:11] (03Merged) 10jenkins-bot: Add test dependency for BlueSpice extension [integration/config] - 10https://gerrit.wikimedia.org/r/447758 (owner: 10Pboehm) [01:08:21] (03Merged) 10jenkins-bot: Bump mediawiki-core-code-coverage-docker timeout to 4h [integration/config] - 10https://gerrit.wikimedia.org/r/449654 (owner: 10Legoktm) [01:08:27] !log deploying https://gerrit.wikimedia.org/r/452580 https://gerrit.wikimedia.org/r/452617 https://gerrit.wikimedia.org/r/447758 [01:08:31] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:10:39] (03PS1) 10Legoktm: CategoryTree: make seccheck voting [integration/config] - 10https://gerrit.wikimedia.org/r/453894 [01:11:31] (03CR) 10Legoktm: [C: 032] Add seccheck for timeline [integration/config] - 10https://gerrit.wikimedia.org/r/453586 (owner: 10Umherirrender) [01:11:52] (03CR) 10Legoktm: [C: 032] CategoryTree: make seccheck voting [integration/config] - 10https://gerrit.wikimedia.org/r/453894 (owner: 10Legoktm) [01:13:02] (03Merged) 10jenkins-bot: Add seccheck for timeline [integration/config] - 10https://gerrit.wikimedia.org/r/453586 (owner: 10Umherirrender) [01:13:14] (03Merged) 10jenkins-bot: CategoryTree: make seccheck voting [integration/config] - 10https://gerrit.wikimedia.org/r/453894 (owner: 10Legoktm) [01:24:45] !log deploying https://gerrit.wikimedia.org/r/453586 https://gerrit.wikimedia.org/r/453894 [01:24:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:32:45] (03PS1) 10Legoktm: Add seccheck to some more repositories [integration/config] - 10https://gerrit.wikimedia.org/r/453895 [01:33:25] (03CR) 10Legoktm: [C: 032] Add seccheck to some more repositories [integration/config] - 10https://gerrit.wikimedia.org/r/453895 (owner: 10Legoktm) [01:34:46] (03Merged) 10jenkins-bot: Add seccheck to some more repositories [integration/config] - 10https://gerrit.wikimedia.org/r/453895 (owner: 10Legoktm) [01:34:56] !log deploying https://gerrit.wikimedia.org/r/453895 [01:35:00] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [01:42:32] !log running jobs manually on contint1001 with gear_client [01:42:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:33:36] (03PS1) 10Legoktm: More seccheck extensions [integration/config] - 10https://gerrit.wikimedia.org/r/453899 [02:33:58] (03CR) 10Legoktm: [C: 032] More seccheck extensions [integration/config] - 10https://gerrit.wikimedia.org/r/453899 (owner: 10Legoktm) [02:35:25] (03Merged) 10jenkins-bot: More seccheck extensions [integration/config] - 10https://gerrit.wikimedia.org/r/453899 (owner: 10Legoktm) [02:35:59] !log deploying https://gerrit.wikimedia.org/r/453899 [02:36:05] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:09:55] (03PS1) 10Thiemo Kreuz (WMDE): Minor cleanups to the UnusedUseStatement sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/453935 [08:57:40] 10Release-Engineering-Team, 10GitHub-Mirrors: Requesting access to the 'wikimedia' organization at GitHub for MarcoAurelio - https://phabricator.wikimedia.org/T201259 (10MarcoAurelio) [09:06:31] 10Beta-Cluster-Infrastructure: Puppetize rebase = true for all relevant hosts (all?) - https://phabricator.wikimedia.org/T84953 (10hashar) 05Open>03Resolved a:03Andrew Solved by @andrew via b3eee4f9f396239237a408a4d111137e68aa2949 The rebase script now attempts the rebase in a temporary directory and abort... [09:18:11] (03PS1) 10Umherirrender: Run seccheck for Score [integration/config] - 10https://gerrit.wikimedia.org/r/453975 [09:25:08] 10Continuous-Integration-Config, 10Quibble: Quibble: shasum check failed - https://phabricator.wikimedia.org/T201638 (10hashar) 05Open>03Resolved That was most probably an issue with the npm js registry or a glitch while downloading it maybe. [09:41:08] 10Continuous-Integration-Config, 10Patch-For-Review: Submitting multiple mjolnir patches receive -1 with OSError: [Errno 12] Cannot allocate memory - https://phabricator.wikimedia.org/T201479 (10hashar) We did switch the maven job for MjoLniR with d063f5a124a94ee41bf30bd40a3132ca991569a2 . The tox job has not... [09:51:04] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, and 2 others: Migrate gated extensions jobs to Quibble/Docker - https://phabricator.wikimedia.org/T197469 (10hashar) [09:53:48] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3, 10Epic, 10Patch-For-Review: [EPIC] Migrate Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512 (10hashar) [09:54:31] (03PS1) 10Umherirrender: Add quibble for FormWizard [integration/config] - 10https://gerrit.wikimedia.org/r/453982 [09:57:38] HI hashar ! :D [09:57:39] (03CR) 10Pablo Grass (WMDE): readme: how to reproduce a CI build (032 comments) [integration/quibble] - 10https://gerrit.wikimedia.org/r/452335 (https://phabricator.wikimedia.org/T200991) (owner: 10Pablo Grass (WMDE)) [09:57:47] Hope you had a lovely vacation [09:58:24] (03CR) 10Hashar: [C: 032] Add quibble for FormWizard [integration/config] - 10https://gerrit.wikimedia.org/r/453982 (owner: 10Umherirrender) [09:59:52] (03Merged) 10jenkins-bot: Add quibble for FormWizard [integration/config] - 10https://gerrit.wikimedia.org/r/453982 (owner: 10Umherirrender) [10:03:57] (03CR) 10Hashar: Add awmd-stats tool to Jenkins (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/453819 (https://phabricator.wikimedia.org/T202234) (owner: 10D3r1ck01) [10:04:07] (03PS3) 10Hashar: Add awmd-stats tool to Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/453819 (https://phabricator.wikimedia.org/T202234) (owner: 10D3r1ck01) [10:04:14] (03CR) 10Hashar: [C: 032] Add awmd-stats tool to Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/453819 (https://phabricator.wikimedia.org/T202234) (owner: 10D3r1ck01) [10:05:48] (03Merged) 10jenkins-bot: Add awmd-stats tool to Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/453819 (https://phabricator.wikimedia.org/T202234) (owner: 10D3r1ck01) [10:06:38] addshore: I did :] Thanks for the WikibaseLexeme and Wikibases switches to quibble [10:10:10] (03CR) 10D3r1ck01: "Thank you Hashar, missed that!" [integration/config] - 10https://gerrit.wikimedia.org/r/453819 (https://phabricator.wikimedia.org/T202234) (owner: 10D3r1ck01) [10:39:17] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Jenkins, 10Zuul: Zuul should not start gate-and-submit for WIP patch sets - https://phabricator.wikimedia.org/T201744 (10hashar) Code-Review +2 are accepted by the Zuul pipeline `gate-and-submit`. The pipeline already filters out changes that are not ope... [10:47:48] hashar: no problem :D [10:48:09] think it is okay to be running both the wmf prefix quibble and the non prefix quibble for Wikibase in test? [10:48:41] it's another job, but we definitely need the list of "dependencies" to be loaded, not just the gated list of extensions for our tests [10:49:24] my other question was, how easy would it be to add something to quibble to include some arbitrary string in the LocalSettings file that is used? [10:49:45] if we manage that then we can switch the WikibaseClient / Repo specific jobs that are still on node pool to use quibble too [11:16:00] addshore: I was lunching sorry :) [11:16:40] addshore: I gotta document the wmf- prefix jobs and the prefix less jobs. I cant even remember off hand why I made the distinction [11:17:01] wmf- runs with the gated extensions only [11:17:10] they are essentially the same, it isjust that wmf- prefixed jobs are for master/wmf branches. They replaces the mediawiki-extensions-* gate jobs [11:17:12] yeah [11:17:22] the whole thing is a bit of a mess to be honest :^\ [11:17:28] all other extensions except wikibase as far as I coudl tell ran them without the prefix [11:17:56] and therefor they load the list of extensions as defined in the "dependencies" of parameterr_functions [11:18:01] yup. I phased out the mediawiki-extensions jobs just before my vacations but lacked time to tackle the wikibase ones [11:18:15] :) [11:24:27] gotta check my car. bbl [11:39:57] :D [11:58:04] addshore: Quibble does add settings to LocalSettings.php , though that is prepended iirc [12:10:19] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) [12:17:56] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10Amire80) The initial discussion in the Incubator wiki mostly supports this idea: https://incubator.wikimedia.org/w/index.php?titl... [12:26:14] okay who broke it [12:27:07] and why is shinken-wm not saying anything here [12:30:03] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) Hi! I was trying to find out more about that extension - the "Download" link on https://www.mediawiki.org/w/index.php?title=Extension:ChangeUserPasswords&oldid=2860241 seems to be broken and there's no "... [12:30:41] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) [12:39:15] sounds like it may have been a network thing? [12:45:09] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-Addshore: Allow use of phan 0.8.5+ in wikimedia CI - https://phabricator.wikimedia.org/T174339 (10hashar) [12:45:12] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Addshore: Provide php-ast 0.1.5 or later as a Debian package for CI - https://phabricator.wikimedia.org/T174338 (10hashar) 05Open>03Resolved a:03hashar Sury now provides 0.1.6 I will clean up our... [12:46:18] 10Continuous-Integration-Config, 10Patch-For-Review, 10User-Addshore: Allow use of phan 0.8.5+ in wikimedia CI - https://phabricator.wikimedia.org/T174339 (10hashar) Sury now provides php-ast 0.1.6 [12:55:40] (03PS1) 10Umherirrender: Run phan-generic for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/454010 [13:03:23] (03CR) 10Hashar: [C: 032] Run phan-generic for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/454010 (owner: 10Umherirrender) [13:04:49] (03Merged) 10jenkins-bot: Run phan-generic for PageTriage [integration/config] - 10https://gerrit.wikimedia.org/r/454010 (owner: 10Umherirrender) [13:17:13] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10Esanders) p:05Triage>03High This is happening far too often. Here it is timing out 4... [13:25:21] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team (Current), 10User-Ladsgroup: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) a:03Ladsgroup I've enabled and was able to use git-lfs in wikimedia org in gi... [13:31:05] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) This extension has been developed and tested at the University of Duisburg Essen. Here are some screenshots to help understand the usage of the Extension: (Change User Passwords Special Page is viewable a... [13:54:52] (03PS1) 10Hashar: Remove now unused mwext-mw-selenium-node-composer-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454021 [13:57:52] (03PS1) 10Hashar: Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 [13:59:23] (03PS2) 10Hashar: Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 [13:59:32] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Ottomata) Don't totally remember the context here, but I just built a new version of git-fat with your fix, and installed on deploy1001 `Unpacking git-fat (0.1.3-1~stretch1) ov... [13:59:43] (03CR) 10Hashar: [C: 032] Remove now unused mwext-mw-selenium-node-composer-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454021 (owner: 10Hashar) [14:01:09] (03CR) 10jerkins-bot: [V: 04-1] Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [14:01:35] (03Merged) 10jenkins-bot: Remove now unused mwext-mw-selenium-node-composer-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454021 (owner: 10Hashar) [14:26:11] 10Project-Admins: Requests for addition to the #acl*Project-Admins group (in comments) - https://phabricator.wikimedia.org/T706 (10Mz83ude) Request to be a Project Admin or create a Project for new ChangUserPasswords extension. https://phabricator.wikimedia.org/T202275 [14:44:48] (03CR) 10Hashar: [C: 04-1] "That mess up the Zuul 'mediawiki' queue in dependent pipelines ..." [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [14:58:33] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) None of the download links on that mw.org page work. Where is the code? :) [15:05:05] 10Release-Engineering-Team, 10MediaWiki-extensions-WikimediaIncubator, 10Epic, 10I18n: Make creating a new Language project easier - https://phabricator.wikimedia.org/T165585 (10StevenJ81) @Amire80, I'd venture that for now, since closing Incubator is not part of the proposal, we also don't need to discuss... [15:16:27] (03PS1) 10Hashar: test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 [15:16:29] (03PS1) 10Hashar: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 [15:16:50] (03PS2) 10Hashar: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 [15:17:16] (03PS3) 10Hashar: Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 [15:20:48] (03CR) 10Hashar: "I am not sure why the queue: parameter had to be dropped in https://gerrit.wikimedia.org/r/#/c/451005/ but I really need the Quibble jobs" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 (owner: 10Hashar) [15:25:55] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10mmodell) ```14:57:01 ImportError: No module named compiler.ast 14:57:01 gyp ERR! configu... [15:26:59] (03CR) 10jerkins-bot: [V: 04-1] Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [15:32:28] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Addshore: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101 (10Addshore) [15:33:20] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Addshore: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101 (10Addshore) @Reedy Is this still seen as being an issue? Looks like our browser tests still make suite some edits. https://wikidata.beta.wmflabs.org/wiki/Speci... [15:33:32] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Addshore: Periodically delete old items from beta wikidatawiki - https://phabricator.wikimedia.org/T168104 (10Addshore) [15:33:40] (03CR) 10Hashar: [C: 04-1] "Jessie does not have ffmpeg, when Jessie got cut they took the decision to use the fork libav which comes with different bin names. From h" [integration/config] - 10https://gerrit.wikimedia.org/r/452703 (https://phabricator.wikimedia.org/T193157) (owner: 10Zfilipin) [15:38:09] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Move Wikibase to using the normal mwext-testextension-* jobs - https://phabricator.wikimedia.org/T188717 (10Addshore) [15:38:44] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Ottomata) It might need to also be updated on targets hm. [15:40:17] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10Addshore) [15:40:26] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10Addshore) [15:40:47] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: Move Wikibase to using the normal mediawiki extension (quibble) jobs - https://phabricator.wikimedia.org/T188717 (10Addshore) I gave this task a bit of love as the description was out of date, although the task itself is still relevant. [15:44:42] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Test-Coverage: jenkins job to generate wikibase.git coverage - https://phabricator.wikimedia.org/T88434 (10Addshore) We have jobs running for a collection of our extensions now: - WikibaseLexeme - WikibaseQ... [15:44:59] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Test-Coverage: generate Wikibase.git code coverage on Jenkins - https://phabricator.wikimedia.org/T88434 (10Addshore) [15:46:19] 10Phabricator: Phabricator account with existing email address - https://phabricator.wikimedia.org/T201960 (10Nettrom) Is this something that can be resolved on the Phabricator end, or should I look for a workaround? Either way is fine with me, as long as I can get a second account set up. [15:47:12] 10Gerrit, 10Wikidata: [Task] trigger packagist.org from repo on gerrit.w.o - https://phabricator.wikimedia.org/T87768 (10Addshore) >>! In T87768#4183211, @demon wrote: > Eh, one less job to waste an executor over -- no delay, etc. Now that most of the jenkins nodes are docker nodes (rather than node pool) and... [15:47:26] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10mmodell) So I think that the python import error is a red herring. It looks like Firefox... [15:47:46] 10Continuous-Integration-Config, 10Wikidata, 10User-Addshore: [Task] Wikibase: Set up Jenkins to run unit tests also against repo without client - https://phabricator.wikimedia.org/T114690 (10Addshore) [15:48:11] (03PS2) 10Hashar: test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 [15:49:24] 10Release-Engineering-Team, 10Wikidata, 10Epic, 10Patch-For-Review, 10User-Addshore: [Epic] Kill the Wikidata build step - https://phabricator.wikimedia.org/T173818 (10Addshore) [15:49:29] (03CR) 10jerkins-bot: [V: 04-1] test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 (owner: 10Hashar) [15:49:34] (03PS3) 10Hashar: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 [15:50:45] (03PS3) 10Hashar: test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 [15:50:53] 10Phabricator, 10Release-Engineering-Team (Kanban): Phabricator DB query request: list repositories and assigned callsigns - https://phabricator.wikimedia.org/T202222 (10mmodell) a:03mmodell [15:50:56] (03CR) 10jerkins-bot: [V: 04-1] Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 (owner: 10Hashar) [15:51:00] 10Phabricator, 10Release-Engineering-Team (Kanban): Phabricator DB query request: list repositories and assigned callsigns - https://phabricator.wikimedia.org/T202222 (10mmodell) p:05Triage>03Normal [15:53:13] (03PS4) 10Hashar: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 [15:58:40] !log deleting Jenkins slave deployment-tin.eqiad the instance has been replaced | T192561 [15:58:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:58:47] T192561: Upgrade deployment-prep deployment servers to stretch - https://phabricator.wikimedia.org/T192561 [15:58:48] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10Esanders) Yes, I always assumed this 30s test timeout was the issue, but don't know what... [15:59:04] (03PS4) 10Thcipriani: test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 (owner: 10Hashar) [16:00:02] (03CR) 10Thcipriani: [C: 032] "Nice! Works for me locally. Should be useful in preventing regressions." [integration/config] - 10https://gerrit.wikimedia.org/r/454037 (owner: 10Hashar) [16:01:35] (03Merged) 10jenkins-bot: test Quibble jobs are skipped on fundraising branches [integration/config] - 10https://gerrit.wikimedia.org/r/454037 (owner: 10Hashar) [16:57:44] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10mmodell) That would probably be wise at least for now, since it seems to be only Firefox... [16:58:38] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10Dzahn) We have had some minor discussion on IRC about it. I mentioned that the point was brought up that even with full root access the u... [17:00:46] (03CR) 10Dduvall: [C: 031] "Looks right!" [integration/config] - 10https://gerrit.wikimedia.org/r/453535 (https://phabricator.wikimedia.org/T202068) (owner: 10Thcipriani) [17:31:54] (03PS5) 10Hashar: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 [17:37:26] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team (Current), 10User-Ladsgroup: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) Everything is moved now: https://github.com/orgs/wikimedia/teams/scoring-platfo... [17:45:12] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865 (10mmodell) @aklapper: FYI it is now possible to change the type of a task using conduit or herald. Examples: {H295} {H296} [17:46:19] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10mmodell) >>! In T193770#4327004, @dbarratt wrote: > Well it would be nice if it could be added to the edit task form and not the create task form. That way it's not there when c... [17:46:38] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10mmodell) 05Open>03Resolved a:03mmodell [17:49:36] 10Phabricator, 10Release-Engineering-Team (Kanban): Deploy "Deadlines" feature - https://phabricator.wikimedia.org/T191865 (10mmodell) [17:49:39] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499 (10mmodell) [17:53:16] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10dduvall) a:03dduvall I filed a related task, {T202160}, this past Friday which has to do with consolidating execut... [18:04:01] 10Phabricator: The Edit Task form of an existing task should expose the 'Due Date' field - https://phabricator.wikimedia.org/T193770 (10dbarratt) YAY! Thank you! [18:04:04] 10Phabricator: CURLE_COULDNT_CONNECT when trying to use Conduit - https://phabricator.wikimedia.org/T201746 (10mmodell) @tgr: no antivandalism doesn't currently affect conpherence, though it might in the future. [18:06:04] 10Phabricator: CURLE_COULDNT_CONNECT when trying to use Conduit - https://phabricator.wikimedia.org/T201746 (10mmodell) Is it possible to build a retry into the script? I'm not really sure what would cause it to fail connecting but it sounds like general network unavailability. [18:16:36] !log Taking integration-slave-docker-1020 node offline for replacement [18:16:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:19:42] !log Deleting integration-slave-docker-1020 instance [18:19:45] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:21:46] (03CR) 10Thcipriani: [C: 032] Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 (owner: 10Hashar) [18:23:10] (03Merged) 10jenkins-bot: Revert "Remove 'queue' to skip quibble on fr branches" [integration/config] - 10https://gerrit.wikimedia.org/r/454038 (owner: 10Hashar) [18:24:03] !log reloading zuul to deploy https://gerrit.wikimedia.org/r/#/c/integration/config/+/454038/ [18:24:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [18:43:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10thcipriani) >>! In T201470#4515591, @Dzahn wrote: > Would that work if we continue with more specific sudo privilege lines for using apt?... [18:48:10] 10Continuous-Integration-Config, 10Collaboration-Team-Triage, 10Growth-Team, 10Notifications, 10Documentation: Generate doxygen for Echo on doc.wm.o - https://phabricator.wikimedia.org/T117025 (10srodlund) [18:49:34] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10thcipriani) >>! In T202100#4515232, @Ottomata wrote: > It might need to also be updated on targets hm. Yeah, we'll need to update the targets since that's where we run into the... [18:50:02] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Addshore: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101 (10Reedy) Deleting periodically would be fine too. https://wikidata.beta.wmflabs.org/wiki/Special:Statistics doesn't look bad atm with just under 200K pages [18:51:26] !Log adding new jenkins node integration-slave-docker-1025 (m1.xlarge instance) w/ 5 executors [18:54:03] 10Scap, 10Operations, 10Patch-For-Review: Intermittent git-fat failure during deploy - https://phabricator.wikimedia.org/T202100 (10Ottomata) we just updated it for wqds* hosts. If that worked fine for @Gehel and Erik, we'll update the rest of the flee (all nodes!) with @MoritzMuehlenhoff when he's back aro... [18:58:00] 10Gerrit, 10Release-Engineering-Team (Kanban): Gerrit has created duplicate accounts for some users - https://phabricator.wikimedia.org/T197083 (10thcipriani) [18:58:02] 10Gerrit, 10Release-Engineering-Team (Kanban): Gerrit plugin to edit accountid - https://phabricator.wikimedia.org/T199749 (10thcipriani) 05Open>03declined >>! In T199749#4499909, @mmodell wrote: > After struggling with the gerrit plugin approach, I really think that it's largely misguided route to the ori... [18:58:12] twentyafterfour: hey, regarding https://phabricator.wikimedia.org/T192042 I turned their github repos to git LFS and now git pull from the mirrors return error of object not found [18:58:28] https://www.irccloud.com/pastebin/5Jg8EWLr/ [19:00:24] (03CR) 10Hashar: [C: 04-1] "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [19:00:37] (03CR) 10jerkins-bot: [V: 04-1] Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [19:03:45] I think for whatever reason the gerrit mirror is not pulling things from github [19:04:49] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Mz83ude) Hi. So it's our first time developing a mediawiki extension. We have the code in our university server. We cannot push the code to Phabricator until a new project is created. https://phabricator.wikime... [19:05:46] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10Ladsgroup) 05Resolved>03Open After migrating the github repos to use git lfs, now it returns this error: ``` amsa@C235:~/ores-prod-dep... [19:05:49] 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Support git-lfs in scap - https://phabricator.wikimedia.org/T180627 (10Ladsgroup) [19:07:33] 10Release-Engineering-Team (Watching / External), 10Scoring-platform-team (Current), 10User-Ladsgroup: Another round of discussion about wiki-ai's GitHub->gerrit mirroring - https://phabricator.wikimedia.org/T194212 (10Ladsgroup) All of the repos with model files now are using git lfs [19:07:57] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867 (10dduvall) The other day I used this nasty little Ruby one-liner to cleanup docker-pkg images that weren't `latest`. It wasn't perfect (Docker complained about trying to delete som... [19:25:36] 10Continuous-Integration-Infrastructure, 10VisualEditor, 10Patch-For-Review: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10Esanders) p:05High>03Normal Lowering priority to reflect this... [19:34:25] 10Phabricator: CURLE_COULDNT_CONNECT when trying to use Conduit - https://phabricator.wikimedia.org/T201746 (10Tgr) The script retries three times: ``` 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-StopForumSpam, 10Wikimedia-extension-review-queue: Deploy StopForumSpam to the Beta Cluster - https://phabricator.wikimedia.org/T181217 (10MarcoAurelio) p:05Triage>03Lowest Per the lack of activity. [19:52:52] !log Taking integration-slave-docker-1016 node offline for replacement [19:52:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:53:41] !log Deleting instance integration-slave-docker-1016 [19:53:44] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:58:16] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Pipeline image build cleanup - https://phabricator.wikimedia.org/T177867 (10thcipriani) >>! In T177867#4516003, @dduvall wrote: > The other day I used this nasty little Ruby one-liner to cleanup docker-pkg images that weren't `latest`. It wasn't perfect... [19:58:21] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10dduvall) A `m1.xlarge` instance is now up and the [[ https://integration.wikimedia.org/ci/computer/integration-slave... [19:59:54] (03PS1) 10Hashar: Name dependent mediawiki queue based on a Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/454087 (https://phabricator.wikimedia.org/T183512) [20:01:10] (03PS4) 10Hashar: Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 [20:02:50] (03CR) 10Hashar: "queue issue is sorted out by https://gerrit.wikimedia.org/r/#/c/integration/config/+/454087/1" [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [20:09:16] 10Release-Engineering-Team, 10Scap: Canary deployment continuation prompt should emit a timestamp to console - https://phabricator.wikimedia.org/T201635 (10thcipriani) p:05Triage>03Normal [20:13:55] (03CR) 10Hashar: [C: 032] "layout diff is way nicer as a result :]" [integration/config] - 10https://gerrit.wikimedia.org/r/454087 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:15:22] (03Merged) 10jenkins-bot: Name dependent mediawiki queue based on a Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/454087 (https://phabricator.wikimedia.org/T183512) (owner: 10Hashar) [20:15:57] (03CR) 10Hashar: [C: 032] Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [20:16:19] !log (beta): Update mobileapps to 95e976d (T202105) [20:16:25] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:16:25] T202105: Separate pagelib CSS from base CSS - https://phabricator.wikimedia.org/T202105 [20:17:22] one less nodepool job! [20:17:56] (03Merged) 10jenkins-bot: Drop mediawiki-core-qunit-selenium-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/454022 (owner: 10Hashar) [20:18:15] !log Adding jenkins node integration-slave-docker-1026 (bigram instance) w/ 8 executors [20:18:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:30:32] !log Correction: integration-slave-docker-1026 configured w/ 9 executors [20:30:36] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:34:23] 10Phabricator: @dbarrat account got disabled by the anti vandalism software - https://phabricator.wikimedia.org/T202328 (10MarcoAurelio) [20:34:44] 10Phabricator: @dbarratt account got disabled by the anti vandalism software - https://phabricator.wikimedia.org/T202328 (10MarcoAurelio) p:05Triage>03Normal [20:36:58] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10dduvall) A `bigmem` instance is up, and the [[ https://integration.wikimedia.org/ci/computer/integration-slave-docke... [20:37:36] !log Reconfigured integration-slave-docker-1026 back to 8 executors (at least 1 vcpu per concurrent job) [20:37:39] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:59:18] 10Continuous-Integration-Config, 10Collaboration-Team-Triage, 10Growth-Team, 10StructuredDiscussions, 10Documentation: Generate Doxygen documentation for Flow PHP classes to doc.wikimedia.org - https://phabricator.wikimedia.org/T93107 (10srodlund) [21:00:34] 10Phabricator, 10User-Ladsgroup: @dbarratt account got disabled by the anti vandalism software - https://phabricator.wikimedia.org/T202328 (10MarcoAurelio) 05Open>03Resolved a:03Ladsgroup Fixed by @Ladsgroup. @dbarratt is also added to #trusted-contributors. [21:10:30] 10Phabricator, 10User-Ladsgroup: @dbarratt account got disabled by the anti vandalism software - https://phabricator.wikimedia.org/T202328 (10dbarratt) Thanks! I'm glad to be back! :) [21:11:17] 10Phabricator, 10User-Ladsgroup: @dbarratt account got disabled by the anti vandalism software - https://phabricator.wikimedia.org/T202328 (10dbarratt) I do want to stop and admire the irony of being blocked for editing a ticket about blocking. :) [21:21:08] (03PS1) 10Hashar: Remove unused mediawiki-phpunit-* jobs [integration/config] - 10https://gerrit.wikimedia.org/r/454154 (https://phabricator.wikimedia.org/T183512) [21:21:45] 10Continuous-Integration-Config, 10Wikipedia-Android-App-Backlog: Remove xdummy config stuff if no longer needed - https://phabricator.wikimedia.org/T171535 (10Dbrant) 05Open>03Resolved a:03Dbrant [21:25:38] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10awight) @Ladsgroup Please be sure to share the update procedure with us once you're ready. I get this warning: > Your branch and 'origin/... [21:29:10] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10Ladsgroup) Yeah, The thing is the whole history is rewritten now, so the hashes of commits are different and now your system thinks they a... [21:29:38] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10awight) I get the same error: ``` $ git status On branch master Your branch and 'origin/master' have diverged, and have 251 and 251 diffe... [21:32:56] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10awight) Here's the error log mentioned above, {F25187675} [21:36:49] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10Ladsgroup) >>! In T192042#4516643, @awight wrote: > In the future it seems prudent to do this sort of rewrite into a named branch, but I g... [21:37:16] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10Ladsgroup) FYI, completely removing and adding a submodule didn't work either. [21:39:39] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10awight) >>! In T192042#4516665, @Ladsgroup wrote: >>>! In T192042#4516643, @awight wrote: >> In the future it seems prudent to do this sor... [21:39:49] Amir1: So gerrit is returning object not found? I'm not really sure how mirroring works with lfs [21:40:16] twentyafterfour: yeah :/ [21:40:29] that's the only blocker from moving everything to lfs right now [21:45:45] It’s using phabricator [21:45:59] Amir1: have you tried `git lfs push --all gerrit` (or whatever the name of your gerrit remote is) [21:47:43] let me try [21:48:12] one thing is I'm not sure if I have push rights on gerrit for these repos [21:48:18] but let's see [21:52:21] twentyafterfour: I get into an infinite look of authorization [21:52:39] https://usercontent.irccloud-cdn.com/file/xOGwIr06/image.png [21:53:26] I'm probably don't have the rights to push [21:58:51] Amir1: I would try using ssh:// protocol [22:00:34] Platonides: yup, you're right [22:00:55] https://www.irccloud.com/pastebin/G7ynBMYt/ [22:01:12] twentyafterfour: can I get push rights on scores/ores ? [22:01:31] at least temporarily to upload lfs files? [22:02:07] *scoring/ores [22:06:20] Amir1: sure let me see [22:06:32] one sec I'll fix you up [22:06:39] Thanks [22:07:14] I hope it gets fixed in a way that in future I don't need to push from github to gerrit any time we change a model and it gets mirrored directly :P [22:09:34] Amir1: ok privs should be fixed [22:10:08] twentyafterfour: Thanks, I'm uploading now! [22:10:25] so in future, do I need to push it every time too? [22:11:01] Amir1: which way is replication set up? Is it replicating from github to phabricator and phabricator to gerrit? [22:11:30] I don't think phabricator automatically replicates lfs and I'm not sure about gerrit [22:12:03] twentyafterfour: I have no idea, per the ticket you set it up [22:12:28] https://phabricator.wikimedia.org/T192042 [22:13:25] Also, I need the push rights in wikiclass and some other lfs repos as well [22:13:29] yeah I think it's phabricator mirroring it. I'll poke at it, I can probably add `git-lfs push` to the phabricator mirroring service [22:14:09] Nice [22:16:40] Amir1: ok the rights are fixed for all of scoring/ores/* [22:17:07] Thank you! [22:17:55] twentyafterfour: btw. all of the repos have been moved from wiki-ai org to wikimedia org in github [22:18:11] it redirects so it doesn't matter [22:18:22] but just in case it's easy to clean up [22:38:03] 10Project-Admins: New Extension: ChangeUserPasswords - https://phabricator.wikimedia.org/T202275 (10Aklapper) Ah, I think there is a misunderstanding here. :) If you want to push your code, see https://www.mediawiki.org/wiki/Gerrit/New_repositories/Requests . We don't use Phabricator for code review but only to... [22:50:43] (03PS5) 10Thcipriani: Disconnect nodes where /srv > 95% full [integration/config] - 10https://gerrit.wikimedia.org/r/451078 (https://phabricator.wikimedia.org/T201224) [22:54:52] 10Continuous-Integration-Infrastructure: castor02 integration node /srv disk is full - https://phabricator.wikimedia.org/T202341 (10thcipriani) [22:56:27] 10Beta-Cluster-Infrastructure, 10Wikidata, 10User-Addshore: Wikidata on beta is getting too many edits - https://phabricator.wikimedia.org/T168101 (10Addshore) >>! In T168101#4515938, @Reedy wrote: > Deleting periodically would be fine too. A delete within mediawiki, or a "real" delete / removal? [22:58:14] (03CR) 10Thcipriani: [C: 032] "Deployed and seems to be working." [integration/config] - 10https://gerrit.wikimedia.org/r/451078 (https://phabricator.wikimedia.org/T201224) (owner: 10Thcipriani) [22:59:56] (03Merged) 10jenkins-bot: Disconnect nodes where /srv > 95% full [integration/config] - 10https://gerrit.wikimedia.org/r/451078 (https://phabricator.wikimedia.org/T201224) (owner: 10Thcipriani) [23:00:53] 10Release-Engineering-Team (Kanban), 10ORES, 10Scoring-platform-team: Create gerrit mirrors for all github-based ORES repos - https://phabricator.wikimedia.org/T192042 (10mmodell) It seems git-lfs does not have any support for mirroring. I think that we will have to roll our own solution for mirroring from g... [23:05:16] 10Continuous-Integration-Infrastructure, 10VisualEditor: Docker headless Chrome running is sometimes silently terminating on start(?), making it impossible to merge code in VE - https://phabricator.wikimedia.org/T200347 (10Jdforrester-WMF) [23:07:30] 10Continuous-Integration-Infrastructure, 10Jenkins, 10Patch-For-Review: Jenkins should auto-depool nodes if they run out of disk space on specific partitions - https://phabricator.wikimedia.org/T201224 (10thcipriani) 05Open>03Resolved a:03thcipriani Deployed a groovy script that is limited to running o... [23:47:01] twentyafterfour: do you know who maintains deployment-db*? [23:48:26] whoever it is has been logging in to the servers directly as root instead of using sudo, so the logs aren't giving me many hints [23:52:42] maybe it is Krenair and/or Riccardo [23:54:24] I tend not to log in directly as root [23:55:04] I normally only do that if an instance is completely broken to the point of LDAP logins not working, or sometimes to push stuff onto deployment-puppetmaster* [23:55:10] (or pull) [23:55:43] 'maintain' might be a strong word with those instances [23:56:39] it's possible someone in ops is logging in who hasn't been added as a deployment-prep member [23:58:10] what sort of commands have they been doing?