[00:04:00] <wikibugs>	 (03PS1) 10Brennen Bearnes: new RESTBase image & improved build instructions [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571601 (https://phabricator.wikimedia.org/T228910)
[00:04:58] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10RESTBase, 10local-charts, and 4 others: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910 (10brennen)
[01:20:50] <wikibugs>	 (03CR) 10Jeena Huneidi: [V: 03+2 C: 03+2] "LGTM" [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571601 (https://phabricator.wikimedia.org/T228910) (owner: 10Brennen Bearnes)
[01:24:36] <brennen>	 !log Updating dev-images docker-pkg files on contint1001 for T228910
[01:24:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[01:24:38] <stashbot>	 T228910: Move restbase chart from local-charts to deployment-charts repository - https://phabricator.wikimedia.org/T228910
[01:54:16] <wikibugs>	 (03PS3) 10Jeena Huneidi: Remove restbase chart and add restrouter chart [releng/local-charts] - 10https://gerrit.wikimedia.org/r/570236 (https://phabricator.wikimedia.org/T228910)
[01:58:13] <wikibugs>	 (03PS1) 10Jeena Huneidi: Disable MariaDB replication [releng/local-charts] - 10https://gerrit.wikimedia.org/r/571621
[08:44:37] <wikibugs>	 10Gerrit, 10Operations, 10Patch-For-Review: gerritro user getting access denied from gerrit1002 - https://phabricator.wikimedia.org/T243800 (10Marostegui) >>! In T243800#5874562, @Dzahn wrote: > @Marostegui I made some changes to make the db_user and db_pass configurable for gerrit.  Thing is just i don't kn...
[08:45:55] <wikibugs>	 10Diffusion, 10GitHub-Mirrors, 10Wikidata, 10User-MarcoAurelio: Create Github and Diffusion mirrors for wikibase/vuejs-components Gerrit repository - https://phabricator.wikimedia.org/T240224 (10WMDE-leszek) thanks @MarcoAurelio !
[09:58:43] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Performance-Team, 10Upstream, 10phan: Constant inheritance not detected in phan, blocking merge of a patch - https://phabricator.wikimedia.org/T244633 (10Daimona) Fixed upstream, will be included in phan 2.4.9. Based on previous release times, we can ex...
[13:41:20] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10DBA, 10Operations: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10Marostegui) 12th (the original date I suggested) has passed, any tentative date @mmodell you'd like to consider, there i...
[15:57:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-logstash03 is CRITICAL: CRITICAL: deployment-prep.deployment-logstash03.diskspace.root.byte_percentfree (<100.00%)
[16:05:55] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867 (10dbarratt)
[16:05:57] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Anti-Harassment, 10MW-1.35-notes (1.35.0-wmf.19; 2020-02-11), 10Patch-For-Review: [Regression 1.35.0-wmf.19] i/p/IndexPager.php:*  PHP Warning: implode(): Invalid arguments passed - https://phabricator.wikimedia.org/T244937 (10dbarratt) 05Open→...
[16:07:02] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Anti-Harassment, 10MW-1.35-notes (1.35.0-wmf.19; 2020-02-11), 10Patch-For-Review: [Regression 1.35.0-wmf.19] i/p/IndexPager.php:*  PHP Warning: implode(): Invalid arguments passed - https://phabricator.wikimedia.org/T244937 (10dbarratt) and @Catr...
[16:23:41] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Fix or delete integration-config-qa - https://phabricator.wikimedia.org/T245037 (10thcipriani)
[16:25:12] <wikibugs>	 (03PS1) 10Thcipriani: integration-config-qa: skip email for now [integration/config] - 10https://gerrit.wikimedia.org/r/571751 (https://phabricator.wikimedia.org/T245037)
[16:30:55] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job integration-config-qa" [integration/config] - 10https://gerrit.wikimedia.org/r/571751 (https://phabricator.wikimedia.org/T245037) (owner: 10Thcipriani)
[16:31:36] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Conpherence notifications about uninteresting events cannot be disabled - https://phabricator.wikimedia.org/T148587 (10Aklapper) 05Open→03Declined We do not use Conpherence anymore, hence declining this task.
[16:32:02] <wikibugs>	 (03Merged) 10jenkins-bot: integration-config-qa: skip email for now [integration/config] - 10https://gerrit.wikimedia.org/r/571751 (https://phabricator.wikimedia.org/T245037) (owner: 10Thcipriani)
[16:37:37] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Irrelevant project notifications if you are not subscribed but a watcher and 'Project watcher list changes' is enabled in notification preferences - https://phabricator.wikimedia.org/T105073 (10Aklapper) 05Stalled→03Invalid >>! In T105073#2144800, @Aklapper wrote: > As...
[16:39:34] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Allow varying notification settings (eg: notify vs email) by tag/project - https://phabricator.wikimedia.org/T165832 (10Aklapper) This is indeed currently not possible. https://www.mediawiki.org/wiki/Phabricator/Help/Managing_mail offers currently available options.  T7835...
[16:41:27] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Need a way to disable web notifications from certain accounts (phabricator_maintenance) - https://phabricator.wikimedia.org/T142918 (10Aklapper) For the records:  * In the meantime, admins can make certain actions "silent", but this is up to the person performing these act...
[16:43:45] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Allow disabling notifications for certain associated projects (sprints) on Maniphest tasks - https://phabricator.wikimedia.org/T110549 (10Aklapper) 05Open→03Declined You can filter your mail notifications on specific projects. See https://www.mediawiki.org/wiki/Phabric...
[16:43:57] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10DBA, 10Operations: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10mmodell) Hey @Marostegui, How about tomorrow? I can be around tomorrow, if you'd like. If you'd like to do it at your le...
[16:45:17] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10DBA, 10Operations: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10Marostegui) >>! In T244566#5877381, @mmodell wrote: > Hey @Marostegui, How about tomorrow? I can be around tomorrow, if...
[16:46:12] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Phabricator bots should not trigger notifications - https://phabricator.wikimedia.org/T198574 (10Aklapper) 05Open→03Declined > In the future, we may provide a way to mark certain API keys as "silent" from the CLI  For the records, this options exists nowadays, but it's...
[16:49:33] <wikibugs>	 (03CR) 10Dduvall: [C: 03+2] Allow helm chart overrides in deploy configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/570944 (https://phabricator.wikimedia.org/T244512) (owner: 10Dduvall)
[16:50:39] <wikibugs>	 (03Merged) 10jenkins-bot: Allow helm chart overrides in deploy configuration [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/570944 (https://phabricator.wikimedia.org/T244512) (owner: 10Dduvall)
[16:57:14] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release Pipeline, 10Patch-For-Review: Allow additional helm overrides in PipelineLib config - https://phabricator.wikimedia.org/T244512 (10dduvall) 05Open→03Resolved
[16:57:16] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release Pipeline, 10User-zeljkofilipin: Experiment with different PipelineLib-/helm-based approaches to system testing - https://phabricator.wikimedia.org/T244313 (10dduvall)
[16:58:08] <wikibugs>	 10Phabricator (Upstream), 10Upstream: Add reason to Phabricator emails why you received that notification - https://phabricator.wikimedia.org/T77917 (10Aklapper) This might be possible to some extent nowadays for mail notifications: * You can filter on values in the `X-Phabricator-Stamps` mail header, e.g. `ac...
[17:06:54] <wikibugs>	 (03Abandoned) 10Zoranzoki21: tox: Install tk [integration/config] - 10https://gerrit.wikimedia.org/r/571026 (https://phabricator.wikimedia.org/T240070) (owner: 10Zoranzoki21)
[17:11:37] <brennen>	 !log restarting jenkins for updates
[17:11:38] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[17:26:46] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Fix or delete integration-config-qa - https://phabricator.wikimedia.org/T245037 (10greg) Introduced in {7b0213a1756d} and then used for various things like {2bac0e28...
[17:29:13] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Arrange PGP keysigning at All Hands 2020 - https://phabricator.wikimedia.org/T242340 (10LarsWirzenius)
[17:46:26] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10akosiaris) >>! In T238658#5830499, @Ottomata wrote: > @akosiaris I just did a bit of benchmarking in stagi...
[17:55:36] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Core Platform Team, 10Goal, 10Patch-For-Review: Decide timing of the automatic mediawiki branch cut - https://phabricator.wikimedia.org/T242446 (10thcipriani) 05Open→03Resolved We've left th...
[17:55:38] <wikibugs>	 10Release-Engineering-Team (Deployment services), 10Release-Engineering-Team-TODO, 10Scap, 10Epic, 10Goal: Automate the Train - https://phabricator.wikimedia.org/T196515 (10thcipriani)
[17:57:19] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review, 10Release, 10Train Deployments: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867 (10Jdforrester-WMF)
[17:57:49] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Release, 10Train Deployments: 1.35.0-wmf.19 deployment blockers - https://phabricator.wikimedia.org/T233867 (10Jdforrester-WMF)
[17:59:09] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10local-charts, 10Developer Productivity, and 2 others: Add pipeline config and publish step for restbase dev docker image - https://phabricator.wikimedia.org/T234580 (10jeena) 05Open→03Declined Closing s...
[17:59:11] <wikibugs>	 10Release-Engineering-Team (Local Dev), 10Release-Engineering-Team-TODO, 10local-charts, 10Developer Productivity, 10Epic: Create official docker images for Mediawiki and services used in the local development environment - https://phabricator.wikimedia.org/T217872 (10jeena)
[18:07:40] <wikibugs>	 10Release-Engineering-Team, 10User-brennen: Read for EngProd bookclub IV:  What nobody tells you about documentation by Daniele Procida - https://phabricator.wikimedia.org/T245044 (10brennen)
[18:11:54] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen: Read for EngProd bookclub IV:  What nobody tells you about documentation by Daniele Procida - https://phabricator.wikimedia.org/T245044 (10brennen)
[18:16:01] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10Patch-For-Review, 10User-LokalProfil: Add Kalle to trusted reviewers - https://phabricator.wikimedia.org/T244839 (10Jdforrester-WMF) a:03Jdforrester-WMF
[18:16:34] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10WMSE-Wikispeech-Speech-Data-Collector-2019, 10Patch-For-Review, 10User-LokalProfil: Add Kalle to trusted reviewers - https://phabricator.wikimedia.org/T244839 (10Jdforrester-WMF) p:05Triage→03Medium
[18:18:41] <wikibugs>	 (03PS2) 10Jforrester: Support for logging json to syslog [tools/scap] - 10https://gerrit.wikimedia.org/r/563468 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi)
[18:18:52] <wikibugs>	 (03CR) 10Jforrester: [C: 03+2] Support for logging json to syslog [tools/scap] - 10https://gerrit.wikimedia.org/r/563468 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi)
[18:22:13] <wikibugs>	 (03Merged) 10jenkins-bot: Support for logging json to syslog [tools/scap] - 10https://gerrit.wikimedia.org/r/563468 (https://phabricator.wikimedia.org/T86969) (owner: 10Filippo Giunchedi)
[18:29:35] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Refine Seakeeper proposal for Security/SRE review - https://phabricator.wikimedia.org/T243436 (10thcipriani) p:05High→03Medium
[18:29:40] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Research available third-party Argo and Kubernetes cloud providers - https://phabricator.wikimedia.org/T244384 (10thcipriani) p:05Triage→03High
[18:31:47] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10DBA, 10Operations: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10mmodell) @Marostegui Thursday 6:00 AM works for me.
[18:33:00] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Development services), 10DBA, 10Operations: Upgrade and restart m3 (phabricator) master (db1128) - https://phabricator.wikimedia.org/T244566 (10Marostegui) >>! In T244566#5877826, @mmodell wrote: > @Marostegui Thursday 6:00 AM works for me.   Excellent! See you to...
[18:36:45] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Incident: Document emergency deploy proceedures - https://phabricator.wikimedia.org/T245046 (10Jdforrester-WMF)
[18:39:46] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen, 10User-greg: Instrument Zuul/geard to gather the statistics we need to make capacity decisions - https://phabricator.wikimedia.org/T245047 (10greg)
[18:42:45] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Incident: Document emergency deploy proceedures - https://phabricator.wikimedia.org/T245046 (10greg) lgtm
[18:43:15] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen, 10User-greg: Instrument Zuul/geard to gather the statistics we need to make capacity decisions - https://phabricator.wikimedia.org/T245047 (10greg) p:05Triage→03Medium
[18:48:57] <wikibugs>	 10Release-Engineering-Team (Pipeline), 10Analytics, 10Analytics-Kanban, 10Release Pipeline, and 2 others: Migrate EventStreams to k8s deployment pipeline - https://phabricator.wikimedia.org/T238658 (10Ottomata) Ok thanks!  Will try that!
[18:58:20] <wikibugs>	 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10User-brennen, 10User-greg: Instrument Zuul/geard to gather the statistics we need to make capacity decisions - https://phabricator.wikimedia.org/T245047 (10greg) a:05greg→03None
[18:58:32] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)): Fix or delete integration-config-qa - https://phabricator.wikimedia.org/T245037 (10thcipriani) p:05Triage→03Low
[18:59:30] <wikibugs>	 10Release-Engineering-Team-TODO, 10Release Pipeline, 10serviceops-radar: Create initial production MW-land images with blubber - https://phabricator.wikimedia.org/T238773 (10thcipriani)
[19:00:14] <wikibugs>	 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Wikimedia-Incident: Document emergency deploy procedures - https://phabricator.wikimedia.org/T245046 (10Aklapper)
[19:33:13] <davidwbarratt>	 hello! this seems to failed on merge.... what should I do to try it again? https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/570698
[19:34:47] <Reedy>	 davidwbarratt: I think wait for the recheck, then possibly kill the C+2 and re-apply it
[19:35:01] <davidwbarratt>	 ah, ok, will do
[19:35:11] <Reedy>	 It looks like it didn't close down properly
[19:46:48] <Reedy>	 It failed again
[19:47:30] <Reedy>	 12:34:23 FATAL: Remote call on integration-agent-docker-1001 failed
[19:47:30] <Reedy>	 12:34:23 Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to integration-agent-docker-1001
[19:48:52] <Reedy>	 Apparently I can't view https://integration.wikimedia.org/ci/computer/integration-agent-docker-1001/builds
[19:50:13] <Reedy>	 James_F: ^
[19:50:29] <James_F>	 Interesting.
[19:50:31] <wikibugs>	 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-Installer: web installer / upgrade doesn't seem to recognize extension.json configuration or overrides to that configuration - https://phabricator.wikimedia.org/T244864 (10WDoranWMF) @kostajh Could you give us a sense of the priority level here? I...
[19:50:43] <wikibugs>	 10Continuous-Integration-Infrastructure (phase-out-jessie), 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Patch-For-Review: Determine if it is possible to migrate remaining integration jessie permanent instances to stretch/buster - https://phabricator.wikimedia.org/T242909 (10jeena) Determined...
[19:50:58] <James_F>	 Reedy: But https://integration.wikimedia.org/ci/computer/integration-agent-docker-1001/ renders OK.
[19:51:09] <Reedy>	 Yeah
[19:51:19] <Reedy>	 I was trying to view /builds to see if it was failing numerous builds etc
[19:51:20] <James_F>	 Possibly broken by the jenkins upgrade today?
[19:51:26] <Reedy>	 Or it just hates checkuser etc
[19:51:42] <James_F>	 Could be.
[19:51:56] <James_F>	 Is it just integration-agent-docker-1001??
[19:53:28] <Reedy>	 Not seen anyone else complaining (yet)
[19:55:39] <James_F>	 Hmm.
[19:57:14] <davidwbarratt>	 oh yeah, just noticed the additional failure https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/570698
[19:58:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10DannyS712)
[19:58:55] <wikibugs>	 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group: web installer / upgrade doesn't seem to recognize extension.json configuration or overrides to that configuration - https://phabricator.wikimedia.org/T244864 (10kostajh) Hi @WDoranWMF not blocking anyth...
[19:59:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10DannyS712)
[20:19:09] <davidwbarratt>	 any ideas?
[20:22:48] <James_F>	 !log Depooling integration-agent-docker-1001; it's being odd T245054
[20:22:51] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[20:22:51] <stashbot>	 T245054: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054
[20:23:28] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10Zuul, 10User-DannyS712: Remote call on integration-agent-docker-1001 failed - https://phabricator.wikimedia.org/T245054 (10Jdforrester-WMF) p:05Triage→03High I've de-pooled it for now.
[20:27:23] * James_F idly wonders whatever happened to integration-agent-docker-1015.
[20:30:38] <davidwbarratt>	 should I do a rechck now?
[20:39:20] <wikibugs>	 10Release-Engineering-Team, 10Core Platform Team, 10MediaWiki-Installer, 10MediaWiki-Stakeholders-Group: web installer / upgrade doesn't seem to recognize extension.json configuration or overrides to that configuration - https://phabricator.wikimedia.org/T244864 (10greg) >>! In T244864#5878118, @WDoranWMF...
[20:46:35] <volans>	 Hi, by any chance any news on adding CI to the operations/homer/mock-private repository that hit a weird behaviour?
[20:46:39] <volans>	 for context: https://phabricator.wikimedia.org/T244690#5867118
[20:49:56] <davidwbarratt>	 I'm getting a pre-merge failure on https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CheckUser/+/571794 now 
[20:58:56] <thcipriani>	 davidwbarratt: that looks like https://tools.wmflabs.org/sal/log/AXA7EYfgvrJzePItxDaf
[20:59:17] <thcipriani>	 > FATAL: Remote call on integration-agent-docker-1001 failed
[20:59:52] <thcipriani>	 running a recheck
[21:00:12] <Amir1>	 thcipriani: nope, I get it too, I just came here to say it
[21:00:19] <Amir1>	 ran recheck as well
[21:00:27] <Amir1>	 https://gerrit.wikimedia.org/r/#/c/570391/
[21:00:31] * thcipriani looks
[21:04:47] <thcipriani>	 Amir1: yep, those last 2 builds of yours were running on integration-agent-docker-1001 which should now be out of the pool afaik the other machines in the pool should be acting normally
[21:04:51] <thcipriani>	 a recheck *should* work (although I also see 'Error: Cannot find module \'@storybook/addon-knobs\'\n , but I don't know if that's a problem or not)
[21:05:24] <Amir1>	 oh okay
[21:23:21] <James_F>	 volans: Sorry, no. :-(
[21:23:49] <James_F>	 thcipriani: Don't suppose you have any idea what "zuul.rpcclient.RPCFailure: Invalid change: 540622,1" means?
[21:23:58] <James_F>	 (Context is T244690)
[21:23:58] <stashbot>	 T244690: Homer: setup CI for static data repositories - https://phabricator.wikimedia.org/T244690
[21:24:27] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10homer, 10Patch-For-Review: Homer: setup CI for static data repositories - https://phabricator.wikimedia.org/T244690 (10thcipriani) Looked briefly at this today with https://gerrit.wikimedia.org/r/#/c/operations/ho...
[21:24:31] <thcipriani>	 James_F: where are you seeing that?
[21:24:54] <James_F>	 thcipriani: In  /var/log/zuul/zuul.log yesterday.
[21:25:00] <James_F>	 I poked but got nowhere.
[21:25:15] <volans>	 James_F: ack, if it can help that repo is not critical so we can mangle it if needed (force push and recreate from scratch included)
[21:30:13] <thcipriani>	 https://gerrit.wikimedia.org/r/plugins/gitiles/integration/zuul/+/patch-queue/debian/jessie-wikimedia/zuul/rpcclient.py#47
[21:30:26] <thcipriani>	 is the only place that gets raised
[21:30:46] <thcipriani>	 but! that may have been a red-herring
[21:31:02] <thcipriani>	 I am not seeing zuul react at all when I comment "recheck" on a patch for that repo
[21:31:15] <thcipriani>	 even though I see the event in stream-events
[21:31:55] <James_F>	 Indeed.
[21:36:35] <thcipriani>	 !log reloading zuul in an attempt to troubleshoot T244690
[21:36:37] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[21:36:37] <stashbot>	 T244690: Homer: setup CI for static data repositories - https://phabricator.wikimedia.org/T244690
[21:38:38] <thcipriani>	 huh, no joy
[21:38:46] <volans>	 :(
[21:39:46] <thcipriani>	 and nothing in any log file about it
[21:40:09] <thcipriani>	 might just wait until zuul is quiet and restart: that's my next troubleshooting idea
[21:40:52] <volans>	 the only thing I know it's slightly different about this repo is that it was inheriting permissions from labs/private
[21:40:54] <James_F>	 Did we try that?
[21:41:03] <volans>	 not sure if might have any connection
[21:41:25] <James_F>	 volans: Maybe? But seems very odd.
[21:42:03] <paladox|UKInEU>	 that would, i think?
[21:42:07] <paladox|UKInEU>	 it blocks "Registered Users"
[21:42:27] <paladox|UKInEU>	 i doin't see an exemption for JenkinsBot
[21:43:25] <thcipriani>	 oh! maybe.
[21:43:33] <volans>	 it's added in the repo itself for submit
[21:43:33] <thcipriani>	 isn't labs/private confusingly public though?
[21:43:36] <volans>	 https://gerrit.wikimedia.org/r/admin/projects/operations/homer/mock-private,access
[21:43:44] <James_F>	 Ha.
[21:43:52] <paladox|UKInEU>	 thcipriani it allows ldap/project-bastion apparently
[21:44:04] <James_F>	 That's… unhelpful.
[21:44:07] <James_F>	 Yeah, that'd do it.
[21:44:25] <James_F>	 Should we fix labs/private or just operations/home/mock-private?
[21:45:55] <paladox|UKInEU>	 I say fix it for the later, unless there's plan to hook more repos (that use jenkins) to labs/private
[21:46:29] <James_F>	 Okie-dokie.
[21:46:30] <paladox|UKInEU>	 oh! I doin't think sub repos can ovveride a bloc 
[21:46:37] <paladox|UKInEU>	 *block
[21:47:05] <volans>	 if it creates so much trouble changing the parent is an option?
[21:47:06] <James_F>	 Ah. In that case, we're a bit lost unless we fix labs/private, but that might be bad.
[21:47:14] <James_F>	 volans: Might be for the best.
[21:47:22] <paladox|UKInEU>	 ^
[21:48:22] <volans>	 feel free, I can't change it
[21:48:30] <volans>	 you can use operations/homer as new parent
[21:48:53] <volans>	 (can't as in it tells me I don't have the permissions to)
[21:49:09] <paladox|UKInEU>	 yeh, only admins can.
[21:51:05] <James_F>	 Which is roots only, now.
[21:52:07] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10homer, 10Patch-For-Review: Homer: setup CI for static data repositories - https://phabricator.wikimedia.org/T244690 (10Jdforrester-WMF) We worked it out; this repo is inheriting its ACL from `labs/private`, which...
[22:01:14] * thcipriani changes parent to operations/homer
[22:02:18] <volans>	 thanks!
[22:02:21] <thcipriani>	 success!
[22:02:35] <James_F>	 Roots only and thcipriani, apparently. :-)
[22:02:49] <volans>	 great!!!
[22:03:03] <volans>	 sorry for the trouble, and thanks a lot for the extensive debugging
[22:05:25] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team-TODO (2020-01 to 2020-03 (Q3)), 10homer, 10Patch-For-Review: Homer: setup CI for static data repositories - https://phabricator.wikimedia.org/T244690 (10Jdforrester-WMF) 05Open→03Resolved a:05Volans→03thcipriani
[22:10:51] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mwmaint01 is CRITICAL: CRITICAL: deployment-prep.deployment-mwmaint01.diskspace.root.byte_percentfree (<10.00%)
[22:20:50] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mwmaint01 is OK: OK: All targets OK
[22:39:57] <wikibugs>	 (03PS1) 10Zoranzoki21: Add *.microsoft.com in email_whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/571818
[22:49:19] <wikibugs>	 (03CR) 10Greg Grossmeier: [C: 04-1] "Is there a need? Why not @apple.com? Or any other random organization?" [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[22:59:07] <Reedy>	 greg-g: I'm presuming because we've got a few different MS people proposing patches here and there...
[22:59:43] <wikibugs>	 (03CR) 10Greg Grossmeier: [C: 04-1] "If there's a need I'm happy to allow it, I just haven't seen it." [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[23:01:24] <wikibugs>	 (03CR) 10Reedy: "I'm guessing related to https://gerrit.wikimedia.org/r/#/q/author:%2540microsoft.com" [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[23:04:54] <wikibugs>	 (03CR) 10Greg Grossmeier: "Thanks Reedy, I should have looked first." [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[23:08:06] <wikibugs>	 (03CR) 10Reedy: "Heh. There's a few MS devs submitting tasks about accessibility issues too" [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[23:08:53] <wikibugs>	 (03CR) 10Zoranzoki21: "> Is there a need? Why not @apple.com? Or any other random" [integration/config] - 10https://gerrit.wikimedia.org/r/571818 (owner: 10Zoranzoki21)
[23:18:24] <wikibugs>	 (03PS1) 10Kosta Harlan: (wip) Fix XDebug configuration [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571828
[23:31:06] <wikibugs>	 (03CR) 10Kosta Harlan: "needs some tuning still, but just a heads up on this." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571828 (owner: 10Kosta Harlan)
[23:35:14] <wikibugs>	 (03CR) 10Brennen Bearnes: "> needs some tuning still, but just a heads up on this." [releng/dev-images] - 10https://gerrit.wikimedia.org/r/571828 (owner: 10Kosta Harlan)
[23:58:22] <wikibugs>	 10Release-Engineering-Team, 10Release-Engineering-Team-TODO, 10Core Platform Team: Audit all existing code to ensure that any extension currently or previously adding blobs to ExternalStore has been registering a reference in the text table (and fix up if wrong) - https://phabricator.wikimedia.org/T106388 (10...