[00:03:34] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10LucasWerkmeister) >>! In T191182#6354229, @valerio.bozzolan wrote: > What I'm saying //here// is that we have a... [06:16:00] PROBLEM - Free space - all mounts on deployment-snapshot01 is CRITICAL: CRITICAL: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found)deployment-prep.deployment-snapshot01.diskspace.root.byte_percentfree (<10.00%) [06:26:02] RECOVERY - Free space - all mounts on deployment-snapshot01 is OK: OK: deployment-prep.deployment-snapshot01.diskspace._data.byte_percentfree (No valid datapoints found) [06:55:36] (03CR) 10JMeybohm: [C: 03+1] "I would maybe add a comment on why those directories need to be 777 (as it looks suspicious in first place). But LGTM like this as well." [integration/config] - 10https://gerrit.wikimedia.org/r/617459 (owner: 10Giuseppe Lavagetto) [06:58:40] 10Continuous-Integration-Config, 10Gerrit, 10User-DannyS712: Jenkins no longer rebases with `action = rebase if necessary` - https://phabricator.wikimedia.org/T259450 (10DannyS712) [07:00:51] Project beta-scap-eqiad build #311348: 04FAILURE in 5 min 59 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/311348/ [07:02:53] (03CR) 10Giuseppe Lavagetto: [C: 03+2] Add envoy to the helm-linter image [integration/config] - 10https://gerrit.wikimedia.org/r/617459 (owner: 10Giuseppe Lavagetto) [07:09:19] <_joe_> !log upgrade helm-linter image [07:09:20] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:15:14] Yippee, build fixed! [07:15:15] Project beta-scap-eqiad build #311349: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/311349/ [07:21:38] 10Continuous-Integration-Config, 10Gerrit, 10User-DannyS712: Jenkins sometimes doesn't rebase with `action = rebase if necessary` - https://phabricator.wikimedia.org/T259450 (10DannyS712) [07:25:34] 10Continuous-Integration-Config, 10Gerrit, 10User-DannyS712: Jenkins sometimes doesn't rebase with `action = rebase if necessary` - https://phabricator.wikimedia.org/T259450 (10RhinosF1) I've seen this before on rare occassions in operations/mediawiki-config. It was a while ago though so can't remember when. [07:51:55] (03PS1) 10Giuseppe Lavagetto: jjb: use the new helm-linter image [integration/config] - 10https://gerrit.wikimedia.org/r/618003 [07:52:21] <_joe_> hashar: what do I need to do to make ^^ apply? [07:54:24] <_joe_> also, the fabfile in that repo doesn't work with fabric 2.5.x [07:55:48] <_joe_> oh I see you converted it? [08:00:28] > INFO:jenkins_jobs.builder:Reconfiguring jenkins job helm-lint [08:00:58] (03CR) 10Legoktm: [C: 03+2] "INFO:jenkins_jobs.builder:Reconfiguring jenkins job helm-lint" [integration/config] - 10https://gerrit.wikimedia.org/r/618003 (owner: 10Giuseppe Lavagetto) [08:01:14] _joe_: tox -e jenkins-jobs -- --conf jenkins_jobs.ini update jjb/ helm-lint [08:01:48] <_joe_> legoktm: on contint1001? [08:01:57] no, locally [08:02:02] <_joe_> wut [08:02:06] (03Merged) 10jenkins-bot: jjb: use the new helm-linter image [integration/config] - 10https://gerrit.wikimedia.org/r/618003 (owner: 10Giuseppe Lavagetto) [08:02:09] <_joe_> I don't want to look inside :P [08:02:19] jenkins (jjb) is updated over the API [08:02:30] zuul is reloaded via contint1001 [08:02:53] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Deploy_changes [08:02:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-dancy, 10User-zeljkofilipin: Selenium quibble jobs have a huge cac... - https://phabricator.wikimedia.org/T258972 [08:03:34] <_joe_> requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): [08:03:41] <_joe_> seems like I need some config [08:04:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [08:04:14] https://www.mediawiki.org/wiki/Continuous_integration/Jenkins_job_builder#Configure_JJB [08:04:16] <_joe_> oh right yes [08:04:19] <_joe_> found it [08:04:38] <_joe_> sigh seriously a file within the repo [08:04:52] <_joe_> ok :) [08:04:55] <_joe_> thanks legoktm [08:05:03] np :) [08:06:11] <_joe_> !log updating helm-lint job [08:06:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:06:58] <_joe_> uhm looks like I don't have the authorization to do so [08:07:25] <_joe_> !log nevermind - I don't have the necessary permissions :) [08:07:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:07:33] well I also already did it for you :p [08:07:41] <_joe_> oh lol [08:07:44] <_joe_> ok [08:11:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-dancy, 10User-zeljkofilipin: Selenium quibble jobs have a huge cac... - https://phabricator.wikimedia.org/T258972 [08:14:15] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.92 ms [08:20:50] 10Release-Engineering-Team, 10MediaWiki-General, 10MediaWiki-Stakeholders-Group, 10serviceops, and 3 others: Drop official PHP 7.2 support in MediaWiki 1.35 - https://phabricator.wikimedia.org/T257879 (10Jdforrester-WMF) [08:24:13] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [08:28:27] PROBLEM - Host deployment-changeprop is DOWN: CRITICAL - Host Unreachable (172.16.5.21) [08:28:27] PROBLEM - Host deployment-cpjobqueue is DOWN: CRITICAL - Host Unreachable (172.16.4.124) [08:28:27] PROBLEM - Host deployment-chromium02 is DOWN: CRITICAL - Host Unreachable (172.16.4.14) [08:42:46] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10valerio.bozzolan) >>! In T191182#6354533, @LucasWerkmeister wrote: > The community has had plenty of time to ap... [08:56:11] (03PS1) 10Alex Mashin: Edit Repo Config [extensions/ExternalData] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618026 [08:57:43] (03PS1) 10Alex Mashin: Edit Repo Config [extensions/ExternalData] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618027 [08:58:41] (03PS1) 10Alex Mashin: Edit Repo Config [extensions/ExternalData] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618028 [09:32:59] 10Release-Engineering-Team (Pipeline), 10CX-cxserver, 10serviceops, 10Language-Team (Language-2020-July-September): Migrate apertium to the deployment pipeline - https://phabricator.wikimedia.org/T255672 (10Pginer-WMF) [11:00:37] PROBLEM - Host deployment-mcs01 is DOWN: CRITICAL - Host Unreachable (172.16.5.64) [11:16:31] 10Gerrit: Triple-clicking Gerrit change subject selects unwanted space at the beginning - https://phabricator.wikimedia.org/T219809 (10Lucas_Werkmeister_WMDE) Update: since the Gerrit 3 upgrade (T254158), triple-clicking no longer seems to work at all. For some reason. (Not just in the commit message, but also e... [12:31:16] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Logstash looks fairly OK, though noted {T257504} is happening again... [12:39:13] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.09 ms [12:49:15] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [13:30:10] Project beta-scap-eqiad build #311393: 04FAILURE in 5 min 36 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/311393/ [13:39:35] Project beta-scap-eqiad build #311394: 04STILL FAILING in 5 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/311394/ [13:45:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-dancy, 10User-zeljkofilipin: Selenium quibble jobs have a huge cac... - https://phabricator.wikimedia.org/T258972 [13:48:31] Yippee, build fixed! [13:48:31] Project beta-scap-eqiad build #311395: 09FIXED in 3 min 54 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/311395/ [14:34:29] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10cscott) It looks like the train was rolled without the fix for T259311, which got... [14:42:41] (03PS1) 10Hashar: docker: point XDG_CONFIG_HOME to a subdirectory of /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/618079 (https://phabricator.wikimedia.org/T220948) [14:51:31] (03PS1) 10Hashar: docker: remove obsolete chromium=71 version pinning [integration/config] - 10https://gerrit.wikimedia.org/r/618082 (https://phabricator.wikimedia.org/T216702) [14:52:50] (03CR) 10Ahmon Dancy: [C: 03+1] docker: point XDG_CONFIG_HOME to a subdirectory of /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/618079 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [15:15:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Patch-For-Review, and 2 others: Selenium quibble jobs have a huge cache o... - https://phabricator.wikimedia.org/T258972 [15:17:18] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10mmodell) ^ merged the other patch. Is there anything else to do here or should this t... [15:17:57] 10Continuous-Integration-Config, 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10User-Vidhi-Mody, 10User-zeljkofilipin: CI not running for phabricator/deployment - https://phabricator.wikimedia.org/T258531 (10mmodell) We don't have any CI for this repo. [15:23:17] 10Phabricator: Custom task forms for #WMNO-General - https://phabricator.wikimedia.org/T258787 (10mmodell) @aklapper: yes, duplicate form is the way to go. It's a customization I added which simply copies all of the existing form configuration so that you don't have to start from scratch each time. [15:36:15] 10VPS-project-codesearch: Codesearch beta: execute search when navigating through repository links - https://phabricator.wikimedia.org/T259364 (10sbassett) Yep, works great. Thanks. [15:36:31] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10Vidhi-Mody) [15:37:24] 10Phabricator, 10Quality-and-Test-Engineering-Team (QTE), 10Patch-For-Review, 10User-Vidhi-Mody, 10User-zeljkofilipin: Upgrade WebdriverIO in the phab-deployment repository - https://phabricator.wikimedia.org/T255471 (10Vidhi-Mody) 05Open→03Resolved a:03Vidhi-Mody @mmodell nothing else needs to be... [15:46:27] !log `service puppetdb start` on deployment-puppetdb03.deployment-prep.eqiad.wmflabs. Looks like it died from OOM [15:46:29] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:52:40] 10Beta-Cluster-Infrastructure, 10Puppet: MIssing hiera settings for deployment-parsoid11.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T259533 (10bd808) [15:52:43] 10Phabricator, 10Mail: Duplicate weekly Phabricator emails (cronjob ran twice?) - https://phabricator.wikimedia.org/T258371 (10Dzahn) a:03Dzahn [15:54:16] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 3.09 ms [16:02:43] 10Phabricator, 10Mail: Duplicate weekly Phabricator emails (cronjob ran twice?) - https://phabricator.wikimedia.org/T258371 (10Dzahn) 05Open→03Resolved Yes, both project_changes.sh and community_metrics.sh had a duplicate cron entry. Removed all 4 manually and ran puppet again which recreated just 2. Thi... [16:07:11] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Development services), 10Release-Engineering-Team-TODO: Stop using Differential for code review - https://phabricator.wikimedia.org/T191182 (10Dzahn) > Not using three or more different code hosting and review tools (Gerrit, Differential, Github, random... [16:09:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [16:48:26] 10Beta-Cluster-Infrastructure, 10Puppet: puppetdb on deployment-puppetdb03 keeps getting OOMKilled - https://phabricator.wikimedia.org/T248041 (10bd808) Another restart on 2020-08-03: https://sal.toolforge.org/log/n4IAtXMBj_Bg1xd3rkvT [16:49:31] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Filed {T259536}. [16:51:28] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10brennen) Seeing several T259536 a minute. I don't know the impact there, but it's... [16:52:57] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Agreed, rolling back. [16:58:55] (03CR) 10Ahmon Dancy: [C: 04-1] jjb: remove legacy castor job [integration/config] - 10https://gerrit.wikimedia.org/r/617108 (owner: 10Hashar) [17:06:44] (03PS2) 10Ahmon Dancy: jjb: remove legacy castor job [integration/config] - 10https://gerrit.wikimedia.org/r/616594 [17:06:46] (03PS2) 10Ahmon Dancy: Move castor-save-filter.bash check into castor-save-workspacecache.bash [integration/config] - 10https://gerrit.wikimedia.org/r/616595 [17:07:20] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10brennen) [17:11:37] (03PS1) 10Dduvall: Pass Helm overrides as YAML values file. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) [17:25:27] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10brennen) [17:28:19] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10LarsWirzenius) Ok, rolling forward again. [17:57:49] 10Beta-Cluster-Infrastructure: deployment-perfapt01 seems to be broken - https://phabricator.wikimedia.org/T259540 (10Krenair) [18:01:14] 10Beta-Cluster-Infrastructure, 10cloud-services-team (Kanban): non-wdqs VMs sometimes getting scheduled on wdqs hardware - https://phabricator.wikimedia.org/T259542 (10Andrew) [18:03:17] (03CR) 10Ahmon Dancy: [C: 04-1] Pass Helm overrides as YAML values file. (033 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [18:08:54] (03PS2) 10Dduvall: Pass Helm overrides as YAML values file. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) [18:10:46] (03CR) 10Dduvall: Pass Helm overrides as YAML values file. (032 comments) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [18:11:11] (03CR) 10Ahmon Dancy: [C: 03+1] Pass Helm overrides as YAML values file. (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [18:11:28] (03CR) 10Dduvall: Pass Helm overrides as YAML values file. (031 comment) [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [18:13:21] 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release, 10Train Deployments, 10User-brennen: 1.36.0-wmf.2 deployment blockers - https://phabricator.wikimedia.org/T257970 (10ssastry) [18:21:54] 10Gerrit, 10Patch-For-Review: Use WMF Gerrit's logo as favicon - https://phabricator.wikimedia.org/T257218 (10QChris) The poll has ended. 54 people voted. The green git logo got most approval (57%) followed by Gerrit's standard icon (50%) and the Community-style logo with Git icon (42%). The rest is far off (... [18:22:09] 10Beta-Cluster-Infrastructure: deployment-perfapt01 seems to be broken - https://phabricator.wikimedia.org/T259540 (10dpifke) I found another way to test what I was working on, and got pulled into other projects so forgot to come back and troubleshoot what went wrong here - sorry. I can delete it, or leave it a... [18:32:58] hello [18:33:18] (03CR) 10Hashar: [C: 03+2] "Lets build!" [integration/config] - 10https://gerrit.wikimedia.org/r/618079 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [18:34:18] (03Merged) 10jenkins-bot: docker: point XDG_CONFIG_HOME to a subdirectory of /tmp [integration/config] - 10https://gerrit.wikimedia.org/r/618079 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [18:38:11] oh my [18:40:42] twentyafterfour: is there a way to set a user agent using the mediawiki OAuth login phab extension? [18:40:48] Me and paladox couldn't see it [18:41:51] (03CR) 10Hashar: "Changes made to dockerfiles do not automatically trigger a build of the images, they have to be build using docker-pkg on contint2001 whic" [integration/config] - 10https://gerrit.wikimedia.org/r/617459 (owner: 10Giuseppe Lavagetto) [18:44:04] Fetched 11.7 MB in 14s (808 kB/s) [18:44:32] turns out fetching mirrors.wikimedia.org material from a production host is SLOWER than from my local machine ;D [18:44:44] * hashar ignores the issue [18:46:59] can't confirm at all. (85.5 MB/s) - ‘ls-lR.gz’ saved [18:47:53] I blame docker ;D [18:48:09] sounds like a good guess [18:48:18] mutante: I still have to look at your email about gerrit ssh key update [18:49:07] (03PS1) 10Ahmon Dancy: Report SUCCESS when no zuul layout diffs [integration/config] - 10https://gerrit.wikimedia.org/r/618129 [18:50:00] dancy: <3 [18:50:27] ok, nppp [18:50:38] hashar: mtail is now running on contint* [19:03:39] (03CR) 10Hashar: [C: 03+2] Report SUCCESS when no zuul layout diffs (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/618129 (owner: 10Ahmon Dancy) [19:04:10] mutante: cool :) dancy is looking in adding monitoring of /var/log/zuul/zuul.error.log :] [19:04:28] (03Merged) 10jenkins-bot: Report SUCCESS when no zuul layout diffs [integration/config] - 10https://gerrit.wikimedia.org/r/618129 (owner: 10Ahmon Dancy) [19:11:13] !log Reloaded Zuul for I215ee6238932be041bff6fa6cc453dc4cfa9512f [19:11:14] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [19:17:43] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/DrawioEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618136 [19:17:45] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [extensions/DrawioEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618136 (owner: 10QChris) [19:17:49] (03PS1) 10QChris: Import done. Revoke import grants [extensions/DrawioEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618137 [19:17:51] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [extensions/DrawioEditor] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618137 (owner: 10QChris) [19:24:32] (03Abandoned) 10Hashar: jjb: remove legacy castor job [integration/config] - 10https://gerrit.wikimedia.org/r/617108 (owner: 10Hashar) [19:25:20] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [analytics/wmde/TW/template-survey] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618138 [19:25:24] (03CR) 10QChris: [V: 03+2 C: 03+2] Allow “Gerrit Managers” to import history [analytics/wmde/TW/template-survey] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618138 (owner: 10QChris) [19:25:28] (03PS1) 10QChris: Import done. Revoke import grants [analytics/wmde/TW/template-survey] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618139 [19:25:32] (03CR) 10QChris: [V: 03+2 C: 03+2] Import done. Revoke import grants [analytics/wmde/TW/template-survey] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/618139 (owner: 10QChris) [19:28:09] (03CR) 10Hashar: [C: 03+2] "And it is a noop for the other job, from integration-config-jjb-diff-docker:" [integration/config] - 10https://gerrit.wikimedia.org/r/616594 (owner: 10Ahmon Dancy) [19:29:16] (03Merged) 10jenkins-bot: jjb: remove legacy castor job [integration/config] - 10https://gerrit.wikimedia.org/r/616594 (owner: 10Ahmon Dancy) [19:46:51] (03CR) 10Jeena Huneidi: "This looks good to me, but I think this will affect anyone who wants to use environment variables (such as those inserted as credentials) " [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [19:50:08] 10Beta-Cluster-Infrastructure, 10Commons, 10MediaWiki-File-management, 10Wikimedia-SVG-rendering: Test resvg on Beta Cluster - https://phabricator.wikimedia.org/T243893 (10AntiCompositeNumber) The beta cluster still uses production Thumbor for thumbnail rendering. Testing resvg via MediaWiki on the Beta cl... [19:54:19] (03CR) 10Hashar: "I have build them all." [integration/config] - 10https://gerrit.wikimedia.org/r/618079 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [19:56:13] (03CR) 10Jeena Huneidi: "> Patch Set 2:" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [20:01:00] (03CR) 10Jeena Huneidi: [C: 03+2] Pass Helm overrides as YAML values file. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [20:01:39] (03Merged) 10jenkins-bot: Pass Helm overrides as YAML values file. [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618098 (https://phabricator.wikimedia.org/T259318) (owner: 10Dduvall) [20:02:49] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline, 10Patch-For-Review: PipelineLib fails on non-string values for deploy.overrides - https://phabricator.wikimedia.org/T259318 (10dduvall) 05Open→03Res... [20:05:14] (03PS1) 10Hashar: jjb: use image to prevent chromium cache pollution [integration/config] - 10https://gerrit.wikimedia.org/r/618144 (https://phabricator.wikimedia.org/T220948) [20:07:14] (03CR) 10Hashar: [C: 03+2] "Jobs updated" [integration/config] - 10https://gerrit.wikimedia.org/r/618144 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [20:08:20] !log Updating various jobs to fix a cache pollution caused by Chromium. https://gerrit.wikimedia.org/r/618144 [20:08:28] (03Merged) 10jenkins-bot: jjb: use image to prevent chromium cache pollution [integration/config] - 10https://gerrit.wikimedia.org/r/618144 (https://phabricator.wikimedia.org/T220948) (owner: 10Hashar) [20:08:49] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:16:11] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-dancy, 10User-zeljkofilipin: Selenium quibble jobs have a huge cac... - https://phabricator.wikimedia.org/T258972 [20:16:28] so that task well [20:16:29] killed me [20:16:47] I am off and going to get some rest :] [20:18:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (CI & Testing services), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10User-dancy, 10User-zeljkofilipin: Selenium quibble jobs have a huge cac... - https://phabricator.wikimedia.org/T258972 [20:31:34] 10Phabricator: Spaces request for ...Tech Sr Leadership - https://phabricator.wikimedia.org/T259156 (10debt) Hi @Aklapper! Thanks so much for all your help and very sage advice! I have a couple asks - can you make me owner of those two private spaces? I'd like to go in and close out all the tickets that we're... [21:42:07] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [22:25:08] (03PS1) 10Dduvall: Purge Helm releases that fail to install [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/618156 (https://phabricator.wikimedia.org/T259319) [22:33:52] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline, 10Patch-For-Review: Ensure failed helm deployments are cleaned up by PipelineLib - https://phabricator.wikimedia.org/T259319 (10dduvall) p:05Triage→... [22:34:02] 10Release-Engineering-Team (Pipeline), 10Release-Engineering-Team-TODO (Release-Engineering-Team-TODO (2020-07-01 to 2020-09-30 (Q1))), 10Release Pipeline, 10Patch-For-Review: Ensure failed helm deployments are cleaned up by PipelineLib - https://phabricator.wikimedia.org/T259319 (10dduvall) a:03dduvall [22:34:14] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 1.90 ms [22:39:14] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [23:19:16] RECOVERY - Host deployment-xhgui02 is UP: PING OK - Packet loss = 0%, RTA = 0.64 ms [23:24:11] PROBLEM - Host deployment-xhgui02 is DOWN: CRITICAL - Host Unreachable (172.16.1.202) [23:54:19] PROBLEM - Host deployment-sentry01 is DOWN: CRITICAL - Host Unreachable (172.16.5.16) [23:57:08] 10Beta-Cluster-Infrastructure, 10Puppet: puppetdb on deployment-puppetdb03 keeps getting OOMKilled - https://phabricator.wikimedia.org/T248041 (10Krenair) a:03Krenair replacing with a medium instance, deployment-puppetdb04