[00:02:04] <wikibugs>	 (03Merged) 10jenkins-bot: Codehealth pipeline: Ignore exit code of npm test:unit [integration/config] - 10https://gerrit.wikimedia.org/r/509943 (https://phabricator.wikimedia.org/T218598) (owner: 10Kosta Harlan)
[00:02:48] <shinken-wm>	 PROBLEM - Puppet errors on deployment-deploy02 is CRITICAL: CRITICAL: 2.22% of data above the critical threshold [3.0]
[00:11:33] <wikibugs>	 (03PS1) 10Dduvall: pipeline: Allow for execution of a single pipeline [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510274
[00:12:20] <wikibugs>	 (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510274 (owner: 10Dduvall)
[00:15:49] <shinken-wm>	 RECOVERY - Content Translation Server on deployment-sca01 is OK: HTTP OK: HTTP/1.1 200 OK - 904 bytes in 0.027 second response time
[00:35:38] <wmf-insecte>	 Project beta-scap-eqiad build #249257: 04FAILURE in 1 min 19 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249257/
[00:45:35] <wmf-insecte>	 Project beta-scap-eqiad build #249258: 04STILL FAILING in 1 min 17 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249258/
[00:55:39] <wmf-insecte>	 Project beta-scap-eqiad build #249259: 04STILL FAILING in 1 min 21 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249259/
[01:00:42] <wmf-insecte>	 Yippee, build fixed!
[01:00:42] <wmf-insecte>	 Project beta-scap-eqiad build #249260: 09FIXED in 1 min 53 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249260/
[01:00:48] <thcipriani>	 !log /usr/local/sbin/keyholder arm
[01:00:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[01:49:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 3 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) I've opened a couple of subtasks and T223346. Considering the Citoid service on sca01 is probably already br...
[02:02:48] <shinken-wm>	 RECOVERY - Puppet errors on deployment-deploy02 is OK: OK: Less than 1.00% above the threshold [2.0]
[02:20:10] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10Krenair) May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal world, the teams developing new services would i...
[02:35:10] <Krenair>	 !log Logged into deployment-sca0[12] as root and given them the correct nameservers to try to unbreak things. T221654
[02:35:13] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[02:35:13] <stashbot>	 T221654: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654
[02:36:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 10serviceops, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) (Citoid migration commit is https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510290/)
[02:37:37] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Cherry-picked the above commit on deployment-puppetmaster, VE-Citoid integration appears to work again (thou...
[02:43:26] <wikibugs>	 10Beta-Cluster-Infrastructure, 10serviceops, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) Removed citoid role from them, and now: Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Could not find clas...
[04:46:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) `krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker run -p 8080:8080 -v /etc/mediawiki-services-cx...
[04:47:53] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<30.00%)
[04:53:57] <wikibugs>	 (03CR) 10Hashar: [C: 04-1] "The '-m checkstyle' should be a different change. The issue is that 1.2.x no more allows options after files and I believe that since 1.2." [integration/config] - 10https://gerrit.wikimedia.org/r/510140 (https://phabricator.wikimedia.org/T219114) (owner: 10Awight)
[04:56:17] <wikibugs>	 (03CR) 10Hashar: Use phan images with disabled progress bar (033 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/510143 (https://phabricator.wikimedia.org/T219114) (owner: 10Awight)
[05:08:50] <wikibugs>	 10Continuous-Integration-Config, 10Patch-For-Review: Whitelisted users without +2 rights can no longer recheck patches which have a +2 - https://phabricator.wikimedia.org/T223212 (10hashar) Well both were annoying regressions, albeit little ones so I guess it was better to just revert :-]  Eventually I will co...
[06:19:39] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2019-April-June): Running translatewiki export for MediaWiki extensions: Too many concurrent connections (4) - max. allowed: 4 - https://phabricator.wikimedia.org/T222546 (10Raymond) @Nikerabbit Tested it this morning agai...
[06:31:31] <wikibugs>	 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10revi) I make heavy use of IRCCloud (for more than 5 years) but I would prefer something other than IRC due to it being inferior with netsplit and etc.  Basicall...
[06:39:49] <wikibugs>	 10Phabricator, 10Developer-Advocacy (Apr-Jun 2019): Re-evaluate our use of Phabricator Conpherence chat - https://phabricator.wikimedia.org/T127640 (10greg) >>! In T127640#5183628, @revi wrote: > I make heavy use of IRCCloud (for more than 5 years) but I would prefer something other than IRC due to it being in...
[06:53:35] <wikibugs>	 10Release-Engineering-Team, 10Operations, 10Release Pipeline, 10Wikidata, and 5 others: Introduce wikidata termbox SSR to kubernetes - https://phabricator.wikimedia.org/T220402 (10WMDE-leszek) >>! In T220402#5183351, @Krenair wrote: > May 14 08:24:34 <_joe_> and thanks for helping with that. In an ideal wo...
[07:02:53] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:22:24] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2019-April-June), 10Patch-For-Review: Running translatewiki export for MediaWiki extensions: Too many concurrent connections (4) - max. allowed: 4 - https://phabricator.wikimedia.org/T222546 (10Raymond) >>! In T222546#51...
[07:45:36] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: How do we update jenkins to php 7 - https://phabricator.wikimedia.org/T223348 (10Reedy)
[08:02:34] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Jdforrester-WMF) >>! In T222918#5183118, @greg wrote: > Should this be a...
[08:03:54] <wikibugs>	 10Release-Engineering-Team (Backlog), 10MediaWiki-extensions-UserMerge, 10Stewards-and-global-tools: Undeploy UserMerge Extension from WMF production - https://phabricator.wikimedia.org/T216089 (10Jdforrester-WMF)
[08:04:03] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Jdforrester-WMF)
[08:08:30] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2019-April-June), 10Patch-For-Review: Running translatewiki export for MediaWiki extensions: Too many concurrent connections (4) - max. allowed: 4 - https://phabricator.wikimedia.org/T222546 (10hashar) I have missed this...
[08:08:42] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Nikerabbit) How will translations be handled? One-off dump of translation...
[08:12:51] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10hdothiduc) Hi! Thanks for all the feedback! @Quiddity, I used that tool, it's great!  @Aklapper, well,...
[08:14:19] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10ArielGlenn) p:05Triage→03Normal
[08:14:38] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Site-requests, 10WikimediaMessages: Put "shim" code for namespaces, logs, and log i18n into WikimediaMessages so we can undeploy extensions - https://phabricator.wikimedia.org/T222918 (10Jdforrester-WMF) >>! In T222918#5183824, @Nikerabbit wrote: > How will tr...
[08:16:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10ArielGlenn) Let's get sign-off from your WMDE manager (who is that?) and from @hashar or @greg in releng....
[08:17:30] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Grant James Forrester access to contint-admins and contint-docker, and to the ciadmin LDAP group - https://phabricator.wikimedia.org/T223137 (10ArielGlenn) p:05Triage→03Normal
[08:17:43] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2019-April-June), 10Patch-For-Review: Running translatewiki export for MediaWiki extensions: Too many concurrent connections (4) - max. allowed: 4 - https://phabricator.wikimedia.org/T222546 (10Nikerabbit) Thanks, I'll f...
[08:18:55] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10akosiaris) >>! In T220235#5183451, @Krenair wrote: > `krenair@deployment-docker-cxserver01:~$ sudo /usr/bin/docker ru...
[08:20:55] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: How do we update jenkins to php 7 - https://phabricator.wikimedia.org/T223348 (10hashar) I think we went with php5.6 because production used Jessie / php5.6 hence why there is a composer job: `wikimedia-fundraising-crm-composer-php56-docker `. It should b...
[08:20:59] <wikibugs>	 10Continuous-Integration-Infrastructure (Slipway), 10Wikimedia-Fundraising-CiviCRM: Migrate wikimedia-fundraising-civicrm to a Docker container - https://phabricator.wikimedia.org/T210287 (10hashar)
[08:21:02] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: How do we update jenkins to php 7 - https://phabricator.wikimedia.org/T223348 (10hashar)
[08:41:41] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: How do we update jenkins to php 7 - https://phabricator.wikimedia.org/T223348 (10Eileenmcnaughton) @Reedy @hashar -our live server was recently upgraded so it's using php 7.0 so it would be good to get our tested onto it so we can start using different ph...
[08:57:18] <wikibugs>	 10Continuous-Integration-Config, 10Fundraising-Backlog: Run wikimedia/fundraising/crm CI jobs on PHP7x, not PHP5x - https://phabricator.wikimedia.org/T223348 (10Jdforrester-WMF)
[09:00:29] <shinken-wm>	 PROBLEM - Host integration-puppetmaster01 is DOWN: CRITICAL - Host Unreachable (172.16.3.17)
[09:00:47] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 1.11% of data above the critical threshold [3.0]
[09:06:47] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10awight) My manager is @Tobi_WMDE_SW, and the CI work would usually full under the maintenance / 20% catego...
[09:09:49] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Backlog), 10Operations, 10SRE-Access-Requests: Request: add awight to contint-docker - https://phabricator.wikimedia.org/T223262 (10ArielGlenn) The ssh key issue has been cleared up.
[09:18:02] <shinken-wm>	 RECOVERY - Host integration-puppetmaster01 is UP: PING OK - Packet loss = 0%, RTA = 1.30 ms
[09:19:36] <shinken-wm>	 PROBLEM - Host integration-castor03 is DOWN: CRITICAL - Host Unreachable (172.16.5.161)
[09:20:51] <shinken-wm>	 PROBLEM - Content Translation Server on deployment-sca01 is CRITICAL: connect to address 172.16.5.13 and port 8080: Connection refused
[09:21:38] <shinken-wm>	 PROBLEM - Content Translation Server on deployment-sca02 is CRITICAL: connect to address 172.16.5.112 and port 8080: Connection refused
[09:28:42] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Grant James Forrester access to contint-admins and contint-docker, and to the ciadmin LDAP group - https://phabricator.wikimedia.org/T223137 (10ArielGlenn) Added to ciadmin ldap group.
[09:28:47] <shinken-wm>	 PROBLEM - Host saucelabs-01 is DOWN: CRITICAL - Host Unreachable (172.16.3.19)
[09:31:58] <wikibugs>	 (03PS1) 10Hashar: Disable castor saving [integration/config] - 10https://gerrit.wikimedia.org/r/510450 (https://phabricator.wikimedia.org/T223148)
[09:34:40] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Disable castor saving [integration/config] - 10https://gerrit.wikimedia.org/r/510450 (https://phabricator.wikimedia.org/T223148) (owner: 10Hashar)
[09:35:48] <hashar>	 !log Regenerating all CI jobs to disable castor saving entirely
[09:35:49] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[09:38:31] <wikibugs>	 (03Merged) 10jenkins-bot: Disable castor saving [integration/config] - 10https://gerrit.wikimedia.org/r/510450 (https://phabricator.wikimedia.org/T223148) (owner: 10Hashar)
[09:38:38] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Grant James Forrester access to contint-admins and contint-docker, and to the ciadmin LDAP group - https://phabricator.wikimedia.org/T223137 (10ArielGlenn) You should check in about half an hour once...
[09:39:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10aborrero)
[09:40:39] <wikibugs>	 (03PS1) 10Hashar: Deactivate castor-save job for now [integration/config] - 10https://gerrit.wikimedia.org/r/510452
[09:42:07] <wikibugs>	 (03PS2) 10Hashar: Deactivate castor-save/castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/510452
[09:42:20] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Deactivate castor-save/castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/510452 (owner: 10Hashar)
[09:43:19] <shinken-wm>	 RECOVERY - Host saucelabs-01 is UP: PING OK - Packet loss = 0%, RTA = 0.96 ms
[09:45:06] <wikibugs>	 (03Merged) 10jenkins-bot: Deactivate castor-save/castor-save-workspace-cache [integration/config] - 10https://gerrit.wikimedia.org/r/510452 (owner: 10Hashar)
[09:46:55] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10aborrero) saucelabs-01.integration.eqiad.wmflabs seems to have the same issue.
[09:53:04] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10aborrero) p:05Triage→03High
[10:16:01] <wikibugs>	 (03PS3) 10Awight: Bump mediawiki-phan* containers [integration/config] - 10https://gerrit.wikimedia.org/r/510140 (https://phabricator.wikimedia.org/T219114)
[10:16:03] <wikibugs>	 (03PS2) 10Awight: Use phan images with disabled progress bar [integration/config] - 10https://gerrit.wikimedia.org/r/510143 (https://phabricator.wikimedia.org/T219114)
[10:16:05] <wikibugs>	 (03PS1) 10Awight: Fix phan parameter order [integration/config] - 10https://gerrit.wikimedia.org/r/510458 (https://phabricator.wikimedia.org/T219114)
[10:16:07] <wikibugs>	 (03PS1) 10Awight: Reduce memory usage by omitting progress bar [integration/config] - 10https://gerrit.wikimedia.org/r/510459 (https://phabricator.wikimedia.org/T219114)
[10:17:37] <wikibugs>	 (03CR) 10Awight: "Thanks!  New patch chain should address the review comments." [integration/config] - 10https://gerrit.wikimedia.org/r/510140 (https://phabricator.wikimedia.org/T219114) (owner: 10Awight)
[10:36:02] <shinken-wm>	 RECOVERY - Host integration-castor03 is UP: PING OK - Packet loss = 0%, RTA = 1.18 ms
[11:05:16] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ArielGlenn) The blocking ticket is closed; what else is needed for this to move forward?
[11:05:23] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10ArielGlenn)
[11:05:28] <_joe_>	 hi, is there anyone around?
[11:05:39] <_joe_>	 no one is available for SWAT
[11:05:45] <awight>	 What's the recommended way to experiment with nodejs-selenium browser tests?  I have a MediaWiki extension patch I'm working on which includes tests/selenium, but this doesn't seem to be enough to trigger quibble's 'npm run selenium' job.
[11:08:53] <apergos>	 anyone here who can do eu swat? https://wikitech.wikimedia.org/wiki/Deployments#deploycal-item-20190515T1100  there's several items on the list today
[11:10:51] <apergos>	 no hashar, no dereckson 
[11:12:06] <_joe_>	 no zjeliko
[11:12:15] <_joe_>	 so I guess not.
[11:14:11] <awight>	 Tangentially, the calendar lists SF deployers for this slot, which seems wrong.
[11:14:42] <apergos>	 Lucas_WMDE: are you available for eu swat by any chance?
[11:15:50] <apergos>	 or Amir1?
[11:15:59] <apergos>	 after that I'm out of possibilities
[11:16:13] <_joe_>	 awight: indeed
[11:16:30] <_joe_>	 this is not a time in which most eu people are at the keyboard
[11:16:37] <_joe_>	 but rather in lunch break :P
[11:20:14] <awight>	 Well I'd be willing to do the SWAT and sort of remember how it's done, but am no longer included in wmf-deployments.
[11:21:58] * awight squints.  Well $#!7 I am in that group.  Just don't have shell access to the deployment server.
[11:23:22] <_joe_>	 awight: it's ok, thanks
[11:24:47] <Krinkle>	 awight: existence of files in that directory is what Quibble uses to decide to run the step
[11:25:46] <Krinkle>	 awight: if not working as expected, file a task under Quibble+ RelEng and Ping them for help. Might be a bug or something else
[11:26:11] <awight>	 Krinkle: Thanks--probably I'm just not getting to that step due to other, unrelated failures in my repo.
[11:29:17] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10translatewiki.net, 10Language-Team (Language-2019-April-June): Running translatewiki export for MediaWiki extensions: Too many concurrent connections (4) - max. allowed: 4 - https://phabricator.wikimedia.org/T222546 (10Nikerabbit)
[11:32:06] <wikibugs>	 (03PS1) 10Hashar: Revert "Deactivate castor-save/castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/510475
[11:32:09] <wikibugs>	 (03PS1) 10Hashar: Revert "Disable castor saving" [integration/config] - 10https://gerrit.wikimedia.org/r/510476
[11:32:16] <hashar>	 !log bringing back castor, integration-castor03 is back
[11:32:16] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:32:37] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Revert "Deactivate castor-save/castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/510475 (owner: 10Hashar)
[11:32:44] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Revert "Disable castor saving" [integration/config] - 10https://gerrit.wikimedia.org/r/510476 (owner: 10Hashar)
[11:34:19] <Lucas_WMDE>	 apergos: I’m here now if you still need someone
[11:34:29] <apergos>	 _joe_: ^^
[11:34:40] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) I created a form. Haven't played much with Forms before, so bare with me. https://phabricato...
[11:34:48] <_joe_>	 Lucas_WMDE: we do, indeed
[11:35:05] <Lucas_WMDE>	 oh, right, I should’ve read the -operations backlog
[11:35:07] * Lucas_WMDE reads
[11:36:05] <Amir1>	 apergos: sorry, me an Lucas were at a meeting. Can we proceed?
[11:37:47] <apergos>	 for later: getting the list of eu swatters updated so it's not just sf folks 
[11:38:01] <apergos>	 I dunno who would do that but if someone can make that happen 
[11:39:18] <Lucas_WMDE>	 apergos: that would be T223087, I think
[11:39:23] <stashbot>	 T223087: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087
[11:41:07] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-SWAT-deployments: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Lucas_Werkmeister_WMDE) I have deploy access and sometimes already do the SWAT (with supervision by @Ladsgroup or @Addshore), and I’d be okay with being added to the list too. (I a...
[11:41:52] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-SWAT-deployments: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10ArielGlenn) It is wrong; we hit this issue today. The other thing that goes along with having a good eu list is to make sure someone on the list is around during the swat time; we...
[11:41:57] <apergos>	 commented there and thanks
[11:42:11] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-SWAT-deployments: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Ladsgroup) I also asked to be added there in the last year's hackathon :)
[11:48:33] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-SWAT-deployments: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Ladsgroup) >>! In T223087#5183011, @Rxy wrote: > I remember that time are almost SWAT-ed by zeljkof. >  > I can available that time if I have a deploy access (I have no deploy acce...
[11:54:31] <hashar>	 I think Jenkins is dead :(
[11:55:20] <Zppix>	 hashar: i mean its overworked, underpaid, and not given all the attention what do we expect :P
[11:55:40] <Lucas_WMDE>	 why is it dead? the builds in gate-and-submit-swat seem to be progressing
[11:55:42] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Deactivate castor-save/castor-save-workspace-cache" [integration/config] - 10https://gerrit.wikimedia.org/r/510475 (owner: 10Hashar)
[11:55:46] <wikibugs>	 (03Merged) 10jenkins-bot: Revert "Disable castor saving" [integration/config] - 10https://gerrit.wikimedia.org/r/510476 (owner: 10Hashar)
[11:55:52] <hashar>	 !log Bring back https://integration.wikimedia.org/ci/computer/integration-castor03/ to restore the central cache behavior. In turn unblocking a wide range of builds
[11:55:53] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[11:56:37] <hashar>	 Lucas_WMDE: the castor03 instance was offline, it is the instance responsible for saving various cached materials :)
[11:56:40] <hashar>	 and jobs block on it
[11:56:42] <hashar>	 anyway it is back
[11:56:46] <Lucas_WMDE>	 oh, ok
[11:56:49] <hashar>	 solved!
[11:57:00] <Lucas_WMDE>	 congrats on the necromancy, whoever’s responsible ;)
[11:57:37] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10serviceops, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10ArielGlenn)
[11:58:27] <paladox>	 Gerrit 3.0 has been released! https://groups.google.com/forum/m/#!topic/repo-discuss/27X1IFSgmo4
[12:12:29] <wikibugs>	 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Aklapper) a:03Aklapper Alright, we have a script in place to convert parent projects into subprojects (see T221112). I've loo...
[12:17:06] <wikibugs>	 (03PS1) 10Hashar: Expand gate-and-submit-swat definition [integration/config] - 10https://gerrit.wikimedia.org/r/510490 (https://phabricator.wikimedia.org/T221410)
[12:27:01] <onimisionipe>	 hello there!
[12:27:22] <onimisionipe>	 any idea why this: https://gerrit.wikimedia.org/r/c/maps/kartotherian/package/+/510456 is not submitted to jenkins for build?
[12:28:58] <onimisionipe>	 I just added the blubber.yaml file to the repo
[12:29:09] <onimisionipe>	 and I'm thinking jenkins should do its thing.
[12:29:14] <onimisionipe>	 except I'm missing something
[12:29:44] <onimisionipe>	 we are trying to migrate kartotherian to kubernetes so I'm following the deployment pipeline
[12:29:46] <Reedy>	 jenkins doesn't just look at any repo
[12:30:09] <Reedy>	 You need to change the config on the jenkins side to run blubber jobs
[12:31:26] <shinken-wm>	 PROBLEM - Puppet errors on deployment-puppetdb02 is CRITICAL: (Service Check Timed Out)
[12:32:21] <shinken-wm>	 PROBLEM - Puppet errors on deployment-ms-be05 is CRITICAL: (Service Check Timed Out)
[12:32:25] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-docker-1056 is CRITICAL: (Service Check Timed Out)
[12:32:28] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-chromium01 is CRITICAL: (Service Check Timed Out)
[12:32:29] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-jobrunner03 is CRITICAL: (Service Check Timed Out)
[12:32:30] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-aqs02 is CRITICAL: (Service Check Timed Out)
[12:32:31] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-restbase01 is CRITICAL: (Service Check Timed Out)
[12:32:32] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-mcs01 is CRITICAL: (Service Check Timed Out)
[12:32:35] <shinken-wm>	 PROBLEM - Puppet errors on deployment-maps05 is CRITICAL: (Service Check Timed Out)
[12:32:36] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-docker-mathoid01 is CRITICAL: (Service Check Timed Out)
[12:35:33] <paladox>	 I guess that’s eqiad network issues?
[12:35:51] <apergos>	 general issues, being looked into
[12:36:03] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out)
[12:36:04] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-etcd-01 is CRITICAL: (Service Check Timed Out)
[12:36:08] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-webperf12 is CRITICAL: (Service Check Timed Out)
[12:36:13] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: (Service Check Timed Out)
[12:36:14] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-aqs01 is CRITICAL: (Service Check Timed Out)
[12:36:14] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-eventgate-1 is CRITICAL: (Service Check Timed Out)
[12:36:15] <shinken-wm>	 PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: (Service Check Timed Out)
[12:36:16] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out)
[12:36:16] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-imagescaler02 is CRITICAL: (Service Check Timed Out)
[12:36:18] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out)
[12:36:18] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-main-2 is CRITICAL: (Service Check Timed Out)
[12:36:19] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: (Service Check Timed Out)
[12:36:19] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-ms-be05 is CRITICAL: (Service Check Timed Out)
[12:36:22] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-mwmaint01 is CRITICAL: (Service Check Timed Out)
[12:36:22] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-docker-1043 is CRITICAL: (Service Check Timed Out)
[12:36:23] <shinken-wm>	 PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: (Service Check Timed Out)
[12:36:25] <shinken-wm>	 PROBLEM - Puppet errors on deployment-prometheus02 is CRITICAL: (Service Check Timed Out)
[12:36:25] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-cache-text05 is CRITICAL: (Service Check Timed Out)
[12:36:26] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-hadoop-test-1 is CRITICAL: (Service Check Timed Out)
[12:36:27] <shinken-wm>	 PROBLEM - Puppet errors on deployment-hadoop-test-1 is CRITICAL: (Service Check Timed Out)
[12:36:28] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-sca01 is CRITICAL: (Service Check Timed Out)
[12:36:31] <shinken-wm>	 PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: (Service Check Timed Out)
[12:37:14] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-hadoop-test-1 is CRITICAL: (Service Check Timed Out)
[12:37:23] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out)
[12:37:29] <onimisionipe>	 Reedy: Thanks!
[12:37:29] <onimisionipe>	 I'm sure I don't have priv to do that? any body I can talk to?
[12:37:40] <Reedy>	 You can certainly make the patch to gerrit though :)
[12:37:42] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-cache-upload05 is CRITICAL: (Service Check Timed Out)
[12:37:42] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-hadoop-test-2 is CRITICAL: (Service Check Timed Out)
[12:37:42] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-logstash2 is CRITICAL: (Service Check Timed Out)
[12:37:44] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-webperf12 is CRITICAL: (Service Check Timed Out)
[12:37:48] <Reedy>	 onimisionipe: integration/config repo
[12:37:50] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-kafka-main-1 is CRITICAL: (Service Check Timed Out)
[12:37:52] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out)
[12:37:56] <shinken-wm>	 PROBLEM - Free space - all mounts on webperformance is CRITICAL: (Service Check Timed Out)
[12:38:13] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:38:26] <onimisionipe>	 Oh.. Ok. I should make a patch
[12:38:27] <onimisionipe>	 Thanks
[12:38:48] <shinken-wm>	 PROBLEM - Puppet errors on deployment-acme-chief03 is CRITICAL: (Service Check Timed Out)
[12:38:53] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out)
[12:39:03] <shinken-wm>	 PROBLEM - Puppet errors on deployment-ms-fe03 is CRITICAL: (Service Check Timed Out)
[12:39:03] <shinken-wm>	 PROBLEM - Puppet staleness on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out)
[12:39:03] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-sessionstore01 is CRITICAL: (Service Check Timed Out)
[12:39:03] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-aqs01 is CRITICAL: (Service Check Timed Out)
[12:39:04] <shinken-wm>	 PROBLEM - Puppet staleness on integration-puppetmaster01 is CRITICAL: (Service Check Timed Out)
[12:39:04] <shinken-wm>	 PROBLEM - Puppet staleness on integration-slave-docker-1055 is CRITICAL: (Service Check Timed Out)
[12:39:05] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-changeprop is CRITICAL: (Service Check Timed Out)
[12:39:09] <shinken-wm>	 PROBLEM - Puppet staleness on integration-slave-docker-1048 is CRITICAL: (Service Check Timed Out)
[12:39:09] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-hadoop-test-3 is CRITICAL: (Service Check Timed Out)
[12:40:24] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-docker-citoid01 is CRITICAL: (Service Check Timed Out)
[12:40:57] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: (Service Check Timed Out)
[12:40:59] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-mediawiki-09 is CRITICAL: (Service Check Timed Out)
[12:41:20] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ircd is CRITICAL: (Service Check Timed Out)
[12:41:23] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-1 is CRITICAL: (Service Check Timed Out)
[12:41:29] <shinken-wm>	 PROBLEM - Free space - all mounts on saucelabs-03 is CRITICAL: (Service Check Timed Out)
[12:41:30] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-hadoop-test-2 is CRITICAL: (Service Check Timed Out)
[12:41:32] <shinken-wm>	 PROBLEM - Puppet errors on deployment-urldownloader02 is CRITICAL: (Service Check Timed Out)
[12:41:37] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-cumin02 is CRITICAL: (Service Check Timed Out)
[12:41:41] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-puppetmaster03 is CRITICAL: (Service Check Timed Out)
[12:41:42] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-memc06 is CRITICAL: (Service Check Timed Out)
[12:41:45] <shinken-wm>	 PROBLEM - Puppet errors on deployment-db05 is CRITICAL: (Service Check Timed Out)
[12:41:47] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-db05 is CRITICAL: (Service Check Timed Out)
[12:41:47] <shinken-wm>	 PROBLEM - Puppet staleness on integration-slave-docker-1043 is CRITICAL: (Service Check Timed Out)
[12:41:47] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-restbase02 is CRITICAL: (Service Check Timed Out)
[12:41:48] <shinken-wm>	 PROBLEM - Puppet staleness on saucelabs-03 is CRITICAL: (Service Check Timed Out)
[12:41:48] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-ircd is CRITICAL: (Service Check Timed Out)
[12:41:49] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-chromium02 is CRITICAL: (Service Check Timed Out)
[12:41:53] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-prometheus02 is CRITICAL: (Service Check Timed Out)
[12:41:53] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-snapshot01 is CRITICAL: (Service Check Timed Out)
[12:41:53] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-kafka-jumbo-2 is CRITICAL: (Service Check Timed Out)
[12:41:53] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: (Service Check Timed Out)
[12:41:54] <shinken-wm>	 PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: (Service Check Timed Out)
[12:41:55] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-maps04 is CRITICAL: (Service Check Timed Out)
[12:41:56] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-acme-chief04 is CRITICAL: (Service Check Timed Out)
[12:41:56] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: (Service Check Timed Out)
[12:41:58] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-sca04 is CRITICAL: (Service Check Timed Out)
[12:41:58] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki-09 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[12:42:00] <shinken-wm>	 PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: (Service Check Timed Out)
[12:42:01] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-memc04 is CRITICAL: (Service Check Timed Out)
[12:42:05] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-hadoop-test-1 is OK: OK: All targets OK
[12:42:12] <shinken-wm>	 RECOVERY - Puppet errors on deployment-ms-be05 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:42:12] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-docker-1056 is OK: OK: All targets OK
[12:42:12] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-acme-chief03 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:42:18] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-restbase01 is OK: OK: deployment-prep.deployment-restbase01.diskspace._var_log.byte_percentfree (No valid datapoints found)
[12:42:20] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-jobrunner03 is OK: OK: All targets OK
[12:42:20] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-chromium01 is OK: OK: All targets OK
[12:42:21] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-mcs01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:42:22] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:42:24] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cache-upload05 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:42:24] <shinken-wm>	 RECOVERY - Puppet errors on deployment-maps05 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:42:25] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-urldownloader02 is OK: OK: All targets OK
[12:42:26] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-docker-mathoid01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:42:28] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-restbase02 is OK: OK: deployment-prep.deployment-restbase02.diskspace._var_log.byte_percentfree (No valid datapoints found)
[12:42:30] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-webperf11 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:42:31] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-castor03 is OK: OK: All targets OK
[12:43:29] <hashar>	 onimisionipe: any clue why we have maps/kartotherian/package   and maps/kartotherian ??
[12:44:46] <onimisionipe>	 hashar: mateusbs17 would be the best person to answer to answer that. He specifically told me to use maps/kartotherian/package
[12:44:57] <onimisionipe>	 mateusbs17: ^
[12:45:09] <shinken-wm>	 RECOVERY - Puppet errors on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:13] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-docker-citoid01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:45:14] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:14] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1043 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:18] <shinken-wm>	 RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:19] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:20] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1054 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:23] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-docker-1034 is OK: OK: All targets OK
[12:45:23] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1021 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:28] <hashar>	 onimisionipe: well I would let someone else dig in all the madness of moving kartotherian to the pipeline ;)
[12:45:32] <shinken-wm>	 RECOVERY - Puppet errors on deployment-chromium01 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:32] <hashar>	 but yeah that needs a CI config change
[12:45:35] <shinken-wm>	 RECOVERY - Puppet staleness on integration-castor03 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:45:35] <shinken-wm>	 RECOVERY - English Wikipedia Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 48058 bytes in 1.208 second response time
[12:45:35] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-eventlog05 is OK: OK: deployment-prep.deployment-eventlog05.diskspace._var_lib_mysql.byte_percentfree (No valid datapoints found)
[12:45:35] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-kafka-jumbo-1 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:45:35] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-imagescaler03 is OK: OK: All targets OK
[12:45:36] <shinken-wm>	 RECOVERY - Puppet errors on deployment-dumps-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:45:48] <wikibugs_>	 (03PS1) 10Hashar: Remove php5/7.0/7.1 from gate-and-submit-swat [integration/config] - 10https://gerrit.wikimedia.org/r/510496 (https://phabricator.wikimedia.org/T221410)
[12:45:50] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-mediawiki-09 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:45:50] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:45:54] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mediawiki-09 is OK: OK: All targets OK
[12:46:00] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-webperf12 is OK: OK: All targets OK
[12:46:05] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:06] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-aqs01 is OK: OK: All targets OK
[12:46:06] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-eventgate-1 is OK: OK: All targets OK
[12:46:07] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-main-2 is OK: OK: All targets OK
[12:46:08] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:09] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:09] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-ms-be05 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:11] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-mwmaint01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:12] <shinken-wm>	 RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:15] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-ircd is OK: OK: All targets OK
[12:46:17] <shinken-wm>	 RECOVERY - Puppet errors on deployment-prometheus02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:17] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-cache-text05 is OK: OK: All targets OK
[12:46:18] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-hadoop-test-1 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:18] <hashar>	 onimisionipe: I am crafting the CI change and will ask for review
[12:46:19] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:19] <shinken-wm>	 RECOVERY - Puppet errors on deployment-hadoop-test-1 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:21] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-1 is OK: OK: deployment-prep.deployment-kafka-jumbo-1.diskspace._mnt_kafka.byte_percentfree (No valid datapoints found)
[12:46:23] <shinken-wm>	 RECOVERY - Puppet errors on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:26] <shinken-wm>	 RECOVERY - Puppet errors on deployment-puppetdb02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:26] <shinken-wm>	 RECOVERY - Free space - all mounts on saucelabs-03 is OK: OK: integration.saucelabs-03.diskspace._srv.byte_percentfree (No valid datapoints found)
[12:46:27] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-hadoop-test-2 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:28] <shinken-wm>	 RECOVERY - Puppet errors on deployment-urldownloader02 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:31] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-cumin02 is OK: OK: All targets OK
[12:46:35] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:36] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-puppetmaster03 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:39] <shinken-wm>	 RECOVERY - Puppet errors on deployment-db05 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:41] <shinken-wm>	 RECOVERY - Puppet staleness on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:41] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-db05 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:43] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:44] <shinken-wm>	 RECOVERY - Puppet staleness on integration-slave-docker-1043 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:45] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-ircd is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:45] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-chromium02 is OK: OK: All targets OK
[12:46:45] <onimisionipe>	 hashar: Thanks!!
[12:46:47] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-kafka-jumbo-2 is OK: OK: deployment-prep.deployment-kafka-jumbo-2.diskspace._mnt_kafka.byte_percentfree (No valid datapoints found)
[12:46:48] <shinken-wm>	 RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:48] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:49] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-maps04 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:49] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-jessie-1004 is OK: OK: All targets OK
[12:46:50] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:50] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-prometheus02 is OK: OK: All targets OK
[12:46:51] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki-09 is OK: HTTP OK: HTTP/1.1 200 OK - 47431 bytes in 1.580 second response time
[12:46:54] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-acme-chief04 is OK: OK: All targets OK
[12:46:54] <shinken-wm>	 RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [2.0]
[12:46:55] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [3600.0]
[12:46:57] <hashar>	 onimisionipe: but most probably the pipeline file should be in maps/kartotherian instead
[12:51:14] <hashar>	 onimisionipe: I just replied at https://phabricator.wikimedia.org/T223275#5184528  but I delegate to other people ;)
[12:52:27] <wikibugs>	 (03CR) 10Hashar: "That is a noop in Zuul, I have just copy pasted the definitions.  This change makes it easier to review the follow up patch https://gerrit" [integration/config] - 10https://gerrit.wikimedia.org/r/510490 (https://phabricator.wikimedia.org/T221410) (owner: 10Hashar)
[12:53:29] <onimisionipe>	 hashar: Ok
[12:54:22] <wikibugs>	 (03CR) 10Hashar: "Since gate-and-submit-swat is only for wmf/ branches, we can just remove the jobs. That saves us from having to deal with the complicated " [integration/config] - 10https://gerrit.wikimedia.org/r/510496 (https://phabricator.wikimedia.org/T221410) (owner: 10Hashar)
[12:59:19] <hashar>	 I am off, be back later this evening
[13:12:22] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] Expand gate-and-submit-swat definition [integration/config] - 10https://gerrit.wikimedia.org/r/510490 (https://phabricator.wikimedia.org/T221410) (owner: 10Hashar)
[13:15:10] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10hdothiduc) Oh cool! I think, in an effort to direct attention for the requester:  - we could hide the...
[13:30:30] <wikibugs>	 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Dvorapa) Copyright and delinker not, they were part of #pywikibot-compat never ported to core  Textlib and pagegenerators also...
[13:35:26] <wikibugs>	 (03PS1) 10Jforrester: Run 'composer-test' template on php72 too [integration/config] - 10https://gerrit.wikimedia.org/r/510506
[13:35:28] <wikibugs>	 (03PS1) 10Jforrester: Only run 'composer-test' SWAT pipeline on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/510507
[13:35:30] <wikibugs>	 (03PS1) 10Jforrester: Collapse the mwgate-composer template to its only user [integration/config] - 10https://gerrit.wikimedia.org/r/510508
[13:35:32] <wikibugs>	 (03PS1) 10Jforrester: Drop the gate-and-submit-swat pipeline for BlueSpice extensions, never used [integration/config] - 10https://gerrit.wikimedia.org/r/510509
[13:37:20] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Collapse the mwgate-composer template to its only user [integration/config] - 10https://gerrit.wikimedia.org/r/510508 (owner: 10Jforrester)
[13:37:22] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Run 'composer-test' template on php72 too [integration/config] - 10https://gerrit.wikimedia.org/r/510506 (owner: 10Jforrester)
[13:37:51] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Drop the gate-and-submit-swat pipeline for BlueSpice extensions, never used [integration/config] - 10https://gerrit.wikimedia.org/r/510509 (owner: 10Jforrester)
[13:38:26] <wikibugs>	 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Xqt) >>! In T195855#5184614, @Dvorapa wrote: > I wonder about cosmetic-changes, these must be discussed first (but can be joine...
[13:39:59] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Only run 'composer-test' SWAT pipeline on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/510507 (owner: 10Jforrester)
[13:40:48] <mateusbs17>	 hashar: onimisionipe: kartotherian is considered a third-party and is supposed to have its own community. The packaging adds it and possible specific code to attend wmf needs
[13:41:20] <onimisionipe>	 mateusbs17: Thanks! can you reply in phab as well?
[13:41:38] <mateusbs17>	 Sure
[13:46:46] <wikibugs>	 10Phabricator, 10Project-Admins, 10Pywikibot: Move #Pywikibot-Other-scripts to #Pywikibot-Scripts and transform it into parent project - https://phabricator.wikimedia.org/T195855 (10Dvorapa) @Aklapper Anyway, the others listed seems ok! :)
[13:49:31] <wikibugs>	 (03PS2) 10Jforrester: Run 'composer-test' template on php72 too [integration/config] - 10https://gerrit.wikimedia.org/r/510506
[13:49:33] <wikibugs>	 (03PS2) 10Jforrester: Only run 'composer-test' SWAT pipeline on php72 [integration/config] - 10https://gerrit.wikimedia.org/r/510507
[13:49:35] <wikibugs>	 (03PS2) 10Jforrester: Collapse the mwgate-composer template to its only user [integration/config] - 10https://gerrit.wikimedia.org/r/510508
[13:49:37] <wikibugs>	 (03PS2) 10Jforrester: Drop the gate-and-submit-swat pipeline for BlueSpice extensions, never used [integration/config] - 10https://gerrit.wikimedia.org/r/510509
[13:49:39] <wikibugs>	 (03PS1) 10Jforrester: dockerfiles: Provide a composer-test-php72 image [integration/config] - 10https://gerrit.wikimedia.org/r/510511
[13:49:41] <wikibugs>	 (03PS1) 10Jforrester: Provide the php72 flavour of composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/510512
[14:33:18] <_joe_>	 anyone can help me understand what tests are failing here: https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/WikimediaEvents/+/510431/
[14:34:45] <Nikerabbit>	 Looks like a issue coming from a recent change in GrowthExperiments extension
[14:35:04] <_joe_>	 that has nothing to do with WikimediaEvents
[14:35:05] <Nikerabbit>	 but I'm not sure if it has something to do it being in a deployment branch
[14:35:26] <Reedy>	 _joe_: But gated extensions, tested together etc
[14:35:35] <Reedy>	 Nikerabbit: Was it fixed in master do you know?
[14:35:51] <_joe_>	 Reedy: so why is it failing now and not before?
[14:35:56] <_joe_>	 Reedy: must be fixed in .5
[14:35:59] <Reedy>	 crap flaky test?
[14:36:04] <_joe_>	 since the tests are passing there
[14:36:20] <Reedy>	 It's possibe
[14:36:30] <Nikerabbit>	 ConfigException: HashConfig::get: undefined option: 'AllowConfirmedEmail'
[14:36:31] <_joe_>	 no I mean why it's failing for my patch, and not for say the one Bartosz merged before?
[14:36:38] <_joe_>	 because his one was in core?
[14:36:52] <Reedy>	 [15:35:58] <Reedy> crap flaky test?
[14:37:01] <Nikerabbit>	 that doesn't look something that would be flaky...
[14:37:24] <Nikerabbit>	 https://gerrit.wikimedia.org/r/c/mediawiki/extensions/Flow/+/434483 looks relevant
[14:37:25] <Reedy>	 https://phabricator.wikimedia.org/T222980
[14:38:13] <Nikerabbit>	 that looks even more relevant ^^
[14:38:16] <Lucas_WMDE>	 so, either backport that fix as well, or ignore the test failure and force-merge at deployment time?
[14:38:23] <Reedy>	 https://gerrit.wikimedia.org/r/#/c/mediawiki/extensions/GrowthExperiments/+/509517/ might be worth backporting to .4
[14:39:00] <_joe_>	 that looks like a bug indeed
[14:39:11] * Reedy backports and merges
[14:39:33] <Reedy>	 looks like James fixed a CI related issue
[14:41:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10Andrew) This is likely a result of puppet going out of sync.  The old recursors were 208.80.155.118 and 208.80.154.20, replaced with...
[14:41:33] <wikibugs>	 (03PS1) 10Mholloway: Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538
[14:43:02] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538 (owner: 10Mholloway)
[14:43:51] <wikibugs>	 (03PS2) 10Mholloway: Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538
[14:44:52] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10Andrew) a:05Andrew→03aborrero ...and it looks like puppet caught up?   ` root@integration-puppetmaster01:~# cat /etc/resolv.conf...
[14:51:23] <wikibugs>	 (03CR) 10Gergő Tisza: [C: 03+1] Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538 (owner: 10Mholloway)
[15:21:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10aborrero) I added those by hand to allow normal operation of the VMs. The original nameserver was being server apparently by the DHC...
[15:35:52] <awight>	 Reading the mw-core wdio.conf.js, I was expecting to find screenshots for failed tests e.g. https://integration.wikimedia.org/ci/job/quibble-vendor-mysql-hhvm-docker/48961/consoleFull but that doesn't seem to be working in this case.
[15:36:08] <wmf-insecte>	 Project beta-scap-eqiad build #249350: 04FAILURE in 1 min 49 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249350/
[15:36:46] <thcipriani>	 ^ me fiddling
[15:36:48] <thcipriani>	 will fix
[15:47:53] <awight>	 Maybe my problem is that the test is erroring out during "before" setup, rather than gracefully failing an assertion.
[15:50:28] <wikibugs>	 (03PS1) 10Thcipriani: clear_message_blobs: no logger in context [tools/scap] - 10https://gerrit.wikimedia.org/r/510560
[15:53:42] <wikibugs>	 (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/510560 (owner: 10Thcipriani)
[15:56:15] <wikibugs>	 (03CR) 10Thcipriani: [C: 03+2] clear_message_blobs: no logger in context [tools/scap] - 10https://gerrit.wikimedia.org/r/510560 (owner: 10Thcipriani)
[15:58:54] <wikibugs>	 (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [tools/scap] - 10https://gerrit.wikimedia.org/r/510560 (owner: 10Thcipriani)
[15:58:56] <wikibugs>	 (03Merged) 10jenkins-bot: clear_message_blobs: no logger in context [tools/scap] - 10https://gerrit.wikimedia.org/r/510560 (owner: 10Thcipriani)
[15:59:41] <wikibugs>	 (03CR) 10jenkins-bot: clear_message_blobs: no logger in context [tools/scap] - 10https://gerrit.wikimedia.org/r/510560 (owner: 10Thcipriani)
[16:10:53] <wmf-insecte>	 Yippee, build fixed!
[16:10:54] <wmf-insecte>	 Project beta-scap-eqiad build #249351: 09FIXED in 2 min 23 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/249351/
[16:15:44] <awight>	 Are docker containers firewalled in any way?  I'm suspecting that my test can't connect to production commons.  I probably *shouldn't*, but first I want to debug the current issue.
[16:17:51] <Krenair>	 awight, for the purposes of outbound connections I think they can do anything?
[16:18:11] <Krenair>	 running within labs your inbound connections are likely to be restricted in like 3 different ways
[16:18:24] <awight>	 Scary :-)  thanks, yeah that seems to be the case at least from the CI host machine.
[16:19:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) I fiddled around with that and now the container exits after 15s without printing anything :|
[16:20:34] <Krenair>	 awight, running within prod docker containers probably can't reach the outside world without going through urldownloader
[16:20:53] <Krenair>	 dunno about CI
[16:21:02] <Krenair>	 that's probably some of the integration.eqiad.wmflabs VMs
[16:21:40] <Krenair>	 so yeah they'll be protected by openstack security groups, maybe ferm, and docker potentially not forwarding all the ports in use to the outside world
[16:22:12] <Krenair>	 plus if the host instance has no floating IP then nothing inbound from the internet directly without the proxy
[16:39:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10Andrew) Ah so I see.  There are some VMs which were built from a bad base image and for some reason re-run the firstboot script on e...
[16:53:39] <wikibugs>	 10Release-Engineering-Team, 10MediaWiki-SWAT-deployments: SWAT Shepherds seem wrong - https://phabricator.wikimedia.org/T223087 (10Addshore) My name will be back on the list once I am fully back from vacation
[16:54:01] <wikibugs>	 (03Abandoned) 10Brennen Bearnes: Add Dockerfile templates for local-charts [integration/config] - 10https://gerrit.wikimedia.org/r/510212 (https://phabricator.wikimedia.org/T218360) (owner: 10Brennen Bearnes)
[16:58:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10Andrew) So it looks like any failure at all in the firstboot script means that firstboot will be run again on subsequent boots.  I'm...
[17:26:10] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Patch-For-Review, 10cloud-services-team (Kanban): CloudVPS DNS: weird behavior in integration-puppetmaster01 - https://phabricator.wikimedia.org/T223370 (10Andrew) a:05aborrero→03Andrew
[17:34:02] <wikibugs>	 (03PS4) 10Jforrester: Disable post-merge doc/coverage publish for l10n commits [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (owner: 10Hashar)
[17:34:32] <wikibugs>	 (03CR) 10Jforrester: "T219017 was marked fixed in March; can this now proceed?" [integration/config] - 10https://gerrit.wikimedia.org/r/498453 (owner: 10Hashar)
[17:38:04] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests, 10Patch-For-Review: Grant James Forrester access to contint-admins and contint-docker, and to the ciadmin LDAP group - https://phabricator.wikimedia.org/T223137 (10Jdforrester-WMF) I think I now have access; will confirm...
[17:39:44] <wikibugs>	 (03PS1) 10Mholloway: Run service pipeline test for mediawiki/services/wikifeeds [integration/config] - 10https://gerrit.wikimedia.org/r/510581
[17:41:26] <wikibugs>	 (03PS2) 10Mholloway: Run service pipeline tests for mediawiki/services/wikifeeds [integration/config] - 10https://gerrit.wikimedia.org/r/510581
[17:44:19] <wikibugs>	 (03CR) 10Mholloway: "I'm trying to ensure that the new wikifeeds service (split off from mobileapps) is "born Kubernetes" (per https://wikitech.wikimedia.org/w" [integration/config] - 10https://gerrit.wikimedia.org/r/510581 (owner: 10Mholloway)
[17:53:11] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538 (owner: 10Mholloway)
[17:56:02] <wikibugs>	 (03Merged) 10jenkins-bot: Add CI dependencies for WikimediaEditorTasks [integration/config] - 10https://gerrit.wikimedia.org/r/510538 (owner: 10Mholloway)
[17:56:18] <wikibugs>	 10Release-Engineering-Team, 10Scap, 10MediaWiki-ResourceLoader, 10Patch-For-Review, and 2 others: Scap deployments are not purging MessageBlobStore (was: Stale localized messages) - https://phabricator.wikimedia.org/T222539 (10thcipriani) >>! In T222539#5169148, @thcipriani wrote: >>>! In T222539#5169146,...
[17:57:51] <wikibugs>	 10Continuous-Integration-Config, 10phan: phan issue due to a composer dev dependency - https://phabricator.wikimedia.org/T223397 (10hashar)
[17:57:59] <wikibugs>	 (03CR) 10Hashar: [C: 03+2] "Filled as T223397" [integration/config] - 10https://gerrit.wikimedia.org/r/508320 (https://phabricator.wikimedia.org/T189567) (owner: 10Hashar)
[18:13:20] <wikibugs>	 (03PS2) 10Dduvall: pipeline: Allow for execution of a single pipeline [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510274
[18:13:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Thank you to whoever has been following  my progress and updating the *-beta.wmflabs.org proxies. I haven't...
[18:14:08] <wikibugs>	 (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510274 (owner: 10Dduvall)
[18:15:59] <wikibugs>	 10Continuous-Integration-Config, 10phan: phan issue due to a composer dev dependency - https://phabricator.wikimedia.org/T223397 (10hashar) Before switching to Quibble, we were using `docker-registry.wikimedia.org/releng/ci-src-setup:0.2.9` and its entrypoint `setup-mwext.sh`: ` lang=sh,name=dockerfiles/ci-src...
[18:22:40] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10mcruzWMF) >>! In T223102#5184577, @hdothiduc wrote: > Oh cool! > I think, in an effort to direct atten...
[18:38:33] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] "Looks sane." [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510274 (owner: 10Dduvall)
[18:38:47] <wikibugs>	 10Continuous-Integration-Config, 10Proton, 10Reading-Infrastructure-Team-Backlog: chromium-renderer/deploy does not run CI tests - https://phabricator.wikimedia.org/T223405 (10Tgr)
[18:43:20] <wikibugs>	 10Continuous-Integration-Config, 10phan: phan issue due to a composer dev dependency - https://phabricator.wikimedia.org/T223397 (10hashar) https://integration.wikimedia.org/ci/job/mwext-php70-phan-docker/28380/consoleFull has the `PhanRedefinedInheritedInterface` error.  There is no mediawiki/vendor: ` 00:00:...
[18:51:32] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) cxserver runs but does not seem to know about the basics: ` krenair@deployment-docker-cxserver01:~$ curl loc...
[18:58:25] <wikibugs>	 10Beta-Cluster-Infrastructure, 10serviceops, 10Patch-For-Review: Puppet broken on VMs in deployment-prep - https://phabricator.wikimedia.org/T221654 (10Krenair) 05Open→03Resolved a:03Krenair Cxserver migration commits are https://gerrit.wikimedia.org/r/#/c/operations/puppet/+/510588/ and https://gerrit...
[18:58:46] <wikibugs>	 10Beta-Cluster-Infrastructure: Figure out future for newly created deployment-prep jessie instances - https://phabricator.wikimedia.org/T218609 (10Krenair) a:03Eevans
[19:05:18] <wikibugs>	 (03CR) 10Jforrester: [C: 03+1] "Addressed two of your FIXMEs in follow-ups. :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/510496 (https://phabricator.wikimedia.org/T221410) (owner: 10Hashar)
[19:07:37] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[19:08:12] <wikibugs>	 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair)
[19:09:00] <wikibugs>	 10Gerrit, 10Code-Health-Metrics: Make it easier to get to code health report in SonarQube from Gerrit - https://phabricator.wikimedia.org/T223407 (10kaldari)
[19:09:04] <hashar>	 so
[19:13:23] <wikibugs>	 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair)
[19:14:48] <wikibugs>	 10Phabricator: Experiment with php 7.1 on phabricator - https://phabricator.wikimedia.org/T151070 (10Jdforrester-WMF) Experimentation done? :-)
[19:15:56] <wikibugs>	 10Phabricator: Experiment with php 7.1 on phabricator - https://phabricator.wikimedia.org/T151070 (10Dzahn) We could rename this to PHP 7.2.
[19:16:18] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.1 on phabricator - https://phabricator.wikimedia.org/T151070 (10Dzahn)
[19:17:36] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.2 on phabricator - https://phabricator.wikimedia.org/T151070 (10Paladox)
[19:17:56] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.2 on phabricator - https://phabricator.wikimedia.org/T151070 (10Jdforrester-WMF) Is there a task beyond T190568 under which "move Phab to PHP72" would go?
[19:20:38] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.2 on phabricator - https://phabricator.wikimedia.org/T151070 (10Dzahn) No, i think the one you linked is the best match currently.
[19:20:40] <wikibugs>	 10Beta-Cluster-Infrastructure: Migrate away from Debian Jessie to Debian Stretch - https://phabricator.wikimedia.org/T218729 (10Krenair)
[19:22:03] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.2 on phabricator - https://phabricator.wikimedia.org/T151070 (10Paladox) Note: We already have support for this in phabricator's puppet module, we just need to tweak so that it best handles phabricator's load.
[19:26:00] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Watching / External), 10Operations, 10serviceops, 10Patch-For-Review: Reimage both phab1001 and phab2001 to stretch - https://phabricator.wikimedia.org/T190568 (10Jdforrester-WMF)
[19:26:04] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Experiment with php 7.2 on phabricator - https://phabricator.wikimedia.org/T151070 (10Jdforrester-WMF)
[19:26:56] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Move Phabricator from PHP 7.0 to PHP 7.2 - https://phabricator.wikimedia.org/T151070 (10Jdforrester-WMF)
[19:28:45] <wikibugs>	 (03PS1) 10Dduvall: pipeline: Use registry discovery name when registering images [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510600
[19:29:13] <wikibugs>	 10Phabricator, 10PHP 7.2 support: Move Phabricator from PHP 7.0 to PHP 7.2 - https://phabricator.wikimedia.org/T151070 (10Dzahn) phab1001, current prod server on jessie is on PHP 5.    phab1003, upcoming stretch server is on PHP 7.2 but we still want to setup php-fpm   Phabricator wouldn't work on PHP 7.0
[19:33:49] <wikibugs>	 (03CR) 10PipelineBot: "pipeline-dashboard: service-pipeline-test" [integration/pipelinelib] - 10https://gerrit.wikimedia.org/r/510600 (owner: 10Dduvall)
[19:45:50] <wikibugs>	 (03PS1) 10Dduvall: service-pipeline: Templates for projects using .pipeline/config.yaml [integration/config] - 10https://gerrit.wikimedia.org/r/510601
[19:45:52] <wikibugs>	 (03PS1) 10Dduvall: service-pipeline: Define pipeline builder jobs for blubber [integration/config] - 10https://gerrit.wikimedia.org/r/510602
[19:48:13] <wikibugs>	 (03CR) 10Dduvall: service-pipeline: Templates for projects using .pipeline/config.yaml (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/510601 (owner: 10Dduvall)
[20:00:40] <wikibugs>	 (03PS1) 10Dduvall: service-pipeline: Use registries defined by pipelinelib [integration/config] - 10https://gerrit.wikimedia.org/r/510605
[20:23:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Editing-team, 10Release Pipeline, 10serviceops, and 4 others: Migrate Beta cluster services to use Kubernetes - https://phabricator.wikimedia.org/T220235 (10Krenair) Niklas found the solution to that one: set `mw_host: en.wikipedia.beta.wmflabs.org` in conf - otherwise it d...
[21:13:48] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Fix phan parameter order [integration/config] - 10https://gerrit.wikimedia.org/r/510458 (https://phabricator.wikimedia.org/T219114) (owner: 10Awight)
[21:14:12] <wikibugs>	 (03CR) 10Thiemo Kreuz (WMDE): [C: 03+1] Reduce memory usage by omitting progress bar [integration/config] - 10https://gerrit.wikimedia.org/r/510459 (https://phabricator.wikimedia.org/T219114) (owner: 10Awight)
[21:38:51] <wikibugs>	 10Continuous-Integration-Config, 10Proton, 10Reading-Infrastructure-Team-Backlog: chromium-renderer/deploy does not run CI tests - https://phabricator.wikimedia.org/T223405 (10hashar) We have first setup `mediawiki/services/chromium-render` with some hacks for puppeteer. Probably the /deploy was not a thing...
[21:39:14] <wikibugs>	 10Continuous-Integration-Config, 10Proton, 10Reading-Infrastructure-Team-Backlog: chromium-renderer/deploy does not run CI tests - https://phabricator.wikimedia.org/T223405 (10hashar) p:05Triage→03Normal
[21:42:26] <wikibugs>	 10Continuous-Integration-Config, 10Code-Health-Metrics: Make it easier to get to code health report in SonarQube from Gerrit - https://phabricator.wikimedia.org/T223407 (10hashar) We can tweak the URL for success and failures, there are plenty of examples in zuul/layout.yaml. That is done by adding a job rule...
[22:43:15] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) >>! In T223102#5184577, @hdothiduc wrote: > - we could hide the //Status//, //Priority//, //...
[22:51:59] <wikibugs>	 10Phabricator, 10CommRel-Design, 10CommRel-Internals: Create Phabricator form for CommRel-Design and Comms requests and add a link to it in the "Star" dropdown - https://phabricator.wikimedia.org/T223102 (10Aklapper) >>! In T223102#5185432, @mcruzWMF wrote: > We need to implement this for design and communit...