[00:20:32] PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<40.00%) [00:46:22] (03PS1) 10Legoktm: Validate .phpcs.xml files using phpcs.xsd [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424970 [00:47:26] (03CR) 10Legoktm: "I'm not really sure how useful this would be, and exactly how we would run it, but it seemed pretty easy to set up." [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424970 (owner: 10Legoktm) [01:22:10] 10Phabricator: List all linked patches at the top of a Phab ticket - https://phabricator.wikimedia.org/T191755#4115415 (10Samwilson) [01:38:33] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115425 (10Legoktm) [01:39:13] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115432 (10Legoktm) [01:39:42] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115433 (10Legoktm) [02:18:09] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115448 (10Legoktm) [02:18:12] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle CategoryTree extension with MW 1.31 - https://phabricator.wikimedia.org/T191735#4115449 (10Legoktm) [02:19:19] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115451 (10Legoktm) [02:19:21] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115206 (10Legoktm) [02:19:24] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle AntiSpoof extension with MW 1.31 - https://phabricator.wikimedia.org/T191736#4115452 (10Legoktm) [02:21:09] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4115453 (10Legoktm) [02:21:14] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115455 (10Legoktm) [02:21:55] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle MobileFrontend extension with MW 1.31 - https://phabricator.wikimedia.org/T191734#4115457 (10Legoktm) [02:23:13] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115458 (10Legoktm) [02:23:26] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115139 (10Legoktm) AFAICT Thanks is useless without Echo, but there's no hard dependency set in extension.json? [02:23:38] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115462 (10Legoktm) [02:23:43] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115461 (10Legoktm) [02:24:35] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle AbuseFilter extension with MW 1.31 - https://phabricator.wikimedia.org/T191740#4115463 (10Legoktm) [02:26:55] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Replace Text extension with MW 1.31 - https://phabricator.wikimedia.org/T191741#4115464 (10Legoktm) [02:27:00] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle CodeEditor extension with MW 1.31 - https://phabricator.wikimedia.org/T191742#4115473 (10Legoktm) [02:28:34] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Minerva Neue skin with MW 1.31 - https://phabricator.wikimedia.org/T191743#4115475 (10Legoktm) [02:29:05] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle MobileFrontend extension with MW 1.31 - https://phabricator.wikimedia.org/T191734#4115477 (10Legoktm) [02:29:11] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Minerva Neue skin with MW 1.31 - https://phabricator.wikimedia.org/T191743#4115175 (10Legoktm) [02:29:25] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle MobileFrontend extension with MW 1.31 - https://phabricator.wikimedia.org/T191734#4115094 (10Legoktm) [02:29:29] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Minerva Neue skin with MW 1.31 - https://phabricator.wikimedia.org/T191743#4115175 (10Legoktm) [02:30:41] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle MobileFrontend extension with MW 1.31 - https://phabricator.wikimedia.org/T191734#4115094 (10Legoktm) [02:31:53] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115489 (10Legoktm) [02:31:54] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115488 (10Legoktm) [02:31:57] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115130 (10Legoktm) [02:32:01] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle LoginNotify extension with MW 1.31 - https://phabricator.wikimedia.org/T191746#4115206 (10Legoktm) [02:33:13] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115130 (10Legoktm) [02:33:16] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115492 (10Legoktm) [02:33:19] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Echo extension with MW 1.31 - https://phabricator.wikimedia.org/T191738#4115130 (10Legoktm) [02:33:22] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Thanks extension with MW 1.31 - https://phabricator.wikimedia.org/T191739#4115139 (10Legoktm) [02:34:22] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Multimedia Viewer extension with MW 1.31 - https://phabricator.wikimedia.org/T191744#4115496 (10Legoktm) [02:35:06] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle OATHAuth extension with MW 1.31 - https://phabricator.wikimedia.org/T191745#4115497 (10Legoktm) [02:39:39] 10MediaWiki-Releasing, 10MW-1.31-release: Bundle Scribunto extension with MW 1.31 - https://phabricator.wikimedia.org/T191737#4115498 (10CCicalese_WMF) While Scribunto does not require CodeEditor, if both are enabled, the following should probably be set: $wgScribuntoUseCodeEditor = true; Similarly, Scribunt... [03:36:15] Project mediawiki-core-code-coverage-php7 build #195: 04STILL FAILING in 36 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/195/ [04:13:58] 10Phabricator: List all linked patches at the top of a Phab ticket - https://phabricator.wikimedia.org/T191755#4115542 (10Aklapper) That would have been the case by migrating from Gerrit to Differential (which is not planned anymore). I'm afraid this would require quite some custom code changes and maintaining... [04:27:21] Project mediawiki-core-code-coverage build #3434: 04STILL FAILING in 1 hr 27 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3434/ [04:33:10] 10Phabricator: List all linked patches at the top of a Phab ticket - https://phabricator.wikimedia.org/T191755#4115564 (10Samwilson) So if we're sticking with Gerrit forever, then the work to implement this might be worth it? (I'm not //not// volunteering to do it, by the way.) And yeah, I'm not sure where the... [05:32:24] 10Phabricator: List all linked patches at the top of a Phab ticket - https://phabricator.wikimedia.org/T191755#4115625 (10Aklapper) There might be a bunch of #Gerritbot tasks (closed or open) which cover similar topics already. [05:33:27] 10Beta-Cluster-Infrastructure, 10Operations, 10HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2627786 (10Joe) I think this task is resolved as it's about the MediaWiki appservers and AFAICS they're all converted to jessie at least. [05:34:50] 10Beta-Cluster-Infrastructure, 10Operations, 10HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#4115633 (10Joe) 05Open>03Resolved [06:05:52] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4115654 (10Joe) [06:05:57] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10Puppet: deployment-etcd-01 puppet errors - https://phabricator.wikimedia.org/T191107#4115653 (10Joe) 05Open>03Resolved [06:13:09] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:41:02] 10Project-Admins: Create ores-support-checklist project in phabricator - https://phabricator.wikimedia.org/T191724#4115671 (10Aklapper) 05Open>03Resolved a:03Aklapper Created https://phabricator.wikimedia.org/project/view/3327/ Do you plan to disable https://github.com/wiki-ai/ores-support-checklist/issues ? [07:10:32] RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK [07:17:53] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [07:20:15] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:20:49] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [07:24:28] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:29:24] PROBLEM - Puppet errors on deployment-mediawiki04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:43:26] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:56:16] 10Project-Admins: Create ores-support-checklist project in phabricator - https://phabricator.wikimedia.org/T191724#4115784 (10Ladsgroup) Done. thanks! [08:03:01] 10Phabricator: Get every previous Bugzilla user with actions have an account, so that actions aren't attributed to bzimport - https://phabricator.wikimedia.org/T847#4115810 (10Aklapper) 05Open>03declined No reply to last questions. Reflecting reality by closing task as declined as my understanding is that da... [08:03:43] 10Gerrit, 10Patch-For-Review: Switch to mariadb java connector - https://phabricator.wikimedia.org/T176164#4115813 (10Marostegui) I don't think this needs any DBA approval, as we are not responsible of how you connect to the DB itself ;-) [08:07:31] 10Phabricator, 10Community-Liaisons, 10Developer-Relations, 10Developer-Wishlist (2017), 10Goal: Consolidate the many tech events calendars in Phabricator's calendar - https://phabricator.wikimedia.org/T1035#4115843 (10Aklapper) [See Also] Similar task for (non-technical) Education events, wondering wher... [08:18:10] 10Continuous-Integration-Infrastructure: CI: run tests with multiple Python3 versions - https://phabricator.wikimedia.org/T191764#4115860 (10Volans) [08:29:45] 10Continuous-Integration-Infrastructure, 10Operations-Software-Development, 10Patch-For-Review: cumin 3.0.1-1 is broken on labs master - https://phabricator.wikimedia.org/T188112#4115897 (10Volans) Patch updated to overcome this problem, once reviewed and merged it should solve the issue. [10:34:27] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4052385 (10hoo) Still happening: https://integration.wikimedia.org/ci/job/mwext-mw-selenium-composer... [11:19:31] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-mira: puppet broken 2018-04-09 - https://phabricator.wikimedia.org/T191786#4116350 (10MarcoAurelio) [11:26:18] eddiegp: hi, d-mira broken again - T191786 [11:26:19] T191786: deployment-mira: puppet broken 2018-04-09 - https://phabricator.wikimedia.org/T191786 [11:29:03] Hauskatze: looking. [11:29:20] Seems like all the appservers, the jobrunner and tin is broken [11:30:00] (on beta that is, in case anyone reads along and starts to panic) [11:30:34] head-desk [11:32:25] Duplicate declaration: Apt::Repository[hhvm-icu57] is already declared in file /etc/puppet/modules/profile/manifests/beta/icu57.pp:3; cannot redeclare at /etc/puppet/modules/profile/manifests/mediawiki/hhvm.pp:102 at /etc/puppet/modules/profile/manifests/mediawiki/hhvm.pp:102:9 on node deployment-mira.deployment-prep.eqiad.wmflabs [11:32:46] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Patch-For-Review: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4116382 (10hoo) After raising the timeout from 10s to 15s, there was another f... [11:33:52] same error on tin [11:35:28] Yeah I think I know where that comes from. [11:37:09] Hauskatze: see -operations [11:37:21] * Hauskatze looks [11:37:53] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-mira: puppet broken 2018-04-09 - https://phabricator.wikimedia.org/T191786#4116392 (10EddieGP) p:05Triage>03Unbreak! Puppet broken on all the appservers, jobrunners and deployment servers in beta. [11:43:52] 10Beta-Cluster-Infrastructure, 10Puppet: deployment-mira: puppet broken 2018-04-09 - https://phabricator.wikimedia.org/T191786#4116399 (10EddieGP) 05Open>03Resolved a:03EddieGP Fixed by changing hiera, should recover with the next puppet run. [11:46:55] !log maurelio@deployment-mira:~$ sudo puppet agent -tv to fix T191786 (success: Notice: Applied catalog in 27.11 seconds) [11:46:57] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:46:57] T191786: deployment-mira: puppet broken 2018-04-09 - https://phabricator.wikimedia.org/T191786 [11:52:52] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [11:53:28] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:25] RECOVERY - Puppet errors on deployment-mediawiki04 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:14] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [12:00:48] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [12:04:28] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:35:52] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4116508 (10CCicalese_WMF) [12:38:39] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:41:31] 10Beta-Cluster-Infrastructure, 10Operations, 10media-storage, 10Patch-For-Review, 10Puppet: Puppet broken on deployment-ms-be0[34] with evaluation error in swift module - https://phabricator.wikimedia.org/T184236#4116525 (10MarcoAurelio) ``` Linux deployment-ms-be04 4.9.0-0.bpo.5-amd64 #1 SMP Debian 4.9.... [12:54:28] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:03:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, and 2 others: selenium test for Wikibase is unstable - https://phabricator.wikimedia.org/T189762#4116593 (10hoo) With the (very high) 90s timeout, it took me quite some tries, but I m... [13:36:32] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [13:54:32] 10Gerrit, 10Release-Engineering-Team (Next), 10DBA, 10Operations, 10Patch-For-Review: Gerrit is failing to connect to db on gerrit2001 thus preventing systemd from working - https://phabricator.wikimedia.org/T176532#4116752 (10Marostegui) [14:03:30] 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release, 10Patch-For-Review: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4116771 (10Tgr) [14:13:20] (03PS1) 10Hashar: Skip webdriver:test when tests/selenium doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/425057 [14:16:12] (03CR) 10Hashar: [C: 032] Skip webdriver:test when tests/selenium doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/425057 (owner: 10Hashar) [14:16:39] (03Merged) 10jenkins-bot: Skip webdriver:test when tests/selenium doesn't exist [integration/quibble] - 10https://gerrit.wikimedia.org/r/425057 (owner: 10Hashar) [14:40:02] hashar: zuul seems stuck [14:40:40] zuul seems fine [14:41:17] there are tests running for 25+ minutes [14:41:49] AF patches for example are all being tested as dependencies when they ain't :P [14:41:54] yes, because alot of repos are being tested [14:41:56] :| * [14:42:02] wikibase takes up alot of tests [14:53:14] is greg-g still out? if so then i need no_justification to comment on https://phabricator.wikimedia.org/T191704 =] [14:53:21] deployment access for Imarlier [15:20:21] zeljkof: hi! have you seen https://phabricator.wikimedia.org/T191537 already by any chance? We're getting kinda blocked on this [15:22:17] 10Continuous-Integration-Infrastructure, 10Lexicographical data, 10Wikidata, 10Browser-Tests, 10User-zeljkofilipin: MediaWiki core's node selenium tests flaky when run as part of mwext-mw-selenium-node-composer-jessie job - https://phabricator.wikimedia.org/T191537#4117205 (10zeljkofilipin) [15:23:44] leszek_wmde: sorry, will take a look, I was busy with some urgent tasks [15:24:24] zeljkof: sure thing. no need to be sorry. If you have any hints, we're open to suggestions what to try out [15:27:43] PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:33:43] PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:37:14] Project mediawiki-core-code-coverage-php7 build #196: 04STILL FAILING in 37 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage-php7/196/ [15:44:40] 10Deployments, 10Release-Engineering-Team (Next), 10MediaWiki-Maintenance-scripts, 10PHP 7.0 support, 10Patch-For-Review: php5 is missing on deploy1001 which breaks foreachwiki & l10nupdate - https://phabricator.wikimedia.org/T190909#4117292 (10Krinkle) [15:44:48] 10Deployments, 10Release-Engineering-Team (Kanban), 10Operations, 10Beta-Cluster-reproducible, and 2 others: Switch mwscript from Zend PHP5 to default php alternative (e.g. HHVM or PHP7) - https://phabricator.wikimedia.org/T146285#4117288 (10Krinkle) 05Open>03Resolved a:03fgiunchedi Per {ff4db0c87156... [15:45:54] 10Deployments, 10Release-Engineering-Team (Next), 10MediaWiki-Maintenance-scripts, 10PHP 7.0 support, 10Patch-For-Review: php5 is missing on deploy1001 which breaks foreachwiki & l10nupdate - https://phabricator.wikimedia.org/T190909#4087254 (10Krinkle) 05Open>03Resolved Per {ff4db0c87156035d79c0378a... [15:59:57] 10Project-Admins, 10Africa-Wikimedia-Developers: Project work board request for WikiFundi - https://phabricator.wikimedia.org/T186754#4117354 (10Anthere) Hi Aklapper. Well, keep it open. It is still unclear to me what should be done (though I begin to suspect it will be best to let that thing slowly sink) [16:01:51] 10Continuous-Integration-Infrastructure, 10PHP 7.0 support, 10Patch-For-Review: Support PHP 7 in CI infra - https://phabricator.wikimedia.org/T144872#4117363 (10Krinkle) [16:03:47] PROBLEM - Puppet errors on integration-cumin is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [16:04:09] oh seems mediawiki has had a big increase in performance in mw 1.31 :) [16:06:44] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#4117390 (10Krinkle) [16:08:02] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136#4117392 (10Krinkle) [16:08:13] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136#3681887 (10Krinkle) [16:08:49] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136#3681887 (10Krinkle) 05Open>03stalled p:05Triage>03Normal [16:09:01] 10MediaWiki-Codesniffer, 10MediaWiki-General-or-Unknown, 10Technical-Debt: Encourage type hints for function parameters and return after moving MediaWiki to PHP 7 - https://phabricator.wikimedia.org/T178136#3681887 (10Krinkle) [16:11:15] 10Continuous-Integration-Config, 10MediaWiki-General-or-Unknown, 10PHP 7.0 support: Make Wikimedia CI run PHP in either PHP 7.0+ or HHVM - https://phabricator.wikimedia.org/T190547#4117403 (10Krinkle) [16:11:19] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#4117402 (10Krinkle) [16:11:37] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Someday): Get rid of Zend 5.5 tests for wmf branches - https://phabricator.wikimedia.org/T94149#1156282 (10Krinkle) [16:24:02] Project mediawiki-core-code-coverage build #3435: 04STILL FAILING in 1 hr 24 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/3435/ [16:26:58] 10MediaWiki-Codesniffer, 10MediaWiki-extensions-Variables: Allow configuring MediaWiki.NamingConventions.ValidGlobalName.wgPrefix to allow additional prefixes - https://phabricator.wikimedia.org/T191812#4117477 (10MGChecker) [16:27:20] 10MediaWiki-Codesniffer, 10MediaWiki-extensions-Variables: Allow configuring MediaWiki.NamingConventions.ValidGlobalName.wgPrefix to allow additional prefixes - https://phabricator.wikimedia.org/T191812#4117490 (10MGChecker) [16:35:12] 10Gerrit, 10Developer-Relations (Jan-Mar-2018), 10Documentation: [[mw:Gerrit/Tutorial]] is way too much information for new contributors - https://phabricator.wikimedia.org/T161901#3146787 (10Dvorapa) Hi, I just want to announce I proposed Getting started page to rename as its purpose is slightly different t... [16:40:53] (03PS1) 10Hashar: On Docker write logs to /log [integration/quibble] - 10https://gerrit.wikimedia.org/r/425088 [16:42:49] (03CR) 10Hashar: [C: 032] On Docker write logs to /log [integration/quibble] - 10https://gerrit.wikimedia.org/r/425088 (owner: 10Hashar) [16:43:15] (03Merged) 10jenkins-bot: On Docker write logs to /log [integration/quibble] - 10https://gerrit.wikimedia.org/r/425088 (owner: 10Hashar) [16:43:18] (03PS1) 10Hashar: quibble to 0.0.6 [integration/config] - 10https://gerrit.wikimedia.org/r/425089 [16:43:31] (03CR) 10Hashar: [C: 032] quibble to 0.0.6 [integration/config] - 10https://gerrit.wikimedia.org/r/425089 (owner: 10Hashar) [16:44:44] (03Merged) 10jenkins-bot: quibble to 0.0.6 [integration/config] - 10https://gerrit.wikimedia.org/r/425089 (owner: 10Hashar) [16:46:18] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10herron) This was approved at the Monday SRE meeting so I'll work on creating a patch now [16:46:55] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4117565 (10herron) [16:50:56] 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Operations-Software-Development, 10Patch-For-Review: New tool to track package updates/status for hosts and images (debmonitor) - https://phabricator.wikimedia.org/T167504#4117582 (10Volans) a:03Volans [16:54:37] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [16:57:25] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [17:04:37] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:25] Hm.. is there a phan rule for unused methods? [17:06:54] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4117671 (10dduvall) So if we're following option #1, we'll need to expose packaged charts somewhere central. Does https://integr... [17:07:07] we'll probably need some opt-out from cases intended for extensions that don't have tests, or for custom sub classes not used by default. But would still be useful probably [17:07:17] Especially when forgetting to remove stuff unused due to a current commit. [17:11:47] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4117687 (10herron) 05Open>03Resolved a:03herron @thcipriani is now a member of `contint-roots` on `contint1... [17:12:06] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4117690 (10herron) [17:12:13] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10herron) [17:14:29] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4117708 (10thcipriani) Looks to be working, thanks @herron! [17:23:00] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<30.00%) [17:33:00] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found) [17:38:54] 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4117789 (10awight) Here's the current scap error from beta cluster deployment: ``` cd /srv/deployment/ores/deploy git fetch # This branch has a git-lfs submodule, in "submodules/assets"... [17:39:48] 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4117797 (10awight) @mmodell I'm still stuck, see the previous comment. Maybe it has to do with URL rewriting? [17:40:39] 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4117799 (10mmodell) @awight: thanks, I haven't had much chance to work on this due to the train taking up most of my time these past two weeks. I'm not deploying the train this week so I... [17:44:58] PROBLEM - SSH on integration-slave-docker-1003 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [17:45:36] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4117802 (10dduvall) [17:45:50] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4117814 (10dduvall) [17:45:52] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4117813 (10dduvall) [17:46:12] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Host packaged helm charts at https://integration.wikimedia.org/charts - https://phabricator.wikimedia.org/T191821#4117802 (10dduvall) p:05Triage>03Normal [17:49:50] RECOVERY - SSH on integration-slave-docker-1003 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [17:50:26] twentyafterfour: Nice, thanks for the heads-up. I’m available any time you want to work on git-lfs, just lemme know. [17:50:29] beh, why do we not have our own CI for postgres [17:51:11] paladox "oh seems mediawiki has had a big increase in performance in mw 1.31 :)" what makes you say that? [17:52:56] addshore: my site started loading faster after upgrading [17:53:04] Though it seems to be a mixed now [17:53:11] Sometimes loads fasts some times not [17:53:18] 10Continuous-Integration-Infrastructure, 10Patch-For-Review: Jenkins: Run PHPUnit tests on MySQL, PostgreSQL and SQLite - https://phabricator.wikimedia.org/T22343#262107 (10Addshore) Given issues we may be having (https://github.com/SemanticMediaWiki/SemanticMediaWiki/issues/3101) it would be great to have WMF... [18:08:58] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<20.00%) [18:22:33] I don't see git_repo_user in the scap documentation, but https://github.com/wikimedia/maps-kartotherian-deploy/blob/master/scap/scap.cfg#L4 lists it in the config. Is it a deprecated/removed setting? [18:25:28] Also the documentation claims that dsh_targets is relative to /etc/dsh/group but I'm pretty sure that's not true, based on https://github.com/wikimedia/maps-kartotherian-deploy/blob/master/scap/scap.cfg#L8 and https://github.com/wikimedia/mediawiki-services-parsoid-deploy/blob/master/scap/scap.cfg#L8 [18:31:18] git_repo_user is a setting that isn't used internally in scap anymore. It was initially used for having the ssh-user and the user doing git fiddling be different. We abandoned that since no one was using it and it was making a headache. [18:32:08] the dsh_targets list will check in several places and stop when it finds the file: envronment-specific-scap-dir, scap dir, then /etc/dsh/group [18:32:53] if you use an absolute path for dsh_targets, that will also work and it won't try any fallback locations. [18:34:00] OK cool [18:34:04] And what about lock_file ? [18:34:16] That's also a setting used by kartotherian that's undocumented [18:35:01] we got rid of lock_file as well, we now use the name of the service or "mediawiki" in the case of mediawiki. [18:36:33] Nice [18:36:45] So, I want to update the docs to reflect the behavior you described for dsh_targets [18:37:05] And it seems that what's in the git repo under docs/ is not the same as what's on the web? [18:37:42] https://doc.wikimedia.org/mw-tools-scap/_sources/scap3/repo_config.rst.txt is slightly different from docs/intro_repo_config.rst and also differently named [18:37:49] Yet git log says the docs directory hasn't been touched since 2015 [18:38:06] oh? that's not good. What's in docs should be the same as what's on doc.wikimedia.org ... lemme check the job [18:38:12] Strike that, this whole repo hasn't been touched since 2015, I must have the wrong repo [18:38:30] I checked out mediawiki/tools/scap.git based on the "mw-tools-scap" part of the URL [18:38:57] oh! We moved to developing on differential and have been there a while. We've talked about moving back, but haven't really pushed forward with that. [18:39:06] I just realized the same thing [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:39:24] https://phabricator.wikimedia.org/source/scap/ is the canonical source [18:49:06] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4117967 (10awight) As mentioned in my [[ https://phabricator.wikimedia.org... [18:52:53] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (MediaWiki-MultiDC): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#4117974 (10aaron) [19:08:07] OK great so now I have a problem with differential [19:08:12] https://www.irccloud.com/pastebin/MXUawMJr/ [19:08:35] When I run arc diff, it tries to run a command called "coverage" that doesn't exist on my system [19:13:00] 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4118039 (10mmodell) >>! In T180627#4117797, @awight wrote: > @mmodell I'm still stuck, see the previous comment. Maybe it has to do with URL rewriting? Indeed that seems like it might... [19:13:23] 10Release-Engineering-Team (Kanban), 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4118041 (10mmodell) a:03mmodell [19:16:26] ( thcipriani --^^ ) [19:25:29] pip install pytest-coverage [19:25:31] Or pytest-cov [19:25:33] I can't remember [19:25:47] (generally, pip install -r test-requirements.txt) [20:00:24] PROBLEM - SSH on integration-slave-docker-1015 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [20:00:53] no_justification should i set the object limit at 20 or 30mb in All-Avatars? (or should there not be a limit?) [20:01:09] object size limit? Why would a single object need to be 20mb? [20:01:14] Who has a 20mb avatar? [20:01:45] lol trying to limit it [20:01:51] so that they doint upload a 300mb one [20:02:18] png's could be large if they have alot of detail [20:03:58] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found) [20:05:14] RECOVERY - SSH on integration-slave-docker-1015 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [20:10:09] paladox: I'd go with 10 [20:10:14] ok [20:10:20] * paladox sets the limit :) [20:11:08] done [20:11:17] https://gerrit.wikimedia.org/r/plugins/gitiles/All-Avatars/+/0e6204e73f0c7e27c3c4acc3acf35f4e365ce52c [20:20:18] PROBLEM - Free space - all mounts on integration-slave-docker-1006 is CRITICAL: CRITICAL: integration.integration-slave-docker-1006.diskspace.root.byte_percentfree (<44.44%) [20:25:44] no_justification im going to merge https://gerrit.wikimedia.org/r/#/c/424715/ (only a few kb and my profile picture from phab) :) [20:28:30] 10Release-Engineering-Team, 10Scap, 10Operations, 10Scoring-platform-team: Deployment git server can't supply ORES hosts in parallel - https://phabricator.wikimedia.org/T191842#4118429 (10awight) [20:30:46] no_justification https://gerrit.wikimedia.org/r/425127 :) [20:34:35] (03CR) 10Krinkle: [C: 031] "LGTM. Is it feasible to also autofix the deprecated ones?" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [20:36:56] 10Gerrit, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4118454 (10Paladox) Im not sure what the default avatar should be, though chad suggested a stick man :) https://gerrit.wikimedia.org/r/c/425127/ [20:42:37] hashar: Regarding https://phabricator.wikimedia.org/T176097#4118440 - could you help me reproduce that locally? [20:42:55] Krinkle: I tried a bit locally but could not :( [20:43:05] hashar: I mean with quibble. [20:43:08] I have Docker installed :) [20:43:10] ah yeah well [20:43:13] Should be possible right? [20:43:14] I am not sure whether it would reproduce [20:43:17] yeah that should [20:43:41] Does anyone here know how /etc/ssh/ssh_known_hosts is managed on deployment-tin? [20:43:44] If you can reproduce it locally, I'd be happy to take it further and narrow it down as to why it fails. [20:43:51] Puppet is clearly managing it because my modifications to it get wiped out [20:43:55] Krinkle: it failed on my last check of https://gerrit.wikimedia.org/r/#/c/375798/ [20:44:01] But grepping the puppet repo I found it hard to figure out how [20:44:21] RoanKattouw puppet is [20:44:23] try the ssh module [20:44:31] RoanKattouw https://github.com/wikimedia/puppet/blob/production/modules/ssh/templates/sshd_config.erb [20:45:29] Huh wait [20:45:38] I just made a local modification and then ran puppet, and it didn't overwrite it [20:47:17] RoanKattouw: it is done randomly iirc [20:47:31] RoanKattouw: ie not on every puppet run. But i might be wrong [20:47:51] Hmm so I found https://github.com/wikimedia/puppet/blob/production/modules/ssh/templates/known_hosts.erb but that's using a query_resources thing that I don't underestand [20:49:28] Krinkle: err it failed on https://gerrit.wikimedia.org/r/#/c/375798/ :) [20:49:29] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4118493 (10awight) Update: I deployed as a safe migration, and the new vir... [20:49:39] against REL1_29 [20:50:02] Krinkle: but on a second check that does not fail. So there is a race condition somewhere :( [20:55:01] Krinkle: as for quibble in theory: docker pull docker-registry.wikimedia.org/releng/quibble-stretch then: docker run -e ZUUL_BRANCH=REL1_29 quibble [20:55:29] Krinkle: but to get local caches, reuse local git repositories as mirrors it is a bit more complicated. README.md has some info at the root of integration/quibble [20:55:38] hashar: Does it clone mw, or do I need to run it from a directory with an existing clone? [20:55:41] Krinkle: I wanna add some doc about it eventualyl [20:55:45] (The latter would be nice, but either is fine) [20:55:52] it supports both [20:56:31] --skip-zuul would bypass zuul-cloner entirely and thus use whatever is mounted in /workspace/src [20:57:06] so if you have a local checkout you can try: docker run -v /path/to/stuff:/workspace/src quibble:latest --skip-zuul --skip-dep [20:57:13] (--skip-dep skips composer/npm install) [20:57:36] the thing is it runs all the tests serially :( [20:57:45] there is no way yet to only run one of the build steps [20:57:53] it is very rough :( [21:05:57] Krinkle: I will add some doc eventually [21:06:46] for now I gotta sleep .. [21:07:21] Au revoir, o/ [21:08:49] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [21:15:07] RoanKattouw: I hit this a while ago, but never found the bottom of it. My investigation notes are mostly on https://phabricator.wikimedia.org/T159332 [21:15:40] IIRC the end of the thread I was pulling on was that puppetdb was giving back different results at different times [21:24:22] (03CR) 10Legoktm: "Maybe? I'll let someone else add that in a separate patch if they want, I don't see much value in it right now since we already migrated " (032 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [21:46:42] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4118685 (10awight) I can't tell whether the fetch check script is failing,... [21:48:49] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [21:49:20] no_justification hmm, i doint think the stick figure looks nice as it does not have a background, thus with the think black lines against a grey background makes it not look nice [21:49:22] see https://gerrit.git.wmflabs.org/r/q/owner:paulfkeffer%40gmail.com [21:49:35] compared to https://gerrit.git.wmflabs.org/r/q/owner:thomasmulhall410%40yahoo.com [22:00:04] 10Gerrit, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4118718 (10Tgr) I'd go with something slightly more professional, e.g. [[https://commons.wikimedia.org/wiki/File:Profile_avatar_placeholder_large.png|this]]. A plain light grey or light blue background is not a t... [22:03:10] 10Gerrit, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4118723 (10Paladox) @Tgr ah thank you, looks good on https://gerrit.git.wmflabs.org/r/c/3/?polygerrit=1 [22:15:20] 10Gerrit, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4118755 (10demon) Professional? The cloud services logo is a unicorn. [22:15:53] 10Gerrit, 10Patch-For-Review: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4118757 (10demon) A unicorn, I'll add, that was my suggestion ;-) [22:27:48] tgr hmm, i have to add a license for the image in the repo too [22:36:31] Do we have any open source ones? [22:36:40] or i guess i will have to include the license [22:41:12] 10Project-Admins: Create a Section-Editing-Support Goal under the Parsoid project - https://phabricator.wikimedia.org/T191854#4118864 (10ssastry) [22:49:55] paladox: find a public domain image from commons? [22:50:21] legoktm hi, yes i have been searching [22:50:37] though haven't found anything as nice as that image [22:51:28] https://commons.wikimedia.org/wiki/File:Facebook_default_male_avatar.gif is PD [22:52:16] but that's gendered, so meh [22:52:30] yeh hmm [22:52:47] could we just use the community logo as a placeholder? [22:53:00] https://meta.wikimedia.org/wiki/Wikimedia_Community_Logo [22:53:56] /60https://commons.wikimedia.org/wiki/File:Missing_avatar.svg ? [22:54:10] legoktm i guess so [22:54:13] * paladox testsss [22:54:40] https://commons.wikimedia.org/wiki/File:Missing_avatar.svg would do, I think [22:54:52] oh, that's nice [22:54:52] well that fits perfectly [22:55:24] :) [22:55:28] legoktm oh so we want to use the missing avatar one? [22:55:56] I like it yeah [22:55:59] ok [22:57:24] (03CR) 10Jforrester: [C: 031] Add common autofix replacements for invalid license tag sniff [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/424737 (owner: 10Legoktm) [22:58:22] legoktm looks nice here: [22:58:26] https://gerrit.git.wmflabs.org/r/c/3/?polygerrit=1 [22:58:45] it fits perfectly heh [23:00:46] https://gerrit.wikimedia.org/r/#/c/425127/ [23:03:57] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4118918 (10awight) Do we have to install the `python3-setuptools` package?... [23:05:56] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#1666838 (10atgo) Hi @mmodell ! I'd like to use "deadline" tasks, and I see that they're tagged as such in the workboard view. What would be more helpful is if the deadline (date) appeared there rather than the task type. For tas... [23:10:39] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4118924 (10mmodell) >>! In T93499#4118919, @atgo wrote: > Hi @mmodell ! I'd like to use "deadline" tasks, and I see that they're tagged as such in the workboard view. What would be more helpful is if the deadline (date) appeared... [23:12:55] 10Phabricator: Add support for task types - https://phabricator.wikimedia.org/T93499#4118930 (10atgo) Thanks @mmodell! Much appreciated. [23:24:34] 10Phabricator: Email sometimes not being sent when a task is created - https://phabricator.wikimedia.org/T182549#3826879 (10EddieGP) I've not received a mail for the creation of T191701, found this task and made a few tests in a local phabricator instance: - If I set maniphest notification settings to **ignore*... [23:36:21] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4118975 (10EddieGP) [23:36:23] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-cache-text04 due to varnishkafka issues - https://phabricator.wikimedia.org/T184234#4118973 (10EddieGP) 05Open>03Resolved Seems fixed. ``` eddie@deployment-cache-text04:~$ sudo puppet agent -tv Info: Using configured environment 'product... [23:39:15] 10Beta-Cluster-Infrastructure, 10Puppet, 10Tracking: Deployment-prep hosts with puppet errors (tracking) - https://phabricator.wikimedia.org/T132259#4118983 (10EddieGP) [23:39:17] 10Beta-Cluster-Infrastructure: deployment-fluorine puppet failure due to '/usr/sbin/usermod -u 10003 datasets' returned 4: usermod: UID '10003' already exists - https://phabricator.wikimedia.org/T117028#4118981 (10EddieGP) 05Open>03Resolved deployment-fluorine no longer exists and deployment-fluorine02 doesn... [23:48:29] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [23:50:41] 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4119011 (10EddieGP) 05Open>03Resolved Puppet is fine on both hosts now. so this seems resolved. Thanks @Ottomata! [23:52:17] 10Beta-Cluster-Infrastructure: Could not find class role::etcd::common for deployment-conf03.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T168520#3367026 (10EddieGP) Puppet run on this instance succeeds. Can this be closed now? [23:58:29] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0]