[00:31:15] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [01:03:46] 10Phabricator, 10Operations: Phabricator is loading really slowly - https://phabricator.wikimedia.org/T191361#4102938 (10greg) [01:03:54] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4102940 (10greg) [02:08:30] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [02:48:30] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [02:59:30] 10Project-Admins, 10Developer-Relations: Create 4 event projects - https://phabricator.wikimedia.org/T191372#4103048 (10Rfarrand) p:05Triage>03Normal [03:07:08] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103063 (10demon) Remember, Differential != Diffusion. There's a pretty obvious tail when it comes to Differential. I ran the numbers the other day and it was Scap, Blubb... [03:07:26] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103064 (10demon) 05stalled>03declined [03:10:52] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Kanban): Move Scap development to Gerrit - https://phabricator.wikimedia.org/T191373#4103068 (10demon) p:05Triage>03Normal [06:56:58] (03CR) 10Hashar: [C: 032] Move logging coloring [integration/quibble] - 10https://gerrit.wikimedia.org/r/423784 (owner: 10Hashar) [06:57:39] (03Merged) 10jenkins-bot: Move logging coloring [integration/quibble] - 10https://gerrit.wikimedia.org/r/423784 (owner: 10Hashar) [07:03:48] (03PS2) 10Hashar: Also run 'npm test' for skins/extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/423785 [07:07:02] (03CR) 10Hashar: [C: 032] Also run 'npm test' for skins/extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/423785 (owner: 10Hashar) [07:07:28] (03Merged) 10jenkins-bot: Also run 'npm test' for skins/extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/423785 (owner: 10Hashar) [07:11:13] 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4103280 (10hashar) 05Open>03Resolved a:03thcipriani @Ottomata is now listed as a member and project admin. I am assuming Tyler did it yest... [07:11:34] !log deployment-prep: adding EddieGP as a member [07:11:37] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [07:14:10] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team: Request for shell access on deployment-prep - https://phabricator.wikimedia.org/T190925#4103291 (10hashar) 05Open>03Resolved a:03hashar I have added @EddieGP as a member of the `deployment-prep` project. Make sure to be careful when running scri... [07:34:05] (03PS3) 10Addshore: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [07:35:16] (03CR) 10jerkins-bot: [V: 04-1] WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [07:46:47] (03CR) 10Addshore: [C: 04-1] WikibaseLexeme: run node selenium tests as part of regular build (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [07:52:48] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4103407 (10hashar) I think @akosiaris told me once we should aim at not using `kubectl` but instead rely on the API. But maybe I got confused. Surely for prototyping that i... [07:59:44] 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4103428 (10akosiaris) >>! In T188933#4103407, @hashar wrote: > I think @akosiaris told me once we should aim at not using `kubectl` Yes. kubectl is a great and essential... [08:07:44] (03PS4) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 [08:08:57] (03CR) 10jerkins-bot: [V: 04-1] WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [08:10:06] (03CR) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [08:18:52] (03PS5) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 [08:27:44] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Upstream, 10Zuul: Zuul coverage pipeline is no more processing mwext-phpunit-coverage-patch jobs - https://phabricator.wikimedia.org/T189859#4103475 (10akosiaris) [08:27:48] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Packaging, 10Patch-For-Review: jenkins-debian-glue should run the lintian version from cowbuilder instead of from host - https://phabricator.wikimedia.org/T186494#4103476 (10akosiaris) [08:27:52] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Operations, 10Packaging, 10Zuul: Upload new zuul and jenkins-debian-glue packages to apt.wikimedia.org - https://phabricator.wikimedia.org/T186786#4103472 (10akosiaris) 05Open>03Resolved a:03akosiaris Done. [08:27:54] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Exception while launching job: TypeError: 'int' object has no attribute '__getitem__' - https://phabricator.wikimedia.org/T186381#4103477 (10akosiaris) [08:43:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Packaging, 10Upstream: jenkins-debian-glue should run the lintian version from cowbuilder instead of from host - https://phabricator.wikimedia.org/T186494#4103512 (10hashar) [08:43:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Packaging, 10Upstream: jenkins-debian-glue should run the lintian version from cowbuilder instead of from host - https://phabricator.wikimedia.org/T186494#3945400 (10hashar) 05Open>03stalled Pending upstream review/merge/re... [08:58:43] (03PS1) 10Hashar: Default ZUUL_PROJECT to mediawiki/core [integration/quibble] - 10https://gerrit.wikimedia.org/r/423856 [08:59:36] (03PS1) 10QChris: Allow “Gerrit Managers” to import history [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423857 [08:59:39] (03CR) 10QChris: [V: 031 C: 032] Allow “Gerrit Managers” to import history [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423857 (owner: 10QChris) [08:59:42] (03PS1) 10QChris: Import done. Revoke import grants [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423858 [08:59:45] (03CR) 10QChris: [V: 031 C: 032] Import done. Revoke import grants [oojs/router] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423858 (owner: 10QChris) [08:59:57] (03PS1) 10Hashar: Helpers to check core/vendor vs ext/skin [integration/quibble] - 10https://gerrit.wikimedia.org/r/423859 [09:00:28] (03PS1) 10Hashar: Reindent a code block [integration/quibble] - 10https://gerrit.wikimedia.org/r/423860 [09:06:32] (03PS2) 10Hashar: Helpers to check core/vendor vs ext/skin [integration/quibble] - 10https://gerrit.wikimedia.org/r/423859 [09:06:34] (03PS2) 10Hashar: Reindent a code block [integration/quibble] - 10https://gerrit.wikimedia.org/r/423860 [09:06:40] (03CR) 10Hashar: [C: 032] Default ZUUL_PROJECT to mediawiki/core [integration/quibble] - 10https://gerrit.wikimedia.org/r/423856 (owner: 10Hashar) [09:07:07] (03Merged) 10jenkins-bot: Default ZUUL_PROJECT to mediawiki/core [integration/quibble] - 10https://gerrit.wikimedia.org/r/423856 (owner: 10Hashar) [09:12:02] (03PS3) 10Hashar: Helpers to check core/vendor vs ext/skin [integration/quibble] - 10https://gerrit.wikimedia.org/r/423859 [09:12:03] (03PS3) 10Hashar: Reindent a code block [integration/quibble] - 10https://gerrit.wikimedia.org/r/423860 [09:12:12] (03CR) 10Hashar: [C: 032] Helpers to check core/vendor vs ext/skin [integration/quibble] - 10https://gerrit.wikimedia.org/r/423859 (owner: 10Hashar) [09:12:16] (03CR) 10Hashar: [C: 032] Reindent a code block [integration/quibble] - 10https://gerrit.wikimedia.org/r/423860 (owner: 10Hashar) [09:12:35] (03Merged) 10jenkins-bot: Helpers to check core/vendor vs ext/skin [integration/quibble] - 10https://gerrit.wikimedia.org/r/423859 (owner: 10Hashar) [09:12:39] (03Merged) 10jenkins-bot: Reindent a code block [integration/quibble] - 10https://gerrit.wikimedia.org/r/423860 (owner: 10Hashar) [09:24:50] (03PS1) 10Hashar: Run extension/skin PHPUnit testsuites [integration/quibble] - 10https://gerrit.wikimedia.org/r/423864 [09:30:31] (03PS2) 10Hashar: Run extension/skin PHPUnit testsuites [integration/quibble] - 10https://gerrit.wikimedia.org/r/423864 [09:35:21] (03PS3) 10Hashar: Run extension/skin PHPUnit testsuites [integration/quibble] - 10https://gerrit.wikimedia.org/r/423864 [09:38:44] (03PS1) 10Hashar: Only run mediawiki/core composer test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423865 [09:40:57] 10Phabricator, 10MediaWiki-extensions-Translate, 10translatewiki.net, 10I18n: Improvements for automatic reporting of tasks from translatewiki to Phabricator - https://phabricator.wikimedia.org/T188379#4103613 (10Aklapper) Where to find that link to Phab's task creation form in some code repository file wh... [09:41:09] (03PS1) 10Hashar: Only run mediawiki/core npm test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423868 [09:42:12] (03CR) 10Hashar: [C: 032] Run extension/skin PHPUnit testsuites [integration/quibble] - 10https://gerrit.wikimedia.org/r/423864 (owner: 10Hashar) [09:42:22] (03CR) 10Hashar: [C: 032] Only run mediawiki/core composer test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423865 (owner: 10Hashar) [09:42:26] (03CR) 10Hashar: [C: 032] Only run mediawiki/core npm test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423868 (owner: 10Hashar) [09:42:36] (03Merged) 10jenkins-bot: Run extension/skin PHPUnit testsuites [integration/quibble] - 10https://gerrit.wikimedia.org/r/423864 (owner: 10Hashar) [09:42:48] (03Merged) 10jenkins-bot: Only run mediawiki/core composer test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423865 (owner: 10Hashar) [09:42:51] (03Merged) 10jenkins-bot: Only run mediawiki/core npm test for itself [integration/quibble] - 10https://gerrit.wikimedia.org/r/423868 (owner: 10Hashar) [09:46:51] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Kanban), 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4103641 (10MarcoAurelio) 05Resolved>03Open Puppet issue still not resolved. [09:47:52] 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4103645 (10MarcoAurelio) a:05thcipriani>03None [09:48:09] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103647 (10Aklapper) It's a huge PITA to maintain correct developer docs when an org has no canonical places. "Code review might happen on Gerrit but for some projects on... [09:49:04] 10Project-Admins, 10Developer-Relations: Create 4 event projects - https://phabricator.wikimedia.org/T191372#4103650 (10Aklapper) Where to find information about the "Wikimedia Technical Conference 2018"? Links welcome. [09:49:32] 10Project-Admins, 10Developer-Relations (Apr-Jun-2018): Create 4 event projects - https://phabricator.wikimedia.org/T191372#4103653 (10Aklapper) [09:51:23] 10Beta-Cluster-Infrastructure: deployment-prep access request - https://phabricator.wikimedia.org/T191296#4103656 (10fgiunchedi) Confirmed, thanks @thcipriani ! [09:54:22] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4103664 (10Elitre) @Aklapper thanks again. Question: is there a particular reason why you went with the "group" format rather than the "tag" one as requested in the title? Just asking as I envisioned Tech A... [09:55:22] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103665 (10MarcoAurelio) >>! In T191182#4103647, @Aklapper wrote: > "Code review might happen on Gerrit but for some projects on GitHub but for some projects on Different... [09:57:47] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103668 (10mmodell) >>! In T191182#4103665, @MarcoAurelio wrote: >(nb: tools != Toolforge; I don't care where tool people host their code). Why are toolforge tools diffe... [10:01:26] hashar: Thanks! :) [10:01:31] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4103677 (10Aklapper) No strong reasons - the name sounded like a group of people and many folks say "tag" nowadays when they mean "project"... [10:12:06] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4103786 (10MarcoAurelio) Toolforge tools are accesories to MediaWiki and coding from non-maintainers is less frequent than mediawiki and its extensions. I also feel Toolf... [10:21:48] 10Project-Admins, 10Developer-Relations (Apr-Jun-2018): Create 4 event projects - https://phabricator.wikimedia.org/T191372#4103048 (10Peachey88) >>! In T191372#4103650, @Aklapper wrote: > Where to find information about the "Wikimedia Technical Conference 2018"? Links welcome. [MediaWiki-l] Announcing the Wi... [10:25:55] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4103887 (10Elitre) If changing would mean much extra work, a new link etc., then it's probably not worth it, at least now? The way people use projects seems to imply involving groups of users anyway. [10:31:06] 10Project-Admins: Replace tracking bug T2209 by new project tag "HTML" - https://phabricator.wikimedia.org/T102492#1366015 (10MarcoAurelio) HTML alone would be confusing, yes. Maybe `HTML-features`? [10:49:12] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4103951 (10Aklapper) Anyone can edit the project and its color and icon :) [10:54:07] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:51] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [10:58:45] 10Project-Admins, 10Developer-Relations (Apr-Jun-2018): Create 4 event projects - https://phabricator.wikimedia.org/T191372#4103978 (10Aklapper) 05Open>03Resolved Thanks! * https://phabricator.wikimedia.org/project/view/3319/ - #Wikimania-Hackathon-2018 * https://phabricator.wikimedia.org/project/view/3320... [10:59:12] 10Project-Admins, 10Developer-Relations (Apr-Jun-2018): Create 4 event projects for Wikimania Hackathon 2018 and 2018 Wikimedia Technical Conference - https://phabricator.wikimedia.org/T191372#4103980 (10Aklapper) [11:06:30] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:08:28] 10Beta-Cluster-Infrastructure, 10Multimedia: Reimage deployment-tmh01 with Debian Jessie - https://phabricator.wikimedia.org/T174477#3563286 (10EddieGP) According to shinken: > deployment-tmh01 is DOWN since 1M 3w 1d 12m 18s Given no one cared about it being down for 7 weeks and the comments above, I guess t... [11:12:22] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: deployment-prep down hosts - fix/remove? - https://phabricator.wikimedia.org/T191293#4104010 (10EddieGP) @MoritzMuehlenhoff: As I saw you commenting about deployment-videoscaler01 on T174477#3737836, do you know (or know who knows) whether... [11:21:08] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: deployment-prep down hosts - fix/remove? - https://phabricator.wikimedia.org/T191293#4104033 (10MoritzMuehlenhoff) Yeah, I think both deployment-tmh01 and deployment-videoscaler01 can be deleted, they are not functional in deployment-prep a... [11:53:30] (03PS6) 10Hashar: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026 [11:53:32] (03PS1) 10Hashar: docker: quibble 0.0.4 [integration/config] - 10https://gerrit.wikimedia.org/r/423891 [11:55:37] (03CR) 10Hashar: [C: 032] docker: quibble 0.0.4 [integration/config] - 10https://gerrit.wikimedia.org/r/423891 (owner: 10Hashar) [11:56:51] (03Merged) 10jenkins-bot: docker: quibble 0.0.4 [integration/config] - 10https://gerrit.wikimedia.org/r/423891 (owner: 10Hashar) [11:58:21] !log Building releng/quibble:0.0.4 [11:58:23] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [11:58:28] !log Building releng/quibble-stretch:0.0.4 [11:58:30] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:13:09] (03CR) 10Hashar: [C: 032] Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026 (owner: 10Hashar) [12:14:25] (03Merged) 10jenkins-bot: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026 (owner: 10Hashar) [12:14:36] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [12:17:10] !log added experimental quibble job to mediawiki core / vendor / skins/Vector [12:17:12] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [12:45:55] 10Phabricator, 10Discourse, 10Developer-Relations (Apr-Jun-2018): Enable Wikimedia Phabricator login in discourse-mediawiki.wmflabs.org - https://phabricator.wikimedia.org/T184987#4104206 (10Qgil) [12:59:19] PROBLEM - Free space - all mounts on integration-slave-docker-1005 is CRITICAL: CRITICAL: integration.integration-slave-docker-1005.diskspace.root.byte_percentfree (<22.22%) [13:09:07] PROBLEM - SSH on integration-slave-docker-1012 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:10:39] (03PS1) 10Hashar: Do not set DISPLAY for quibble jobs [integration/config] - 10https://gerrit.wikimedia.org/r/423907 [13:13:58] RECOVERY - SSH on integration-slave-docker-1012 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [13:29:18] PROBLEM - Free space - all mounts on integration-slave-docker-1005 is CRITICAL: CRITICAL: integration.integration-slave-docker-1005.diskspace.root.byte_percentfree (<11.11%) [13:29:38] PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:30:08] PROBLEM - SSH on integration-slave-docker-1012 is CRITICAL: CRITICAL - Socket timeout after 10 seconds [13:31:30] 10Beta-Cluster-Infrastructure, 10Multimedia: Reimage deployment-tmh01 with Debian Jessie - https://phabricator.wikimedia.org/T174477#4104437 (10TheDJ) ping @brion [13:34:29] RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [13:34:59] RECOVERY - SSH on integration-slave-docker-1012 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0) [13:41:42] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1342 bytes in 0.004 second response time [13:41:42] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1976 bytes in 0.035 second response time [13:46:45] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36493 bytes in 4.125 second response time [13:46:45] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 47531 bytes in 3.802 second response time [14:00:49] 10Release-Engineering-Team (Kanban), 10Advanced-Search, 10TCB-Team, 10Patch-For-Review, 10User-zeljkofilipin: Cannot find module nodemw - https://phabricator.wikimedia.org/T190307#4104619 (10Lea_WMDE) [14:18:59] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Beta: Could not find class role::ores::worker - https://phabricator.wikimedia.org/T188316#4104688 (10Ladsgroup) [14:19:02] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: ores-beta grafana is broken - https://phabricator.wikimedia.org/T190075#4104690 (10Ladsgroup) [14:19:27] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Beta: Could not find class role::ores::worker - https://phabricator.wikimedia.org/T188316#4003417 (10Ladsgroup) Oops :/ [14:19:39] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: ores-beta grafana is broken - https://phabricator.wikimedia.org/T190075#4061906 (10Ladsgroup) [14:25:23] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) [14:26:03] PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1343 bytes in 0.005 second response time [14:36:09] RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 47453 bytes in 5.397 second response time [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [15:07:17] 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4104947 (10Ottomata) a:03Ottomata [15:08:45] 10Beta-Cluster-Infrastructure, 10ORES, 10Scoring-platform-team (Current), 10User-Ladsgroup: Beta: Could not find class role::ores::worker - https://phabricator.wikimedia.org/T188316#4104950 (10Ladsgroup) 05Open>03Resolved It has been two months since this report and we have reimaged this instance and t... [15:26:00] (03PS6) 10Addshore: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [15:26:04] (03CR) 10Addshore: [C: 032] WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [15:27:28] (03Merged) 10jenkins-bot: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [15:30:24] !log reload zuul for https://gerrit.wikimedia.org/r/423652 [15:30:26] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [15:35:44] (03CR) 10Thiemo Kreuz (WMDE): Prohibit PHP's vanilla execution (034 comments) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423030 (owner: 10MaxSem) [15:58:22] (03PS1) 10Thiemo Kreuz (WMDE): Allow @dataProvider annotations in traits [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423953 [16:00:06] (03PS1) 10Thiemo Kreuz (WMDE): Remove unused methods and function arguments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423955 [16:03:34] (03CR) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek) [16:03:47] (03PS1) 10Thiemo Kreuz (WMDE): Add array type hints for improved type safety [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423956 [16:04:19] (03PS1) 10Thiemo Kreuz (WMDE): Fix typo and capitalize PHPDoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423957 [16:05:53] (03PS1) 10WMDE-leszek: Run mwext-node-selenium-composer-jessie job in extension-node-selenium-composer [integration/config] - 10https://gerrit.wikimedia.org/r/423958 [16:06:52] (03CR) 10Addshore: [C: 032] Run mwext-node-selenium-composer-jessie job in extension-node-selenium-composer [integration/config] - 10https://gerrit.wikimedia.org/r/423958 (owner: 10WMDE-leszek) [16:08:06] (03Merged) 10jenkins-bot: Run mwext-node-selenium-composer-jessie job in extension-node-selenium-composer [integration/config] - 10https://gerrit.wikimedia.org/r/423958 (owner: 10WMDE-leszek) [16:09:25] !log reloaded zuul for https://gerrit.wikimedia.org/r/423958 [16:09:28] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [16:13:34] (03PS1) 10Hashar: Add quibble job using composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/423961 [16:15:48] 10Beta-Cluster-Infrastructure, 10Operations, 10Puppet: Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) - https://phabricator.wikimedia.org/T187736#4105312 (10EddieGP) @Krenair created that instance according to openstack browser. Can you tell whether this instance is still neede... [16:18:41] (03PS1) 10Thiemo Kreuz (WMDE): Don't report forbidden tags as "should be used inside test classes" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423962 [16:21:15] (03CR) 10Thiemo Kreuz (WMDE): Allow @dataProvider annotations in traits (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423953 (owner: 10Thiemo Kreuz (WMDE)) [16:23:39] (03CR) 10Hashar: [C: 032] Add quibble job using composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/423961 (owner: 10Hashar) [16:23:54] (03CR) 10Hashar: [C: 032] "Passed on master branch https://integration.wikimedia.org/ci/job/mediawiki-core-quibble-composer-mysql-php7-docker/1/console" [integration/config] - 10https://gerrit.wikimedia.org/r/423961 (owner: 10Hashar) [16:25:12] (03Merged) 10jenkins-bot: Add quibble job using composer for dependencies [integration/config] - 10https://gerrit.wikimedia.org/r/423961 (owner: 10Hashar) [16:27:55] 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4105372 (10HappyDog) The way I see it, there areas of interest (which overlap slightly): 1. You are interested in contributing to the MediaWiki software. This includes... [16:35:26] (03PS1) 10Hashar: Considalite composer require in a single call [integration/quibble] - 10https://gerrit.wikimedia.org/r/423966 [16:36:11] (03CR) 10Hashar: [C: 032] Considalite composer require in a single call [integration/quibble] - 10https://gerrit.wikimedia.org/r/423966 (owner: 10Hashar) [16:36:37] (03Merged) 10jenkins-bot: Considalite composer require in a single call [integration/quibble] - 10https://gerrit.wikimedia.org/r/423966 (owner: 10Hashar) [16:54:44] wth zuul [17:08:23] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: Long-lived cherry-picks on deployment-puppetmaster02.deployment-prep.eqiad.wmflabs - https://phabricator.wikimedia.org/T191294#4105607 (10EddieGP) [17:08:57] (03PS1) 10Thiemo Kreuz (WMDE): Don't require documenting self-explaining parameter-less functions [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423982 [17:20:21] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team (Watching / External), 10Performance-Team, 10Availability (MediaWiki-MultiDC): Performance Q2 2017/18 goal: Install and use mcrouter in deployment-prep - https://phabricator.wikimedia.org/T151466#2818204 (10EddieGP) This, specifically https://gerrit... [17:43:24] (03CR) 10Umherirrender: [C: 032] Remove unused methods and function arguments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423955 (owner: 10Thiemo Kreuz (WMDE)) [17:44:20] (03Merged) 10jenkins-bot: Remove unused methods and function arguments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423955 (owner: 10Thiemo Kreuz (WMDE)) [17:44:47] (03CR) 10jenkins-bot: Remove unused methods and function arguments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423955 (owner: 10Thiemo Kreuz (WMDE)) [17:46:18] (03PS2) 10Umherirrender: Fix typo and capitalize PHPDoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423957 (owner: 10Thiemo Kreuz (WMDE)) [17:46:25] (03CR) 10Umherirrender: [C: 032] Fix typo and capitalize PHPDoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423957 (owner: 10Thiemo Kreuz (WMDE)) [17:47:08] (03Merged) 10jenkins-bot: Fix typo and capitalize PHPDoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423957 (owner: 10Thiemo Kreuz (WMDE)) [17:47:49] (03CR) 10jenkins-bot: Fix typo and capitalize PHPDoc [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423957 (owner: 10Thiemo Kreuz (WMDE)) [17:48:30] (03PS2) 10Umherirrender: Add array type hints for improved type safety [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423956 (owner: 10Thiemo Kreuz (WMDE)) [17:48:35] (03CR) 10Umherirrender: [C: 032] Add array type hints for improved type safety [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423956 (owner: 10Thiemo Kreuz (WMDE)) [17:49:19] (03Merged) 10jenkins-bot: Add array type hints for improved type safety [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423956 (owner: 10Thiemo Kreuz (WMDE)) [17:49:44] (03CR) 10jenkins-bot: Add array type hints for improved type safety [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423956 (owner: 10Thiemo Kreuz (WMDE)) [17:50:18] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101697 (10mobrovac) I think we should seize this opportunity to incorporate the idea of getting rid of deploy repos altogether,... [17:51:57] (03CR) 10Umherirrender: [C: 031] Don't report forbidden tags as "should be used inside test classes" [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423962 (owner: 10Thiemo Kreuz (WMDE)) [17:54:14] 10Project-Admins: Please create a Tech Ambassadors Tag project - https://phabricator.wikimedia.org/T190300#4105870 (10Elitre) Great, then I'll consider that. TYVM! [17:55:59] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.31-release: Upgrade patches for tarball releases don't apply cleanly to tarball installation - https://phabricator.wikimedia.org/T73379#4105881 (10Krinkle) [17:57:01] 10MediaWiki-Releasing, 10Release-Engineering-Team, 10MW-1.31-release: Upgrade patches for tarball releases don't apply cleanly to tarball installation - https://phabricator.wikimedia.org/T73379#748261 (10Krinkle) Not sure if still this is still an issue with current processes. In case it's still an issue, ta... [18:08:36] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: Request for one additional RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10dduvall) [18:26:02] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: Request for one additional RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105973 (10RobH) [18:26:42] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: Request for one additional RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10RobH) [18:27:07] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10RobH) [18:30:43] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105919 (10demon) I authorize this as @greg's delegate while he's on vacation. [18:31:00] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team, 10Operations, 10Ops-Access-Requests: grant thcipriani RelEng root on contint1001 - https://phabricator.wikimedia.org/T191453#4105983 (10RobH) I've updated the task description a bit, and included the checklist we require. I'll also note... [18:34:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.005 second response time [18:35:58] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:43:52] (03CR) 10Umherirrender: Allow @dataProvider annotations in traits (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423953 (owner: 10Thiemo Kreuz (WMDE)) [18:52:37] 10Continuous-Integration-Config, 10Security-Team, 10phan-taint-check-plugin, 10MW-1.31-release, and 2 others: Make jenkins run phan-taint-check-plugin non-voting and then voting - https://phabricator.wikimedia.org/T182599#4106037 (10CCicalese_WMF) [18:54:27] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:58:53] (03CR) 10Thiemo Kreuz (WMDE): Allow @dataProvider annotations in traits (031 comment) [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423953 (owner: 10Thiemo Kreuz (WMDE)) [19:00:55] 10Release-Engineering-Team (Watching / External), 10Epic, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10User-notice: Deploy refactored comment storage - https://phabricator.wikimedia.org/T166733#4106096 (10CCicalese_WMF) [19:03:54] 10Continuous-Integration-Infrastructure, 10MediaWiki-Platform-Team (MWPT-Q4-Apr-Jun-2018), 10Test-Coverage: Migrate https://tools.wmflabs.org/coverage/mediawiki/ to CI infrastructure - https://phabricator.wikimedia.org/T182751#4106152 (10CCicalese_WMF) [19:13:08] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#4106216 (10mmodell) [19:21:34] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubber should error on unknown/obsolete config fields - https://phabricator.wikimedia.org/T191460#4106273 (10dduvall) [19:22:54] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#4106308 (10mmodell) [19:24:50] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Git commit is missing from compiled Blubber version when packaged for Debian - https://phabricator.wikimedia.org/T191462#4106323 (10dduvall) [19:25:07] I'm getting some new failures from Selenium tests: 17:33:36 /tmp/jenkins4121753139233067129.sh: line 13: ./node_modules/.bin/grunt: No such file or directory [19:25:13] https://integration.wikimedia.org/ci/job/mwext-node-selenium-composer-jessie/5/console [19:25:20] anything changed? [19:26:30] greg-g, twentyafterfour thcipriani ^^ [19:35:51] SMalyshev: I'm not sure [19:37:15] this error doesn't seem like bad test, seems like something in the build is not working... [19:37:25] I did recheck and it still happens [19:48:11] SMalyshev: i dont think that job ever worked :D [19:48:20] all 5 builds are failures [19:48:24] and it lists ./node_modules/.bin/grunt: No such file or directory [19:49:17] SMalyshev: addshore would know but he is probably not around at this time [19:49:23] o/ [19:49:43] *reads up* [19:49:57] https://integration.wikimedia.org/ci/job/mwext-node-selenium-composer-jessie/ fails constantly :] apparently got added today [19:50:00] aaah yes, on WikibaseLexeme [19:50:09] for mediawiki/extensions/WikibaseLexeme [19:50:16] I guess it depends on a change in the source repo [19:50:16] we havn't been able to make it work, I can set it as non voting for now! [19:50:28] +1 [19:50:59] seems it is missing an "npm install" step [19:51:31] that is needed for the macro which ends up invoking grunt webdriver:test [19:51:47] oooh [19:52:35] and it could work with the vendor.git repository [19:53:01] one has to use the composer merge plugin (by adding the extension to composer.local.json), then "composer require" the devDependencies [19:53:17] * addshore cant even remember how to make something non voting [19:53:23] (since it seems that job is a variant of a vendor based job) [19:53:28] - job: foobar [19:53:31] voting: false [19:53:54] or rename it with a non-voting suffix and there is a wildcard job filter in zull that will make it non voting [19:54:01] (03PS1) 10Addshore: mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 [19:54:10] hashar: i'll let you CR :) [19:54:20] NAAAA [19:54:23] not in jenkins :] [19:54:30] in zuul/layout.yaml hehe [19:54:31] bah [19:54:58] - name: ^.*-non-voting$ [19:54:58] voting: false [19:54:59] - name: ^.*-non-voting$ [19:54:59] voting: false [19:55:03] that is the wildcard pattern [19:55:09] (03PS2) 10Addshore: mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 [19:55:38] * hashar watches https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/15117/console [19:56:27] (03CR) 10jerkins-bot: [V: 04-1] mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [19:56:33] :( [19:56:43] (03CR) 10Hashar: mwext-node-selenium-composer-jessie voting false (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [19:57:19] you make a good point [19:57:33] (03CR) 10Hashar: "spoiler" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [19:57:43] addshore: there is the solution ^^ [19:58:17] (03PS3) 10Addshore: mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 [19:58:31] :D [19:58:37] 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4106435 (10demon) (1) works for me too, actually. [19:58:42] and the zuul diff job should spurt a nice diff for it [19:58:59] https://integration.wikimedia.org/ci/job/integration-zuul-layoutdiff/15118/console [19:59:03] 00:00:20.560 -INFO:zuul.DependentPipelineManager: [19:59:03] 00:00:20.561 +INFO:zuul.DependentPipelineManager: [nonvoting] [19:59:11] (03CR) 10Hashar: [C: 032] "Hurrah!" [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [19:59:28] (03CR) 10jerkins-bot: [V: 04-1] mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [19:59:32] .... [19:59:49] cant we just deploy directly to prod and fix it there? [20:00:09] non voting jobs are not allowed in gate-and-submit [20:01:01] (03CR) 10jerkins-bot: [V: 04-1] mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [20:01:32] (03PS4) 10Addshore: mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 [20:04:03] (03CR) 10Hashar: [C: 032] mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [20:04:12] addshore: neat. will let you deploy it ? [20:04:17] yarp [20:05:15] (03Merged) 10jenkins-bot: mwext-node-selenium-composer-jessie voting false [integration/config] - 10https://gerrit.wikimedia.org/r/424045 (owner: 10Addshore) [20:07:22] hashar: it is deploying some quibble stuff tooo [20:07:26] but only in experimental [20:07:53] !log reload zuul for https://gerrit.wikimedia.org/r/424045 (and some quibble experimental stuff) [20:07:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:14:23] addshore: yeah no worries [20:14:29] i forgot to deploy it before dinner i guess [20:14:32] !! [20:14:37] addshore: I am off. Kudos [20:14:41] o/ [20:17:26] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0] [20:19:41] !log deployed to BC: [mobileapps/deploy@0460519]: Update mobileapps to 2d5ab5b [20:19:43] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:25:10] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#4106548 (10mmodell) [20:25:49] 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#3868579 (10mmodell) [20:25:58] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found) [20:31:52] (03CR) 10Thcipriani: "Looks good. Couple of nitpicks and one `else` that I think should be an `elif`." (033 comments) [tools/release] - 10https://gerrit.wikimedia.org/r/423776 (owner: 1020after4) [20:31:57] PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<20.00%) [20:36:33] (03PS5) 1020after4: deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776 [20:37:39] (03CR) 10Thcipriani: [C: 032] deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776 (owner: 1020after4) [20:38:10] (03Merged) 10jenkins-bot: deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776 (owner: 1020after4) [20:39:04] thanks thcipriani [20:40:41] sure thing, looks useful :) [20:41:58] RECOVERY - Free space - all mounts on deployment-ores01 is OK: OK: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found) [20:42:59] ok, seems to be building ok now [20:43:04] :D [20:45:08] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4106658 (10awight) I discovered in beta cluster testing that we'll have to... [20:48:27] 10Release-Engineering-Team (Watching / External), 10Operations, 10Patch-For-Review, 10Scoring-platform-team (Current), 10Wikimedia-Incident: Cache ORES virtualenv within versioned source - https://phabricator.wikimedia.org/T181071#4106708 (10awight) [21:07:00] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Git commit is missing from compiled Blubber version when packaged for Debian - https://phabricator.wikimedia.org/T191462#4106792 (10dduvall) 05Open>03Resolved a:03dduvall Added to `debian/control`. [21:08:23] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:21:32] no_justification i found this over the weekend https://gwtmaterialdesign.github.io/gwt-material-demo/ heh :) [21:21:45] * paladox wishes there was a javascript one (polymer) [21:21:56] since the ui on there looks really good [21:29:40] Project mwext-phpunit-coverage-publish build #2956: 04FAILURE in 2 min 9 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2956/ [21:32:28] Yippee, build fixed! [21:32:28] Project mwext-phpunit-coverage-publish build #2957: 09FIXED in 53 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/2957/ [21:50:24] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<22.22%) [21:50:38] Project beta-scap-eqiad build #202402: 04FAILURE in 16 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/202402/ [21:51:09] 21:47:58 Job ['/usr/bin/scap', 'pull', '--no-update-l10n', 'deployment-mira.deployment-prep.eqiad.wmflabs', 'deployment-tin.deployment-prep.eqiad.wmflabs', 'deployment-tin.deployment-prep.eqiad.wmflabs'] called with an empty host list. [21:51:21] 21:48:34 pull failed: Command '['sudo', '-u', 'mwdeploy', '-n', '--', '/usr/bin/rsync', '--archive', '--delete-delay', '--delay-updates', '--compress', '--delete', '--exclude=**/cache/l10n/*.cdb', '--exclude=*.swp', '--no-perms', '--exclude=**/.git', 'deployment-tin.deployment-prep.eqiad.wmflabs::common', '/srv/mediawiki']' returned non-zero exit status 11 [21:52:12] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<55.56%) [21:58:46] Project beta-scap-eqiad build #202403: 04STILL FAILING in 7 min 22 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/202403/ [22:02:38] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubber should error on unknown/obsolete config fields - https://phabricator.wikimedia.org/T191460#4106938 (10dduvall) p:05Triage>03Normal a:03dduvall [22:02:42] PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 1976 bytes in 0.058 second response time [22:02:42] PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1342 bytes in 0.009 second response time [22:06:15] Project beta-scap-eqiad build #202404: 04STILL FAILING in 6 min 50 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/202404/ [22:07:46] RECOVERY - App Server Main HTTP Response on deployment-mediawiki04 is OK: HTTP OK: HTTP/1.1 200 OK - 47503 bytes in 3.857 second response time [22:07:47] RECOVERY - English Wikipedia Mobile Main page on beta-cluster is OK: HTTP OK: HTTP/1.1 200 OK - 36493 bytes in 6.513 second response time [22:12:01] Project beta-scap-eqiad build #202405: 04STILL FAILING in 5 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/202405/ [22:17:05] 10Beta-Cluster-Infrastructure, 10Operations, 10HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#4106961 (10EddieGP) [22:17:07] 10Beta-Cluster-Infrastructure, 10Multimedia: Reimage deployment-tmh01 with Debian Jessie - https://phabricator.wikimedia.org/T174477#4106957 (10EddieGP) 05Open>03declined Per @brions comment here and @MoritzMuehlenhoff s comment T191293#4104033, deployment-tmh01 will be deleted instead of reimaged. [22:17:15] RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK [22:17:45] !log deployment-mediawiki0{4,5} clear apt-cache, restart clear hhvm cache, restart hhvm [22:17:47] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [22:18:59] 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: deployment-prep down hosts - fix/remove? - https://phabricator.wikimedia.org/T191293#4100460 (10EddieGP) @brion commented on T174477. We can go ahead a delete deployment-tmh01. Still need confirmation for deployment-videoscaler01. [22:20:37] Yippee, build fixed! [22:20:37] Project beta-scap-eqiad build #202406: 09FIXED in 6 min 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/202406/ [22:23:29] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-redis0[12] - https://phabricator.wikimedia.org/T191163#4106974 (10EddieGP) [22:23:34] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#4106977 (10EddieGP) [22:25:25] 10Beta-Cluster-Infrastructure, 10Puppet: Puppet broken on deployment-mx due to systemd on trusty - https://phabricator.wikimedia.org/T184244#4106983 (10EddieGP) [22:25:29] 10Beta-Cluster-Infrastructure, 10Operations, 10HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#4106982 (10EddieGP) [22:25:31] 10Beta-Cluster-Infrastructure, 10Operations, 10Patch-For-Review, 10Prometheus-metrics-monitoring, 10User-fgiunchedi: Move deployment-prep redis instances to stretch - https://phabricator.wikimedia.org/T179371#3722645 (10EddieGP) [22:29:58] 10Beta-Cluster-Infrastructure, 10Operations, 10HHVM: Move the MW Beta appservers to Debian - https://phabricator.wikimedia.org/T144006#2589035 (10EddieGP) There's 4 trusty instances left in deployment-prep: - deployment-tmh01 is to be deleted per T174477/T191293 - deployment-redis0[12] are to be replaced by... [22:30:25] RECOVERY - Free space - all mounts on deployment-mediawiki05 is OK: OK: All targets OK [22:35:43] 10Release-Engineering-Team (Kanban), 10Release Pipeline (Blubber): Blubber should error on unknown/obsolete config fields - https://phabricator.wikimedia.org/T191460#4107016 (10dduvall) [23:21:24] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) [23:31:23] RECOVERY - Free space - all mounts on deployment-mediawiki05 is OK: OK: All targets OK