[00:12:58] no_justification: heh [00:14:18] The logic here is all wrong. [00:14:40] We can *tell* if a repo is local [00:14:43] Why not link those? [00:14:44] :) [00:14:47] * no_justification has a half-patch [00:15:35] Yeh [00:15:54] + code.* is no longer in use [00:24:06] no_justification: oh your creating a patch for it? [00:31:17] PROBLEM - Long lived cherry-picks on puppetmaster on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [00:37:34] no_justification: I really want upstream to make a polygerrit type ui for gitiles :) [00:37:43] With a search bar :) [00:38:58] Yeah, it's a one line patch [00:39:00] I think it'll work [00:40:27] :) [00:42:43] Eh, I had it wrong. Still, shouldn't be hard. [00:43:20] Oh [00:45:54] There's two bugs here, actually [00:45:57] One will fix upstream [00:46:29] :) [00:57:08] no_justification: there’s a new read only plugin for gerrit [00:59:55] * paladox goes as it’s 1am have to get up at 7:30 [01:04:35] PROBLEM - puppet last run on contint2001 is CRITICAL: CRITICAL: Catalog fetch fail. Either compilation failed or puppetmaster has issues [01:04:35] Bleh [01:04:38] Stupid gitiles [01:04:42] I keep getting confused. [01:28:26] paladox: https://gerrit-review.googlesource.com/#/c/gitiles/+/163252 [01:28:30] I'm calling it a night, later [01:29:41] RECOVERY - puppet last run on contint2001 is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures [02:11:13] !log manually queued jenkins jobs [02:11:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [02:27:39] 10Beta-Cluster-Infrastructure: nlwikipedia on Beta Cluster is wrongly shown under "Other Wikimedia Projects" - https://phabricator.wikimedia.org/T188582#4012705 (10Liuxinyu970226) [02:27:49] PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%) [04:28:04] legoktm: Hm.. looks like Zuul/Jenkins isn't doing anything. Lots of queued on https://integration.wikimedia.org/zuul/ but nothig currently blue/running [04:28:28] I see blue things... [04:28:55] Top one is stuck, that happens sometimes, once the request of the queue clears it'll get processed too [04:30:37] Ah yeah, it's going now [04:30:46] It was definitely all grey/green for a while [04:30:59] maybe out of slaves? [04:31:00] Looks like it exhausted all the nodepool slots before np could start new ones in time [04:31:04] er, nodepool slots* [04:31:04] yeah [04:31:16] I manually queued ~100+ jobs [04:31:20] Yeah, the job is completing slightly faster than it takes to spawn one [05:59:06] 10Beta-Cluster-Infrastructure: nlwikipedia on Beta Cluster is wrongly shown under "Other Wikimedia Projects" - https://phabricator.wikimedia.org/T188582#4012705 (10Jayprakash12345) langlist-labs does not have nl lang code. See:- https://phabricator.wikimedia.org/source/mediawiki-config/browse/master/langlist-labs [06:02:06] PROBLEM - Puppet errors on integration-publishing is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:02:36] PROBLEM - Puppet errors on jenkinstest is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:03:17] PROBLEM - Puppet errors on integration-slave-docker-1007 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:03:27] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:03:31] PROBLEM - Puppet errors on deployment-aqs03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:03:48] PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:03:52] PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:04:15] PROBLEM - Puppet errors on deployment-memc07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:04:19] PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:04:21] PROBLEM - Puppet errors on deployment-redis05 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:04:50] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:06:03] PROBLEM - Puppet errors on deployment-cumin is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:07:22] PROBLEM - Puppet errors on deployment-kafka05 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:07:22] PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:07:45] PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:08:35] PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:08:35] PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:08:51] PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:08:58] PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:09:01] PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:09:41] PROBLEM - Puppet errors on integration-slave-docker-1005 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:09:55] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:10:06] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:10:31] PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:10:43] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:11:16] PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:12:04] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:12:26] PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:12:51] PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:13:11] PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:13:23] PROBLEM - Puppet errors on deployment-kafka-jumbo-1 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:13:51] PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:13:53] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:14:34] PROBLEM - Puppet errors on deployment-cassandra3-02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:15:18] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:15:41] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:17:42] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [06:18:37] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:21:59] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [06:22:39] PROBLEM - Puppet errors on deployment-restbase02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:23:12] PROBLEM - Puppet errors on integration-slave-docker-1002 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:23:18] PROBLEM - Puppet errors on integration-slave-docker-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:24:31] hmm zuul looks stuck this time [06:24:35] PROBLEM - Puppet errors on deployment-sca04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:24:43] PROBLEM - Puppet errors on saucelabs-02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [06:27:17] PROBLEM - Puppet errors on integration-r-lang-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [06:27:50] a bunch of mysql related errors it looks like [06:27:54] I wonder what's wrong with puppet [06:28:11] PROBLEM - Puppet errors on deployment-pdfrender02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:28:14] PROBLEM - Puppet errors on deployment-snapshot01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:28:59] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:29:17] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [06:29:34] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [06:39:50] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [06:42:05] RECOVERY - Puppet errors on integration-publishing is OK: OK: Less than 1.00% above the threshold [0.0] [06:42:37] RECOVERY - Puppet errors on jenkinstest is OK: OK: Less than 1.00% above the threshold [0.0] [06:44:22] RECOVERY - Puppet errors on deployment-redis05 is OK: OK: Less than 1.00% above the threshold [0.0] [06:48:15] PROBLEM - Puppet staleness on deployment-eventlog02 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0] [06:53:38] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:58:14] RECOVERY - Puppet errors on integration-slave-docker-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [06:59:43] RECOVERY - Puppet errors on saucelabs-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:01:59] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [07:02:41] RECOVERY - Puppet errors on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:03:17] RECOVERY - Puppet errors on integration-slave-docker-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [07:04:18] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:04:34] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:04:36] RECOVERY - Puppet errors on deployment-sca04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:07:17] RECOVERY - Puppet errors on integration-r-lang-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:08:11] RECOVERY - Puppet errors on deployment-pdfrender02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:08:11] RECOVERY - Puppet errors on deployment-snapshot01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:08:59] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:15] RECOVERY - Puppet errors on integration-slave-docker-1007 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:25] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:33] RECOVERY - Puppet errors on deployment-eventlog05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:34] RECOVERY - Puppet errors on deployment-aqs03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:52] RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:52] RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:55] RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0] [07:13:56] RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:00] RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:15] RECOVERY - Puppet errors on deployment-memc07 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:19] RECOVERY - Puppet errors on deployment-redis06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:14:43] RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0] [07:15:42] RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:16:04] RECOVERY - Puppet errors on deployment-cumin is OK: OK: Less than 1.00% above the threshold [0.0] [07:16:14] RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:23] RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:23] RECOVERY - Puppet errors on deployment-kafka05 is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:45] RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [07:17:49] RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:12] RECOVERY - Puppet errors on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:36] RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:18:50] RECOVERY - Puppet errors on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [0.0] [07:19:45] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:19:56] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:06] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:31] RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:20:40] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:07] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:23] RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:22:40] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:20] RECOVERY - Puppet errors on deployment-kafka-jumbo-1 is OK: OK: Less than 1.00% above the threshold [0.0] [07:23:49] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:24:33] RECOVERY - Puppet errors on deployment-cassandra3-02 is OK: OK: Less than 1.00% above the threshold [0.0] [07:25:18] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:44:44] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [07:50:43] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [08:15:46] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [08:16:51] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint): Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#4013024 (10hashar) I am probably going to attempt it as part of {T187797} [08:17:09] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint): Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#4013027 (10hashar) [08:17:12] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#3986123 (10hashar) [09:07:21] no_justification: heh [09:27:09] (03CR) 10Hashar: [C: 032] "Since https://gerrit.wikimedia.org/r/#/c/414180/" [integration/config] - 10https://gerrit.wikimedia.org/r/414962 (owner: 10Legoktm) [09:28:05] (03CR) 10Hashar: [C: 032] "Deployed but I forgot to get it merged :/" [integration/config] - 10https://gerrit.wikimedia.org/r/415280 (owner: 10Hashar) [09:28:21] (03Merged) 10jenkins-bot: Run phan for Gadgets [integration/config] - 10https://gerrit.wikimedia.org/r/414962 (owner: 10Legoktm) [09:29:30] (03Merged) 10jenkins-bot: Tie maven docker jobs to 4GBytes RAM instances [integration/config] - 10https://gerrit.wikimedia.org/r/415280 (owner: 10Hashar) [09:33:53] 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin: Use a more complex password for WikiAdmin in selenium tests - https://phabricator.wikimedia.org/T188520#4013133 (10zeljkofilipin) [09:35:26] 10Release-Engineering-Team (Kanban), 10MinervaNeue, 10Readers-Web-Backlog, 10Vector, and 3 others: Vector browser test blocking merge in Minerva - https://phabricator.wikimedia.org/T188553#4013138 (10zeljkofilipin) [09:35:32] PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [10.0] [09:53:32] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Release Pipeline: On CI, upgrade docker-ce from 17.06.2 to 17.12.1 - https://phabricator.wikimedia.org/T177499#4013197 (10hashar) [10:10:29] RECOVERY - Mediawiki Error Rate on graphite-labs is OK: OK: Less than 1.00% above the threshold [1.0] [10:29:05] 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10Patch-For-Review, 10User-zeljkofilipin, 10Wikimedia-Incident: Create selenium-core-jessie daily Jenkins job - https://phabricator.wikimedia.org/T185011#4013253 (10zeljkofilipin) http://webdriver.io/guide/testrunner/organizesuite.html#Run-Sele... [10:42:39] 10Release-Engineering-Team (Kanban), 10Vector, 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#4013284 (10zeljkofilipin) This could help: ``` useskin=Vector useskin=Minerva ``` [10:54:08] PROBLEM - Host deployment-videoscaler01 is DOWN: CRITICAL - Host Unreachable (10.68.19.130) [10:54:52] PROBLEM - Host deployment-tmh01 is DOWN: CRITICAL - Host Unreachable (10.68.16.211) [11:23:01] (03PS2) 10Hashar: Migrate oojs/ui coverage and demos job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/415295 [11:24:12] (03CR) 10Hashar: [C: 032] "Thanks! Lets head to Chrome 64 :]" [integration/config] - 10https://gerrit.wikimedia.org/r/415270 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [11:24:47] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4013335 (10hashar) [11:25:34] (03Merged) 10jenkins-bot: Migrate oojs npm job to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/415270 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [11:51:40] (03PS3) 10Hashar: Merge oojs/ui publish job and move them to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/415295 (https://phabricator.wikimedia.org/T187797) [11:52:40] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4013442 (10hashar) [12:22:47] (03PS4) 10Hashar: Merge oojs/ui publish job and move them to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/415295 (https://phabricator.wikimedia.org/T187797) [12:23:13] (03CR) 10Hashar: [C: 032] "Manually tested and that worked as expected :]" [integration/config] - 10https://gerrit.wikimedia.org/r/415295 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [12:25:00] (03Merged) 10jenkins-bot: Merge oojs/ui publish job and move them to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/415295 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [12:25:17] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4013591 (10hashar) [12:28:43] (03PS1) 10Hashar: Promote integration-jjb-config-diff-docker [integration/config] - 10https://gerrit.wikimedia.org/r/415554 (https://phabricator.wikimedia.org/T187797) [12:29:58] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q4, 10Patch-For-Review: Migrate leftover Nodepool jobs to Docker - https://phabricator.wikimedia.org/T187797#4013615 (10hashar) [12:30:04] (03PS15) 10Zoranzoki21: Add few extensions in zuul/layout.yaml to Jenkins can run builds and remove mediawiki/extensions/Collection/OfflineContentGenerator/node_modules [integration/config] - 10https://gerrit.wikimedia.org/r/406524 (https://phabricator.wikimedia.org/T183674) [12:30:53] PROBLEM - Puppet errors on deployment-ores01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [12:31:33] (03Draft2) 10Jayprakash12345: Whitelist Tulsi Bhagat in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/415553 [12:32:01] (03CR) 10Hashar: [C: 032] Promote integration-jjb-config-diff-docker [integration/config] - 10https://gerrit.wikimedia.org/r/415554 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [12:33:03] (03CR) 10Zoranzoki21: "I support to this user be whitelisted." (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/415553 (owner: 10Jayprakash12345) [12:34:27] (03PS3) 10Jayprakash12345: Whitelist Tulsi Bhagat in CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/415553 [12:35:01] (03CR) 10Zoranzoki21: [C: 031] "Ok, as I told. I support to Tulsi be whitelisted." [integration/config] - 10https://gerrit.wikimedia.org/r/415553 (owner: 10Jayprakash12345) [12:36:46] (03Merged) 10jenkins-bot: Promote integration-jjb-config-diff-docker [integration/config] - 10https://gerrit.wikimedia.org/r/415554 (https://phabricator.wikimedia.org/T187797) (owner: 10Hashar) [12:42:20] Hauskatze https://gerrit-review.googlesource.com/c/gerrit/+/129130 [12:42:26] hashar: is this a no-op? https://gerrit.wikimedia.org/r/#/c/415555/ [12:42:30] checking [12:43:27] work in progress, good [13:28:42] hmm someone managed to produce a null pointer on http://gerrit-test.wmflabs.org/gerrit/q/status:open [13:29:24] oh [13:29:25] [2018-03-01 13:28:28,224] [HTTP-80] ERROR com.google.gerrit.httpd.restapi.RestApiServlet : Error in GET /gerrit/changes/?O=81&S=0&n=25&q=status%3Aopen [13:29:25] com.google.gwtorm.server.OrmException: unable to check permissions [13:31:13] fixed it now [14:53:16] PROBLEM - Host deployment-puppetdb01 is DOWN: CRITICAL - Host Unreachable (10.68.23.76) [14:55:56] !log delete deployment-eventlog02 ubuntu instance in favor of the brand new deployment-eventlog05 (stretch) [14:56:01] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:57:25] PROBLEM - Host deployment-eventlog02 is DOWN: CRITICAL - Host Unreachable (10.68.18.138) [14:58:57] bye bye [15:58:41] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:33:17] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:33:32] (03PS1) 10Hashar: WIP Polling job for MediaWiki doxygen [integration/config] - 10https://gerrit.wikimedia.org/r/415588 (https://phabricator.wikimedia.org/T115755) [16:56:50] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Patch-For-Review: Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#4014476 (10hashar) It temporarily pushes to https:/... [17:05:38] (03PS2) 10Hashar: WIP Polling job for MediaWiki doxygen [integration/config] - 10https://gerrit.wikimedia.org/r/415588 (https://phabricator.wikimedia.org/T115755) [17:10:29] 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10Release Pipeline: On CI, upgrade docker-ce from 17.06.2 to 17.12.1 - https://phabricator.wikimedia.org/T177499#4014563 (10hashar) We talked about it during the pipeline meeting. #Blubber is going to need it as well ({D... [17:17:51] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014568 (10Sau226) [17:19:00] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014580 (10Aklapper) @Sau226: https://gerrit.wikimedia.org/r/#/settings/web-identities does not work for you? [17:20:02] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014583 (10Sau226) I can't tell it to use the one I wish to use or delete my old one (that I don't want to use). Is there an active ID toggle somewhere? [17:21:15] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014584 (10Sau226) 05Open>03Resolved a:03Sau226 Must be a momentary server glitch of some sort. Issue appears to be resolved [17:21:40] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014568 (10Paladox) I think what the user wants to do is use a different email. You can go to https://gerrit.wikimedia.org/r/#/settings/contact then click register an email and choose a preferred one. [17:23:35] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014593 (10Aklapper) 05Resolved>03declined a:05Sau226>03None Glad it works for you! Setting status to declined as no code was changed. [17:24:22] 10Gerrit: Gerrit email change - https://phabricator.wikimedia.org/T188639#4014597 (10Sau226) @Paladox As an afterthought I'll make clear that I had the other email registered and could just not switch to it. [17:44:10] 10Phabricator, 10MediaWiki-extensions-Translate, 10translatewiki.net, 10I18n: Improvements for automatic reporting of tasks from translatewiki to Phabricator - https://phabricator.wikimedia.org/T188379#4014651 (10Aklapper) [17:49:03] 10Phabricator, 10MediaWiki-extensions-Translate, 10translatewiki.net, 10I18n: Improvements for automatic reporting of tasks from translatewiki to Phabricator - https://phabricator.wikimedia.org/T188379#4014667 (10Nemo_bis) > The i18n tag didn't get the attention it deserves for a long time And this despit... [17:51:17] 10Phabricator, 10MediaWiki-extensions-Translate, 10translatewiki.net, 10I18n: Improvements for automatic reporting of tasks from translatewiki to Phabricator - https://phabricator.wikimedia.org/T188379#4005297 (10greg) Because it would prevent these issues from going into a bucket of unrelated and unsorted... [17:58:36] (03PS3) 10Hashar: WIP Polling job for MediaWiki doxygen [integration/config] - 10https://gerrit.wikimedia.org/r/415588 (https://phabricator.wikimedia.org/T115755) [18:34:36] PROBLEM - App Server Main HTTP Response on deployment-mediawiki07 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 hphp_invoke - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 287 bytes in 0.010 second response time [18:54:27] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [19:05:32] 10Release-Engineering-Team (Kanban), 10Vector, 10Patch-For-Review, 10User-zeljkofilipin: Move one Selenium tests from mediawiki/core to mediawiki/skins/Vector - https://phabricator.wikimedia.org/T187859#4015225 (10Jdlrobson) > useskin=Vector > useskin=Minerva I don't follow... the Minerva skin should not... [19:14:34] PROBLEM - Free space - all mounts on integration-slave-docker-1001 is CRITICAL: CRITICAL: integration.integration-slave-docker-1001.diskspace.root.byte_percentfree (<20.00%) [19:29:35] RECOVERY - Free space - all mounts on integration-slave-docker-1001 is OK: OK: All targets OK [19:55:56] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Patch-For-Review: Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#4015404 (10hashar) Some got magically generated via... [19:56:10] 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure (Little Steps Sprint), 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Rewrite mediawiki-core-doxygen-publish Jenkins job to poll scm instead of being triggered by Zuul - https://phabricator.wikimedia.org/T115755#4015408 (10ha... [20:02:48] PROBLEM - Free space - all mounts on deployment-mediawiki05 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki05.diskspace.root.byte_percentfree (<11.11%) [20:07:52] RECOVERY - Free space - all mounts on deployment-mediawiki05 is OK: OK: All targets OK [20:09:42] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10User-MarcoAurelio: nlwikipedia on Beta Cluster is wrongly shown under "Other Wikimedia Projects" - https://phabricator.wikimedia.org/T188582#4015439 (10Jayprakash12345) 05Open>03Resolved a:03MarcoAurelio https://deployment.wikimedia.beta.wmflabs.org/... [20:28:34] Project selenium-Wikibase-chrome » chrome,beta,Linux,DebianJessie && contintLabsSlave build #128: 04FAILURE in 41 min: https://integration.wikimedia.org/ci/job/selenium-Wikibase-chrome/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=DebianJessie%20&&%20contintLabsSlave/128/ [20:36:14] 10Beta-Cluster-Infrastructure, 10Patch-For-Review, 10User-MarcoAurelio: nlwikipedia on Beta Cluster is wrongly shown under "Other Wikimedia Projects" - https://phabricator.wikimedia.org/T188582#4015479 (10MarcoAurelio) Glad to hear that. Regards. [20:36:24] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<44.44%) [20:36:52] 10Beta-Cluster-Infrastructure, 10User-MarcoAurelio: nlwikipedia on Beta Cluster is wrongly shown under "Other Wikimedia Projects" - https://phabricator.wikimedia.org/T188582#4012705 (10MarcoAurelio) [20:39:05] (03PS1) 10Niharika29: Switch default dashboard [extensions/PageAssessments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/415653 [20:40:37] (03PS2) 10Niharika29: Switch default dashboard [extensions/PageAssessments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/415653 [20:43:25] (03PS3) 10Niharika29: Switch default dashboard [extensions/PageAssessments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/415653 [20:48:10] !log maurelio@deployment-tin:~$ foreachwiki extensions/AbuseFilter/maintenance/purgeOldLogIPData.php [20:48:17] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:49:34] (03CR) 10Niharika29: [C: 032] Switch default dashboard [extensions/PageAssessments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/415653 (owner: 10Niharika29) [20:55:28] (03CR) 10Niharika29: [V: 032 C: 032] Switch default dashboard [extensions/PageAssessments] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/415653 (owner: 10Niharika29) [20:57:28] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T183962#4015598 (10Krinkle) [21:01:26] PROBLEM - Free space - all mounts on integration-slave-jessie-1001 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found)integration.integration-slave-jessie-1001.diskspace._srv.byte_percentfree (<11.11%) [21:04:53] no_justification i've managed to switch highlight.js to prism.js (it's a wip change) https://gerrit-review.googlesource.com/c/gerrit/+/163090 :) [21:08:13] no_justification the colours look really good for the .php file. [21:08:21] PROBLEM - Puppet errors on deployment-mx02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [21:11:40] PROBLEM - Free space - all mounts on integration-slave-jessie-1004 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1004.diskspace._srv.byte_percentfree (<20.00%) [21:27:49] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T183962#4015740 (10Mholloway) [21:36:30] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.23 deployment blockers - https://phabricator.wikimedia.org/T183962#4015772 (10greg) [21:41:21] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.22 deployment blockers - https://phabricator.wikimedia.org/T183961#4015780 (10greg) 05Open>03Resolved [21:51:03] no_justification this is what it looks like http://recordit.co/3edXgkhPMZ [21:51:04] :) [21:51:07] (nice :)) [21:58:11] Project mwext-phpunit-coverage-publish build #1648: 04FAILURE in 43 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1648/ [21:58:52] Yippee, build fixed! [21:58:54] Project mwext-phpunit-coverage-publish build #1649: 09FIXED in 40 sec: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/1649/ [22:13:49] is there a way to a CI job to commit/push change into another repo? are there any jobs that already do that? [22:14:27] I mean for post-merge job - i.e. if you merge a change to a repo, it gets deployed into another repo [22:17:46] SMalyshev a submodule? [22:18:34] paladox: well, not exactly a submodule of this one. So I have the WDQS GUI repo, which is source of the GUI, and I have production deploy repo, which is built (minified, etc.) from that source [22:19:00] so right now when something is merged into GUI repo, I manually build it and manually submit it to production repo [22:19:13] I wonder if it's possible to do the same automatically [22:19:19] oh, not sure [22:20:06] I see portals build might be doing something like that [22:20:44] but not sure whether it's fully auto or not [22:31:57] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.25 deployment blockers - https://phabricator.wikimedia.org/T183964#4015913 (10greg) a:03demon [22:32:11] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.26 deployment blockers - https://phabricator.wikimedia.org/T183965#4015915 (10greg) a:03demon [22:34:18] SMalyshev: the only trick to it would be securing the credentials for the account making the gerrit commit. I don't remember if there is a good solution for that in our main Jenkins setup or not. [22:34:48] bd808: I think I can just create a new account for Gerrit.... [22:35:08] but not sure what happens with keys etc. [22:35:20] portal builder somehow makes it work as it seems [22:35:22] yeah, you would definitely not want to use a "real" person's account [22:38:25] SMalyshev: it looks like the magic in https://integration.wikimedia.org/ci/job/wikimedia-portals-build/configure is the "portalsbuilder [22:38:43] grrr. "portalsbuilder" credential account [22:38:59] so I'd say, yes it is possible. [22:39:16] yeah what I haven't figured out yet is how to get secret key to CI? [22:40:07] Jenkins has a secrets store [22:40:24] bd808: also I don't have permissions to see the URL you've posted :( [22:40:44] are you logged in? [22:41:05] * bd808 isn't sure what things he can do in Jenkins are super powers [22:41:54] SMalyshev: there is a "credentials" system in Jenkins that seems to hold various ssh keys and other secrets [22:42:14] ssh-credentials [22:42:18] aha, sounds good [22:42:18] I'd say the easy way to do it is to write up a phab task and get help from hashar :) [22:43:02] ^ [22:43:24] ok, I already have general phab task: https://phabricator.wikimedia.org/T160943 but need to figure out specific steps [22:47:05] SMalyshev: this is the definition for the wikimedia-portals-build job -- https://github.com/wikimedia/integration-config/blob/89f0f924fa342819e2654ffa7cc5ea32049e8dcd/jjb/wikimedia.yaml#L1-L87 [22:47:50] the shell step that starts on line 40 is probably what you'd kind of like to duplicate [22:47:59] bd808: yeah thanks found that one and writing mine as copy of it... there are some differences like it needs to be triggered by postmerge build etc. but I'll try to figure it out [22:48:58] bd808: something I am still not fully understanding is relationship between jobs and projects and where I tell it when to run what [22:49:30] its kind of magic. Hashar can help with that part for sure [22:49:37] ok :) [22:49:59] I think basically you want the job to exist and then to wire it into zuul so that it is triggered at the right time [22:50:14] right. the question is how :) [22:51:01] adding the right stuff in zuul/layout.yaml [22:51:33] you would add a postmerge stanza to the driver project and have it fire the new job [22:52:24] ah ok that's what I thought. But not sure what "projects" is doing [22:52:47] I mean "project:" part - how it relates to layout. [22:53:00] I see jobs mentioned in both places, so I wonder... [23:01:25] RECOVERY - Free space - all mounts on integration-slave-jessie-1001 is OK: OK: integration.integration-slave-jessie-1001.diskspace._mnt.byte_percentfree (No valid datapoints found) [23:02:29] (03PS1) 10Smalyshev: Create job for building WDQS GUI [integration/config] - 10https://gerrit.wikimedia.org/r/415769 (https://phabricator.wikimedia.org/T160943) [23:28:22] 10Release-Engineering-Team (Kanban), 10Wiki-Setup (Close): Close chairwiki - https://phabricator.wikimedia.org/T184961#3901664 (10MarcoAurelio) Wondering if deletion wouldn't be more appropriate after the content there (if any) is moved to the boardwiki. A closed private wiki is just weird. Of course, deletion... [23:40:06] 10Beta-Cluster-Infrastructure, 10Continuous-Integration-Config, 10Release-Engineering-Team (Next): Use cron instead of Jenkins for beta deployments - https://phabricator.wikimedia.org/T188367#4016096 (10greg) [23:40:16] 10Scap: Provide a mechanism ('scap lock'?) to exclude an individual host from deploys - https://phabricator.wikimedia.org/T188347#4016098 (10greg)