[06:31:36] PROBLEM - Puppet errors on deployment-zotero01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [07:06:36] RECOVERY - Puppet errors on deployment-zotero01 is OK: OK: Less than 1.00% above the threshold [0.0] [07:52:52] yay, node10 [07:53:21] legoktm: thoughts, is it evil that I use a jenkins job to do this https://phabricator.wikimedia.org/T192006 ? [08:34:06] !log ores ae96071 is going to beta [08:34:08] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [08:43:25] yup, ores went down :/ [08:58:39] (03PS1) 10Hashar: Experimental job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459509 [08:59:52] (03CR) 10Hashar: [C: 032] "That should be enough for the CI config side :]" [integration/config] - 10https://gerrit.wikimedia.org/r/459509 (owner: 10Hashar) [09:00:32] (03CR) 10jerkins-bot: [V: 04-1] Experimental job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459509 (owner: 10Hashar) [09:02:56] (03PS2) 10Hashar: Experimental job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459509 [09:05:54] (03CR) 10Hashar: [C: 032] Experimental job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459509 (owner: 10Hashar) [09:08:15] (03Merged) 10jenkins-bot: Experimental job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459509 (owner: 10Hashar) [09:25:19] okay ores is back up now, it was bad hiera config :D [10:08:56] 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.32.0-wmf.20 deployment blockers - https://phabricator.wikimedia.org/T191066 (10ArielGlenn) [10:58:45] (03PS1) 10Hashar: Promote job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459527 [10:58:56] (03CR) 10Hashar: [C: 032] Promote job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459527 (owner: 10Hashar) [11:00:38] (03Merged) 10jenkins-bot: Promote job for wikimedia-cz/tracker [integration/config] - 10https://gerrit.wikimedia.org/r/459527 (owner: 10Hashar) [11:28:06] 10Continuous-Integration-Infrastructure (shipyard): Rebuild quibble images for Chrome 69 and Firefox 60 - https://phabricator.wikimedia.org/T203902 (10hashar) [11:52:03] (03PS1) 10Hashar: Modify access rules [wikimedia-cz/tracker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/459539 [11:52:39] (03CR) 10Hashar: [V: 032 C: 032] Modify access rules [wikimedia-cz/tracker] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/459539 (owner: 10Hashar) [11:55:44] hashar: hey, sorry to bother you non-stop. This is about something different. I enabled git lfs for a native gerrit repo (not a mirror of anything) and I need push right for half an hour to rewrite history with git lfs (I have done this already for other repos): https://gerrit.wikimedia.org/r/admin/projects/research/ores/wheels [11:55:52] it would be great, I'm so sorry [11:57:57] part of https://phabricator.wikimedia.org/T197096 [12:15:40] 10Gerrit: Gerrit can not create parent project when child exists ( wikimedia-cz ) - https://phabricator.wikimedia.org/T203956 (10hashar) [12:16:55] 10Gerrit: Gerrit can not create parent project when child exists ( wikimedia-cz ) - https://phabricator.wikimedia.org/T203956 (10hashar) [13:57:46] (03PS1) 10Hashar: Update MediaWiki jobs to Quibble 0.0.24 [integration/config] - 10https://gerrit.wikimedia.org/r/459550 [13:59:35] (03CR) 10Hashar: "88 jobs switched." [integration/config] - 10https://gerrit.wikimedia.org/r/459550 (owner: 10Hashar) [14:06:33] (03PS2) 10Hashar: Update MediaWiki jobs to Quibble 0.0.24 [integration/config] - 10https://gerrit.wikimedia.org/r/459550 [14:06:35] (03PS1) 10Hashar: Use latest quibble-stretch-bundle container [integration/config] - 10https://gerrit.wikimedia.org/r/459555 [14:07:03] (03CR) 10Hashar: [C: 032] Update MediaWiki jobs to Quibble 0.0.24 [integration/config] - 10https://gerrit.wikimedia.org/r/459550 (owner: 10Hashar) [14:07:09] (03CR) 10Hashar: [C: 032] Use latest quibble-stretch-bundle container [integration/config] - 10https://gerrit.wikimedia.org/r/459555 (owner: 10Hashar) [14:09:00] (03Merged) 10jenkins-bot: Update MediaWiki jobs to Quibble 0.0.24 [integration/config] - 10https://gerrit.wikimedia.org/r/459550 (owner: 10Hashar) [14:09:15] (03Merged) 10jenkins-bot: Use latest quibble-stretch-bundle container [integration/config] - 10https://gerrit.wikimedia.org/r/459555 (owner: 10Hashar) [14:15:48] (03PS2) 10Hashar: Explicitly set Quibble --log-dir [integration/config] - 10https://gerrit.wikimedia.org/r/458788 [14:16:57] (03CR) 10Hashar: [C: 032] "Rebased, notably to add --log-dir to the DonationInterface experimental jobs." [integration/config] - 10https://gerrit.wikimedia.org/r/458788 (owner: 10Hashar) [14:17:35] !log Update CI Quibble jobs to explicitly set the log directory to /log (--log-dir /log) | https://gerrit.wikimedia.org/r/#/c/integration/config/+/458788/ [14:17:38] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [14:18:49] (03Merged) 10jenkins-bot: Explicitly set Quibble --log-dir [integration/config] - 10https://gerrit.wikimedia.org/r/458788 (owner: 10Hashar) [14:33:08] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Mail, 10Operations, and 2 others: Ensure Jenkins mail configuration supports outbound smtp server failover - https://phabricator.wikimedia.org/T203607 (10hashar) 05Open>03Resolved Turns out releases1001 / releases2002 Jenki... [15:36:41] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10thcipriani) Friday I removed the integration-slave-docker-1026 node since it was constantly ru... [15:43:30] 10Release-Engineering-Team (Kanban), 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic, and 2 others: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10greg) >>! In T125618#4476717, @Ree... [15:45:21] 10Continuous-Integration-Infrastructure, 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review, 10Zuul: Zuul cancel all changes when a change is manually merged - https://phabricator.wikimedia.org/T203846 (10greg) >>! In T203846#4568265, @hashar wrote: > I don't think that can be fixed in #Gerrit shor... [15:57:31] (03PS42) 10Zfilipin: Selenium daily tests for beta using Docker/wdio [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [16:02:47] 10Release-Engineering-Team (Kanban), 10Education-Program-Dashboard, 10MediaWiki-extensions-EducationProgram, 10Epic, and 2 others: Deprecate and remove the EducationProgram extension from Wikimedia servers after June 30, 2018 - https://phabricator.wikimedia.org/T125618 (10Reedy) Not really. Waiting on T174... [16:03:13] (03CR) 10Hashar: "recheck" [integration/quibble] - 10https://gerrit.wikimedia.org/r/458786 (owner: 10Hashar) [16:05:30] (03CR) 10Zfilipin: "PS42 implements changes required by @hashar" (032 comments) [integration/config] - 10https://gerrit.wikimedia.org/r/443931 (https://phabricator.wikimedia.org/T188742) (owner: 10Hashar) [16:10:17] (03PS1) 10Lucas Werkmeister (WMDE): Add abian to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/459572 [16:15:24] (03CR) 10Abián: [C: 031] "Thanks for the request." [integration/config] - 10https://gerrit.wikimedia.org/r/459572 (owner: 10Lucas Werkmeister (WMDE)) [16:17:34] (03PS3) 10Hashar: In Docker default log dir to be under workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/458786 [16:23:31] 10Diffusion, 10Repository-Admins, 10User-Luke081515: Connect/Link Phabricator/Maniphest projects to Diffusion repositories - https://phabricator.wikimedia.org/T120915 (10mmodell) @mgchecker: I don't see a problem with doing that. [16:23:43] PROBLEM - Puppet errors on deployment-certcentral03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [16:24:33] (03CR) 10Hashar: [C: 032] In Docker default log dir to be under workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/458786 (owner: 10Hashar) [16:25:18] (03Merged) 10jenkins-bot: In Docker default log dir to be under workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/458786 (owner: 10Hashar) [16:25:28] (03PS5) 10Hashar: Allow specifying database data directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/458826 [16:25:49] (03CR) 10jenkins-bot: In Docker default log dir to be under workspace [integration/quibble] - 10https://gerrit.wikimedia.org/r/458786 (owner: 10Hashar) [16:25:54] (03CR) 10Hashar: [C: 032] Allow specifying database data directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/458826 (owner: 10Hashar) [16:26:33] (03Merged) 10jenkins-bot: Allow specifying database data directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/458826 (owner: 10Hashar) [16:27:03] (03CR) 10jenkins-bot: Allow specifying database data directory [integration/quibble] - 10https://gerrit.wikimedia.org/r/458826 (owner: 10Hashar) [16:28:44] RECOVERY - Puppet errors on deployment-certcentral03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:33:58] (03PS1) 10Hashar: docker: rebuild Quibble stretch for Chromium/Firefox [integration/config] - 10https://gerrit.wikimedia.org/r/459575 (https://phabricator.wikimedia.org/T203902) [16:34:59] (03CR) 10Dduvall: [C: 04-1] "It would be prudent to roll this back until we can provide a stable storage space for Docker based jobs. (See https://phabricator.wikimedi" [integration/config] - 10https://gerrit.wikimedia.org/r/457070 (https://phabricator.wikimedia.org/T203181) (owner: 10Legoktm) [16:36:05] (03PS1) 10Hashar: Quibble Stretch jobs with Chromium 69 [integration/config] - 10https://gerrit.wikimedia.org/r/459576 (https://phabricator.wikimedia.org/T203902) [16:37:10] (03PS1) 10Hashar: Bump wdio selenium jobs Chromium 60/Firefox 60 [integration/config] - 10https://gerrit.wikimedia.org/r/459577 (https://phabricator.wikimedia.org/T203902) [16:49:46] 10Phabricator, 10Release-Engineering-Team (Kanban), 10User-MModell, 10User-Ryasmeen: 404 on workboard for an existing project (due to custom filter applied which did not exist in database) - https://phabricator.wikimedia.org/T199207 (10mmodell) [16:50:04] (03PS1) 10Hashar: docker: update npm-browser for Firefox/Chromium [integration/config] - 10https://gerrit.wikimedia.org/r/459580 (https://phabricator.wikimedia.org/T203902) [16:50:26] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-Addshore: Add some more m4executor docker slaves for Jenkins - https://phabricator.wikimedia.org/T201972 (10dduvall) >>! In T201972#4516467, @dduvall wrote: > | instance type | executors (n) | vcpu/n |... [16:51:38] (03PS1) 10Hashar: Update oojs-ui jobs for Chromium 69/Firefox 60 [integration/config] - 10https://gerrit.wikimedia.org/r/459582 (https://phabricator.wikimedia.org/T203902) [16:54:58] 10Release-Engineering-Team (Watching / External), 10Operations, 10SRE-Access-Requests: Add contint-roots to releases{1,2}001 - https://phabricator.wikimedia.org/T201470 (10Dzahn) a:05ArielGlenn>03None [16:55:03] 10Phabricator, 10Release-Engineering-Team (Kanban), 10monitoring, 10Browser-Tests, and 2 others: Develop tests for phabricator search to detect regressions / search quality issues - https://phabricator.wikimedia.org/T182160 (10mmodell) a:05mmodell>03None [16:56:30] 10Deployments, 10Release-Engineering-Team (Someday), 10wikitech.wikimedia.org, 10User-MModell: Create an easier way to add/remove/modify patches for SWAT - https://phabricator.wikimedia.org/T171940 (10mmodell) [16:58:52] If there's now a milestone in CI-infrastructure called "shipyard", can we add one called Habourmaster or is that too on-the-nose? ;-) [17:00:08] 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 3 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832 (10mmodell) a:05mmodell>03None Unassigning but I'm sti... [17:05:13] James_F: :) :) [17:11:45] next CI project will be named slipway [17:12:03] I am not sure what the codename will be used for though ;]] [17:13:33] 10Continuous-Integration-Infrastructure, 10MediaWiki-Cache, 10Patch-For-Review, 10Technical-Debt: BagOStuff should detect obsolete serialization or an unserialization resulting in a "wrong" object - https://phabricator.wikimedia.org/T156541 (10mmodell) [17:21:27] Would it be possible to change the gerrit posted messages from zuul/jenkins from "SUCCESS"/"FAILURE" to "✅ SUCCESS"/"❌ FAILURE" so it has a bit of colour? [17:23:04] James_F yes, unless python dosen't support emojies? [17:23:32] James_F something like: https://gerrit-review.googlesource.com/c/gerrit/+/195350#message-d8d2d722ca43c1b4b1c2eccd68f8495c8151835c [17:23:33] ? [17:24:10] Yeah, exactly. [17:26:10] they are just bytes [17:26:36] the database collation would be a bigger concern imho [17:28:05] not really [17:28:10] we use notedb for comments [17:28:18] the db is for groups and other smaller stuff now. [17:28:25] Platonides ^^ [17:28:56] notedb uses git storage? [17:30:47] yep [17:30:52] gerrit is getting rid of the db [17:32:24] starting with the removal of needing a db in 3.0 [17:32:31] it's already been made a blocker. [17:32:58] it'll be webscale then [17:33:29] which will also unblock gerrit2001 once the db is fully drop from gerrit :) [17:35:00] Gotta store the data somewher [17:36:02] heh [17:37:09] * paladox knows that it's a blocker because i created that task upstream [17:37:17] after groups were migrated to notedb on master [18:20:46] 10MediaWiki-Codesniffer: Disallow use of @private and @protected using MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203980 (10Umherirrender) [18:29:56] 10MediaWiki-Codesniffer: Disallow use of @private and @protected using MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203980 (10Legoktm) I think it's important that we retain `@private`, because there are cases where functions/methods should be considered an... [18:48:20] 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Next), 10Wikimedia-Logstash, 10Jenkins, 10Patch-For-Review: Send Jenkins build log and results to ElasticSearch - https://phabricator.wikimedia.org/T78705 (10dduvall) I'd like to revisit this as a way to record build durations along wit... [19:55:49] 10MediaWiki-Codesniffer: Disallow use of @private and @protected using MediaWiki.Commenting.FunctionAnnotations.UnrecognizedAnnotation - https://phabricator.wikimedia.org/T203980 (10Umherirrender) It seems it should only be documented in the sniff and there must stay. Some @protected have comments about unit tes... [20:21:59] 10Release-Engineering-Team (Kanban), 10MediaWiki-General-or-Unknown, 10MW-1.32-release-notes (WMF-deploy-2018-09-18 (1.32.0-wmf.22)), 10Security, 10User-zeljkofilipin: `npm audit` for mediawiki/core found 24 vulnerabilities - https://phabricator.wikimedia.org/T194280 (10Jdforrester-WMF) 05Open>03Resol... [20:58:13] Project beta-scap-eqiad build #221598: 15ABORTED in 5 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221598/ [20:58:23] ^ me [21:54:18] Project beta-scap-eqiad build #221602: 15ABORTED in 7 min 18 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/221602/ [21:54:30] ^ me again [22:57:32] hashar what will slipway be? [23:34:43] PROBLEM - Puppet errors on deployment-certcentral03 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [23:36:48] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [23:44:43] RECOVERY - Puppet errors on deployment-certcentral03 is OK: OK: Less than 1.00% above the threshold [0.0]