[00:01:40] RoanKattouw with that change, do you see it affecting any other things? [00:01:48] No it looks fine otherwise [00:01:55] It's fixed at the cost of ugliness, yes [00:01:55] Ok thanks [00:02:01] yep [00:02:01] Thanks [00:02:05] your welcome :) [00:14:25] The 'new' SWAT hours will be 06:00/11:00/16:00 SF, right? [00:14:42] 06Release-Engineering-Team, 10Gerrit, 10Wikimedia-Logstash, 07Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#2494353 (10demon) [00:14:51] bd808: Should be fun ^ :) [00:15:12] ostriches i fixed the problem [00:16:20] ostriches: we get logs from cassandra [00:16:37] I think their config should have log4j magic in it [00:17:17] ostriches https://gerrit.wikimedia.org/r/#/c/301027/2 [00:17:20] paladox: I'll review and get it merged tomorrow. [00:20:14] bd808: Hadoop or cassandra? [00:20:18] I see something in the former. [00:20:26] ok thanks [00:21:15] ostriches: I'm not sure that hadoop is still wired up. cassandra has https://github.com/wikimedia/operations-puppet/blob/production/modules/cassandra/manifests/logging.pp and https://github.com/wikimedia/operations-puppet/blob/production/modules/cassandra/templates/logback.xml-2.2.erb [00:21:33] looks like it uses logback rather than palin log4j [00:21:49] something similar should be possible [00:21:51] Ah I was looking at puppet/modules/cdh/templates/hadroop/log4j.properties.erb [00:22:10] yah, maybe hadoop is still wired up too [00:23:12] Either way I'll probably have to bring in net.logstash.logback.* lib or log4j.appender.gelf.* to get it to work [00:23:17] Anyway, I got enough to noodle now, thx [00:23:56] twentyafterfour: I see that the patch to drop 1.25 tests was merged, but the job is still running--maybe there's some cache purge or restarting left to do? [00:25:18] awight: Going afk, but zuul needs reload to pick up config changes :) [00:25:26] (although it might not affect an in-process job) [00:25:40] (might have to hard stop/start) [00:25:45] (anyway, afk for realz now) [00:26:22] ostriches mutante merged it [00:26:42] RoanKattouw mutante merged https://gerrit.wikimedia.org/r/301027 :) [00:30:35] Cool [00:34:43] (03Draft2) 10Paladox: Testing [integration/zuul] - 10https://gerrit.wikimedia.org/r/301033 [00:34:47] (03Abandoned) 10Paladox: Testing [integration/zuul] - 10https://gerrit.wikimedia.org/r/301033 (owner: 10Paladox) [00:34:57] It has been deployed now [00:35:05] RoanKattouw ^^ [00:38:01] RoanKattouw: i noticed the line wrap issue too. and merged the fix from paladox. looks better to me [00:38:10] :) [00:39:20] 06Release-Engineering-Team, 10Gerrit, 10Wikimedia-Logstash, 07Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#2494353 (10bd808) See also {T64505}. Note I'm not sure if that is really useful or not. I think it was from the early Logstash days when I wa... [01:08:45] !log deployed ores a291da1 in sca03, ores-beta.wmflabs.org works as expected [01:08:48] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [01:46:08] ostriches: FYI https://phabricator.wikimedia.org/T141329 is probably caused by the new Gerrit version [01:55:33] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:26:53] RoanKattouw: Well yes, you couldn't create patch sets from the web before ;-) [02:27:04] Yes exactly :) [02:27:45] Ah ok, so it kept the original author as author and made you committer. [02:28:03] Actually no you could, the old commit message editor is gone so you have to use the new editor for that too now [02:28:05] I can see that happening. Similar would happen if I cherry-picked you, amended, and re-pushed (unless I reset the author) [02:28:14] And rebase patchsets still work [02:28:22] ostriches: No, I don't think that's true [02:28:36] Pretty sure it is :) [02:28:41] It uses the user submitting the patch to Gerrit (the committer typically), not the author [02:28:59] Then why do we have "Forge Author" granted to everyone? [02:29:09] because that is totally how it happens 90% of the time. [02:29:11] Because you need to be able to amend others' patches [02:29:17] --amend does not change the author, only the committer [02:29:28] That's my point. [02:29:34] It's treating the change as an --amend [02:29:39] Rather than a brand new patch [02:29:50] If I rebase your patch from the web, or edit its commit message in the pre-upgrade Gerrit, or download your patch and amend it and push it back up, all of those have you as the author and me as the committer, and all of those had grrrit-wm report (PS2) Catrope: blah blah (owner: Demon) [02:30:40] What's a bit odd to me is that this behavior didn't change for rebases but it did for commit msg changes. I'm guessing the way Gerrit outputs this in its event stream must have changed [02:33:46] That wouldn't surprise me re: event stream. [02:33:55] Commit message change logic is probably 100% different now [02:34:01] Since it's integrated with the inline editor [02:35:35] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [02:40:51] RoanKattouw: True story, upstream is experimenting with *a new* UI. https://gerrit-review.googlesource.com/c/79298/?polygerrit=0 [02:40:59] (whoops, swap for =1 to enable) [02:41:41] Oh looks like upstream fixed the sillyness where the unified diff screen is still the old code [02:42:27] (We can't pull that in yet. Not feature complete...tons of things just Aren't There Yet, also only in master) [02:42:29] Wow that's actually better IMO [02:42:33] Much clearer [02:42:36] :) [02:42:51] And less of a difference between the old UI and this one in terms of moving ALL OF THE CHEESE [02:44:48] As soon as it's more feature-complete we can bring it in :) [03:28:54] reading https://wikitech.wikimedia.org/wiki/Icinga , it's a bit disappointing [03:29:19] some parts of that page were written by me in 2006 [03:31:56] I think half of it must have been written by me, altogether [03:35:50] I'm guessing spence was a pmtpa host [03:37:46] wtf is the section about the USB dongle [03:44:39] for paging [03:45:06] it was so unreliable that eventually mark made his own [03:45:11] out of an arduino or something [03:47:35] 2015-08-25 [03:47:35] 09:40 YuviPanda: restarted ircecho on shinken-01 [03:47:35] ==='"`UNIQ--h-4--QIN... (more) [03:48:12] says https://wikitech.wikimedia.org/wiki/Nova_Resource:Shinken [03:57:42] it's {{#invoke:String|clip|s={{Nova_Resource:{{{Project Name}}}/SAL}}|max=600|trail=... ([[Nova_Resource:{{{Project Name}}}/SAL|more]])}} (from Template:Nova_Project) [04:02:09] which essentially does mw.ustring.sub( s, 1, max ) .. trail [04:18:31] Yippee, build fixed! [04:18:32] Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #85: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/85/ [04:26:32] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:35:21] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [04:38:07] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [04:42:31] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [05:01:33] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:13:08] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [05:15:20] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [05:22:33] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [05:36:22] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [06:13:45] 06Release-Engineering-Team, 10Gerrit, 06Operations, 13Patch-For-Review: replace gerrit server (ytterbium) with jessie server (lead) - https://phabricator.wikimedia.org/T125018#2494697 (10Dzahn) decom: https://gerrit.wikimedia.org/r/#/c/300806/ https://gerrit.wikimedia.org/r/#/c/300812/ 01:36 ostriches:... [06:16:22] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [08:07:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [08:20:11] (03CR) 10Hashar: [C: 04-1] "This change is no more needed. I talked with Paladox about it on Monday and it has a few issues. The "proper" fix has been to inject the " [integration/config] - 10https://gerrit.wikimedia.org/r/300790 (owner: 10Paladox) [08:22:08] (03CR) 10Hashar: "I have hacked the job over the week-end and it seems to work under Jessie + Java 8. I have extended details on T139137 and a .plan to pol" [integration/config] - 10https://gerrit.wikimedia.org/r/299674 (https://phabricator.wikimedia.org/T139137) (owner: 10Mholloway) [08:28:21] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog, 13Patch-For-Review: [Dev] Fix periodic tests - https://phabricator.wikimedia.org/T139137#2494952 (10hashar) The hacky job I have created does poll the git repository and it is now passing just fine! https://integration.wikimedia.org/ci/job/app... [08:37:30] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [08:42:10] !log T141269 On integration-slave-trusty-1018 , deleting workspace that has a corrupt git: rm -fR /mnt/jenkins-workspace/workspace/mediawiki-extensions-hhvm* [08:42:11] T141269: zuul-cloner fails mediawiki-extensions-hhvm job with "error: object file .git/objects/30 is empty" - https://phabricator.wikimedia.org/T141269 [08:42:13] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [08:42:45] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:43:00] 10Continuous-Integration-Infrastructure: zuul-cloner fails mediawiki-extensions-hhvm job with "error: object file .git/objects/30 is empty" - https://phabricator.wikimedia.org/T141269#2494976 (10hashar) 05Open>03Resolved a:03hashar Both ran on slave integration-slave-trusty-1018 that had a corrupt git repo... [08:44:04] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2494980 (10hashar) 05Resolved>03Open If it happened again, then it is not fixed! [09:12:30] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [09:42:19] PROBLEM - SSH on integration-slave-trusty-1011 is CRITICAL: Server answer [09:47:25] (03PS1) 10Hashar: (WIP) overhaul / document debian-glue jobs (WIP) [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [09:47:42] (03PS2) 10Hashar: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [09:48:12] (03CR) 10jenkins-bot: [V: 04-1] Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [09:50:15] (03PS3) 10Hashar: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [09:50:47] (03CR) 10jenkins-bot: [V: 04-1] Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [09:54:33] (03PS4) 10Hashar: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [09:58:30] (03CR) 10Hashar: "That is large of overhaul of the jobs building the debian packages. I have sprinted a working 'debian-glue' job over the last few days/wee" [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [10:18:11] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2495180 (10Paladox) What I meant is, legoktm said it wasent this it was something to do with Depnds-On: not Circular dependencies @Legoktm would you be able to explain t... [10:21:27] 06Release-Engineering-Team, 10MediaWiki-History-or-Diffs, 07Beta-Cluster-reproducible, 05MW-1.28-release-notes, 05WMF-deploy-2016-07-26_(1.28.0-wmf.12): Viewing diffs throws 503@beta - https://phabricator.wikimedia.org/T141272#2495201 (10Danny_B) [10:23:16] hashar hi, and :), i see you overhaul the debian-glue test :) [10:26:29] (03PS1) 10Hashar: Add debian-glue job to all operations/debs repos [integration/config] - 10https://gerrit.wikimedia.org/r/301093 [10:41:57] (03PS20) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [10:52:45] (03PS21) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) [10:57:28] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:58:13] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:01:05] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:03:01] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 70.00% of data above the critical threshold [0.0] [11:03:11] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:31:36] !log enable puppet agent on integration-puppetmaster . Had it disabled while hacking on https://gerrit.wikimedia.org/r/#/c/300830/ [11:31:40] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:37:15] !log rebased integration puppetmaster git repo [11:37:18] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:37:29] RECOVERY - Puppet run on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:01] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:11] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [11:38:13] RECOVERY - Puppet run on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:05] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:43:32] PROBLEM - Puppet run on integration-slave-trusty-1003 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:45:10] PROBLEM - Puppet run on integration-aptly01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:45:34] PROBLEM - Puppet run on zuul-dev-jessie is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [11:45:38] RECOVERY - Puppet staleness on integration-puppetmaster is OK: OK: Less than 1.00% above the threshold [3600.0] [11:47:02] PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:47:34] PROBLEM - Puppet run on integration-raita is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:50:20] PROBLEM - Puppet run on castor is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [11:50:52] PROBLEM - Puppet run on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [11:52:32] PROBLEM - Puppet run on integration-slave-trusty-1001 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:53:12] PROBLEM - Puppet run on integration-slave-trusty-1016 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [11:55:11] PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:55:45] PROBLEM - Puppet run on integration-slave-jessie-android is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:57:40] (03PS1) 10Legoktm: Run tests for ParserMigration [integration/config] - 10https://gerrit.wikimedia.org/r/301109 [11:57:51] PROBLEM - Puppet run on integration-slave-trusty-1004 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [11:58:20] (03CR) 10Legoktm: [C: 032] Run tests for ParserMigration [integration/config] - 10https://gerrit.wikimedia.org/r/301109 (owner: 10Legoktm) [11:59:14] (03Merged) 10jenkins-bot: Run tests for ParserMigration [integration/config] - 10https://gerrit.wikimedia.org/r/301109 (owner: 10Legoktm) [11:59:18] !log deploying https://gerrit.wikimedia.org/r/301109 [11:59:22] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [11:59:53] !log also pulled in I73f01f87b06b995bdd855628006225879a17fee5 [11:59:56] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:00:21] PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:00:33] PROBLEM - Puppet run on integration-publisher is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:00:54] (03CR) 10Legoktm: "@twentyafterfour: Please deploy zuul changes when merging...were the jjb jobs deleted too?" [integration/config] - 10https://gerrit.wikimedia.org/r/299940 (owner: 10Ejegg) [12:03:11] PROBLEM - Puppet run on integration-slave-trusty-1023 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:05:12] PROBLEM - Puppet run on integration-jessie-lego-test01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:05:24] PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [12:06:50] PROBLEM - Puppet run on integration-slave-trusty-1014 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [12:09:10] PROBLEM - Puppet run on integration-slave-trusty-1018 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [12:20:57] 10Continuous-Integration-Config, 06Wikipedia-Android-App-Backlog: Jenkins should build release Android APKs - https://phabricator.wikimedia.org/T104207#1409982 (10hashar) Is that essentially the same as T136662 ? [12:25:45] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2495362 (10Reedy) It was https://gerrit.wikimedia.org/r/#/c/300804 >[23:43:56] yes that was the problem >[23:44:03] the vendor repo is not controlled... [12:26:21] hashar: ^ It's a different bug AFAIK [12:39:20] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2495367 (10Paladox) Maybe that needs to be filled upstream so that zuul can pull it in either if it is not controlled by zuul. [12:46:09] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2495377 (10hashar) Zuul deadlocking due to unknown project was T128569. It is supposedly fixed by rCIZU0deaaadac7143692961b9d28abee8cea570ff3ce which I have deployed end o... [12:48:10] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream: Circular dependencies break Zuul - https://phabricator.wikimedia.org/T129938#2495394 (10hashar) 05Open>03Resolved /var/log/zuul/error.log has a spam of: ``` 2016-07-25 22:19:30,252 ERROR zuul.Scheduler: Exception in run handler: Traceback (most... [12:49:23] !log cherry-pick https://gerrit.wikimedia.org/r/#/c/300827/ on deployment-puppetmaster [12:49:27] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master [12:49:48] how often cherry-picks get wiped out from deployment-puppetmaster ? [12:50:05] 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream, 07WorkType-Maintenance: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2495396 (10hashar) 05Resolved>03Open Happened again with Zuul version: 2.1.0-151-g30a433b-wmf4precise1 >>! In T129938#2... [12:56:50] godog: they are not wiped out [12:56:57] godog: at least not intentionally [12:57:13] there is a cronjob that attempts to git remote update && git rebase origin/production every so often [12:57:18] which sometimes bails out on rebase conflict [12:57:30] some of us manually handle the rebase conflict [12:57:36] so it should stick :] [12:57:39] I am off [12:57:45] be bcack this evening [13:00:50] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:25:57] faster than expected [13:31:02] hashar: I see, thanks, I asked because I had to apply again 300827 [13:45:05] (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [13:45:11] (03PS5) 10Hashar: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [13:53:34] (03Abandoned) 10Hashar: Revert "Pin Zuul version used for testing" [integration/config] - 10https://gerrit.wikimedia.org/r/292346 (https://phabricator.wikimedia.org/T136610) (owner: 10Hashar) [14:06:41] (03PS1) 10Hashar: Fix up zuul layoutdiff [integration/config] - 10https://gerrit.wikimedia.org/r/301120 [14:06:52] (03CR) 10Hashar: [C: 032] Fix up zuul layoutdiff [integration/config] - 10https://gerrit.wikimedia.org/r/301120 (owner: 10Hashar) [14:07:07] (03PS6) 10Hashar: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) [14:07:19] (03PS3) 10Hashar: Add debian-glue job to all operations/debs repos [integration/config] - 10https://gerrit.wikimedia.org/r/301093 [14:07:58] (03Merged) 10jenkins-bot: Fix up zuul layoutdiff [integration/config] - 10https://gerrit.wikimedia.org/r/301120 (owner: 10Hashar) [14:09:12] (03CR) 10Hashar: [C: 032] Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [14:10:12] (03Merged) 10jenkins-bot: Overhaul / document debian-glue jobs [integration/config] - 10https://gerrit.wikimedia.org/r/301084 (https://phabricator.wikimedia.org/T117869) (owner: 10Hashar) [14:12:33] 10Continuous-Integration-Config: Make debian-glue to use Gerrit as the upstream repository - https://phabricator.wikimedia.org/T117869#2495655 (10hashar) 05Open>03Resolved Repositories now solely use the `debian-glue` job which first clone from Gerrit then fetch/checkout the proposed patch from Zuul. [14:12:36] 10Continuous-Integration-Config, 10MediaWiki-Debian: Set up CI auto-building for mediawiki/debian repository - https://phabricator.wikimedia.org/T122978#2495657 (10hashar) [14:14:05] (03PS1) 10Hashar: Make debian-glue concurrent [integration/config] - 10https://gerrit.wikimedia.org/r/301121 [14:22:01] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T139214#2495691 (10Jdforrester-WMF) [14:24:01] (03CR) 10Hashar: [C: 032] Make debian-glue concurrent [integration/config] - 10https://gerrit.wikimedia.org/r/301121 (owner: 10Hashar) [14:26:04] (03Merged) 10jenkins-bot: Make debian-glue concurrent [integration/config] - 10https://gerrit.wikimedia.org/r/301121 (owner: 10Hashar) [14:27:46] 10Continuous-Integration-Config, 10MediaWiki-Debian: Set up CI auto-building for mediawiki/debian repository - https://phabricator.wikimedia.org/T122978#2495702 (10hashar) 05Open>03Resolved a:03hashar CI is configured. The repository trigger the job `debian-glue` which lookup for the distribution in `deb... [14:47:50] PROBLEM - Puppet run on deployment-zookeeper01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:48:08] PROBLEM - Puppet run on deployment-urldownloader is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:48:18] PROBLEM - Puppet run on deployment-salt02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:50:46] PROBLEM - Puppet run on deployment-sca03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:50:48] PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:52:10] PROBLEM - Puppet run on deployment-elastic06 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:52:21] PROBLEM - Puppet run on deployment-eventlogging04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:52:29] PROBLEM - Puppet run on mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [14:52:30] 06Release-Engineering-Team, 10Gerrit, 10Wikimedia-Logstash, 07Technical-Debt: Look into shoving gerrit logs into logstash - https://phabricator.wikimedia.org/T141324#2495773 (10demon) >>! In T141324#2494439, @bd808 wrote: > See also {T64505}. Note I'm not sure if that is really useful or not. I think it wa... [14:52:35] PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [14:53:15] PROBLEM - Puppet run on deployment-ores-redis is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:53:21] PROBLEM - Puppet run on deployment-mediawiki03 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:53:21] PROBLEM - Puppet run on deployment-parsoid05 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:53:25] PROBLEM - Puppet run on deployment-parsoid07 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [14:55:21] PROBLEM - Puppet run on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:55:45] PROBLEM - Puppet run on deployment-mx is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:55:51] PROBLEM - Puppet run on deployment-parsoid08 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:55:59] PROBLEM - Puppet run on deployment-eventlogging03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:57:24] PROBLEM - Puppet run on deployment-conftool is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:58:09] PROBLEM - Puppet run on deployment-db2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [14:58:19] PROBLEM - Puppet run on deployment-restbase02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:58:31] PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [14:59:07] PROBLEM - Puppet run on deployment-tin is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [14:59:13] PROBLEM - Puppet run on deployment-ircd is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:59:17] PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:00:11] PROBLEM - Puppet run on deployment-pdf02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:00:53] PROBLEM - Puppet run on deployment-fluorine is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:00:55] PROBLEM - Puppet run on deployment-pdf01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:01:49] PROBLEM - Puppet run on deployment-imagescaler01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:02:01] PROBLEM - Puppet run on deployment-kafka01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:02:03] PROBLEM - Puppet run on deployment-ms-be01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [15:02:09] <|---|> o.O [15:02:25] PROBLEM - Puppet run on deployment-kafka03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:03:18] PROBLEM - Puppet run on deployment-memc04 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:03:32] PROBLEM - Puppet run on deployment-tmh01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:04:02] PROBLEM - Puppet run on deployment-sentry01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:05:42] PROBLEM - Puppet run on deployment-cache-upload04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:05:48] PROBLEM - Puppet run on deployment-poolcounter01 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:05:54] PROBLEM - Puppet run on deployment-redis01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:07:02] PROBLEM - Puppet run on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:07:20] PROBLEM - Puppet run on deployment-ms-be02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:07:25] hasharAway it seems there are alot of failures ^^ [15:07:52] PROBLEM - Puppet run on deployment-elastic05 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [15:08:16] PROBLEM - Puppet run on deployment-sentry2 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:08:43] PROBLEM - Puppet run on deployment-ms-fe01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:08:55] PROBLEM - Puppet run on deployment-memc05 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:10:27] PROBLEM - Puppet run on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:10:31] PROBLEM - Puppet run on deployment-restbase01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:10:57] PROBLEM - Puppet run on deployment-parsoid06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:14:05] PROBLEM - Puppet run on deployment-db1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [15:15:08] PROBLEM - Puppet run on deployment-mediawiki01 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:15:10] ugh [15:15:16] g'morning to you, too, shinken-wm [15:15:35] PROBLEM - Puppet run on deployment-redis02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [15:15:36] PROBLEM - Puppet run on deployment-elastic08 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:15:54] PROBLEM - Puppet run on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [15:16:50] PROBLEM - Puppet run on deployment-kafka04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [15:17:30] PROBLEM - Puppet run on deployment-cache-text04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:17:44] PROBLEM - Puppet run on deployment-mathoid is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [15:18:06] PROBLEM - Puppet run on deployment-conf03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [15:19:21] huh, seems to be the same error everywhere (from a quick spot check) Could not evaluate: No such file or directory - /etc/default/puppetmaster [15:19:48] thcipriani, issue on the server side? [15:19:57] which is seems to be false. deployment-puppetmaster:/etc/default/puppetmaster is a file. [15:20:06] seems like it. Might just need a kick. [15:24:58] hm, nope, issue on the clients. tried restarting puppetmaster [15:25:29] hmm [15:33:32] looks to be line 109 in modules/puppet/manifests/self/config.pp [15:33:58] cherry picked somewhere on the puppetmaster... [15:34:35] Oh could it be godog> !log cherry-pick https://gerrit.wikimedia.org/r/#/c/300827/ on deployment-puppetmaster [15:34:58] paladox: no, that's thumbor [15:35:02] Oh [15:35:05] sorry [15:35:27] np, anything I can do to help? [15:35:39] oh, nvmd, not cherry picked. Looks like 2742291fd54bd05a6597607ae96c7736a71b665a [15:36:13] godog: making our puppet in beta freak out a bit https://gerrit.wikimedia.org/r/#/c/301071/ :) [15:37:12] File:sad_trombone.wav [15:39:50] yeah probably needs a guard for $is_puppetmaster [15:40:05] _joe_: ^ what do you think? [15:43:59] ostriches do you mind having a look at https://gerrit.wikimedia.org/r/301001 please? Since i think i am correct at the classes but mutante needs some other people from releng to have a look at it please. [15:44:11] I will later. [15:44:14] ok [15:44:16] thanks [15:52:34] 06Release-Engineering-Team, 03releng-201617-q1, 15User-greg, 15User-zeljkofilipin: Perform a technical debt analysis of software and services maintained by WMF Release Engineering - https://phabricator.wikimedia.org/T138225#2495964 (10greg) Quick status update: Upon @dduvall's return and digging into this... [15:58:06] <_joe_> godog: uhm, yes [15:58:08] <_joe_> I'll fix it [15:58:44] _joe_: made a patch https://gerrit.wikimedia.org/r/#/c/301144/ seems to have fixed the issue, FWIW [15:58:45] nice, thanks _joe_ [15:58:56] and thcipriani ! [15:59:20] <_joe_> thcipriani: yeah that should work [15:59:37] <_joe_> merging [16:00:11] RECOVERY - Puppet run on deployment-mediawiki01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:43] RECOVERY - Puppet run on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [16:00:57] RECOVERY - Puppet run on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:02:33] RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:07] RECOVERY - Puppet run on deployment-db2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:03:15] RECOVERY - Puppet run on deployment-ores-redis is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:08] thanks all ( godog _joe_ thcipriani ) [16:05:09] RECOVERY - Puppet run on deployment-pdf02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:22] RECOVERY - Puppet run on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:52] RECOVERY - Puppet run on deployment-parsoid08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:05:58] RECOVERY - Puppet run on deployment-pdf01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:06:50] RECOVERY - Puppet run on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:04] RECOVERY - Puppet run on deployment-ms-be01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:24] RECOVERY - Puppet run on deployment-conftool is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:24] RECOVERY - Puppet run on deployment-kafka03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:07:54] 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.12 deployment blockers - https://phabricator.wikimedia.org/T139214#2495997 (10Jdforrester-WMF) [16:08:18] RECOVERY - Puppet run on deployment-restbase02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:08:20] RECOVERY - Puppet run on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:02] RECOVERY - Puppet run on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:08] RECOVERY - Puppet run on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:12] RECOVERY - Puppet run on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [16:09:48] 06Release-Engineering-Team, 15User-greg: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2496002 (10GWicke) [16:10:54] RECOVERY - Puppet run on deployment-fluorine is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:01] RECOVERY - Puppet run on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:01] RECOVERY - Puppet run on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:12:21] RECOVERY - Puppet run on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:19] RECOVERY - Puppet run on deployment-sentry2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:33] RECOVERY - Puppet run on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:45] RECOVERY - Puppet run on deployment-ms-fe01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:55] RECOVERY - Puppet run on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:27] RECOVERY - Puppet run on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:43] RECOVERY - Puppet run on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:49] RECOVERY - Puppet run on deployment-poolcounter01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:15:53] RECOVERY - Puppet run on deployment-redis01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:22] RECOVERY - Puppet run on deployment-ms-be02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:17:54] RECOVERY - Puppet run on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:32] RECOVERY - Puppet run on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:38] RECOVERY - Puppet run on deployment-elastic08 is OK: OK: Less than 1.00% above the threshold [0.0] [16:20:56] RECOVERY - Puppet run on deployment-parsoid06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:21:48] RECOVERY - Puppet run on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:04] RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:30] RECOVERY - Puppet run on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:34] RECOVERY - Puppet run on integration-raita is OK: OK: Less than 1.00% above the threshold [0.0] [16:22:44] RECOVERY - Puppet run on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:07] RECOVERY - Puppet run on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:23:28] anyone here who would want to create a new wiki? [16:23:38] as in the actual create_wiki script [16:24:07] RECOVERY - Puppet run on deployment-db1 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:09] RECOVERY - Puppet run on integration-aptly01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:35] RECOVERY - Puppet run on deployment-redis02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:35] RECOVERY - Puppet run on zuul-dev-jessie is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:47] RECOVERY - Puppet run on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:53] RECOVERY - Puppet run on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [16:25:53] RECOVERY - Puppet run on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:13] RECOVERY - Puppet run on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:29] RECOVERY - Puppet run on mira is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:31] RECOVERY - Puppet run on integration-slave-trusty-1001 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:42] mutante, what create_wiki script? [16:27:49] RECOVERY - Puppet run on deployment-zookeeper01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:27:55] you mean addWiki.php? [16:28:12] RECOVERY - Puppet run on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:13] RECOVERY - Puppet run on integration-slave-trusty-1016 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:17] RECOVERY - Puppet run on deployment-salt02 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:19] RECOVERY - Puppet run on deployment-parsoid05 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:20] RECOVERY - Puppet run on deployment-mediawiki03 is OK: OK: Less than 1.00% above the threshold [0.0] [16:28:25] RECOVERY - Puppet run on deployment-parsoid07 is OK: OK: Less than 1.00% above the threshold [0.0] [16:29:09] Krenair: yes, sorry, i mean addWiki [16:29:18] what was the order of things again [16:29:27] we need to get the mw-config changed landed first? [16:29:30] wikitech:Add_a_wiki [16:30:05] No, addWiki.php gets run before the deployment [16:30:12] RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:22] RECOVERY - Puppet run on castor is OK: OK: Less than 1.00% above the threshold [0.0] [16:30:48] RECOVERY - Puppet run on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0] [16:31:16] Krenair: ok, thanks [16:31:33] Krenair: just one more question, you think that kind of mw-config change is swat-able? [16:32:26] Arguably maybe, I wouldn't personally [16:32:35] I'd add a separate deployment window [16:32:44] fair enough, ok yep [16:32:50] RECOVERY - Puppet run on integration-slave-trusty-1004 is OK: OK: Less than 1.00% above the threshold [0.0] [16:35:46] RECOVERY - Puppet run on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0] [16:38:10] RECOVERY - Puppet run on integration-slave-trusty-1023 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:11] RECOVERY - Puppet run on integration-jessie-lego-test01 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:21] RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0] [16:40:33] RECOVERY - Puppet run on integration-publisher is OK: OK: Less than 1.00% above the threshold [0.0] [16:41:49] RECOVERY - Puppet run on integration-slave-trusty-1014 is OK: OK: Less than 1.00% above the threshold [0.0] [16:44:11] RECOVERY - Puppet run on integration-slave-trusty-1018 is OK: OK: Less than 1.00% above the threshold [0.0] [16:45:25] RECOVERY - Puppet run on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0] [16:49:45] is this channel trying to rival -operations for puppet spam? ;) [16:53:32] RECOVERY - Puppet run on integration-slave-trusty-1003 is OK: OK: Less than 1.00% above the threshold [0.0] [16:59:31] bd808: we're trying to step up our game [17:03:05] nothing better than puppet fail showers [17:04:34]

RECOVERY - Puppet run on integration-* is OK: OK: Less than 1.00% of instances failed [0.0] YAY! [17:06:13] hehe I'm also turning bots notification into NOTICE or I'll go crazy trying to tell humans and bots apart [17:23:20] (03PS3) 10Awight: Use composer in DonationInterface hhvm tests [integration/config] - 10https://gerrit.wikimedia.org/r/301025 (https://phabricator.wikimedia.org/T141309) [17:36:33] 06Release-Engineering-Team, 15User-greg: Identify "first responders" for "all" "components" deployed on Wikimedia servers - https://phabricator.wikimedia.org/T141066#2496346 (10faidon) I'm still taking this proposal in, but a few preliminary notes: "First responders" or "fixers" confuses me a little bit. A fe... [17:50:01] Hmm, could https://integration.wikimedia.org/ci/job/mwext-VisualEditor-sync-gerrit/3690/console be an indication that we no longer need this job/ Did Gerrit fix their bug? [17:50:23] RoanKattouw: I'm seeing the old entries in the DB. [17:50:32] I'm wondering if we swap them all, it'll work and not revert itself. [17:50:43] Old entries? I don't follow [17:50:52] Uno momento [17:51:59] Yeah, it looks right for master. [17:52:02] And the new wmf branch [17:52:05] But old branches no [17:52:06]