[00:17:52] Reedy: on Test Wikipedia I set up a simple JSON-model page with the value as { "hey": "alert(1)" }, and then I wrote a vector.js script in my user space to load that alert(1), and it doesn't execute, so I take that to be a good sign. [00:18:01] https://test.wikipedia.org/w/index.php?title=User:Harej/vector.js&oldid=323450 [00:18:28] wrap it in eval()? :P [00:18:46] That would definitely do the trick, but I'm not actually eval-ing anything in practice, so. [00:18:54] Any other pitfalls to watch out for? [00:19:21] (Can non-admins change content model yet???) [00:35:20] PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [00:38:30] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0] [00:48:30] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:02:42] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [01:10:20] RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0] [01:12:22] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [01:15:25] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [01:23:32] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [01:40:25] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [01:52:21] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:02:43] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [02:05:41] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [02:08:31] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [02:10:17] (03PS1) 10MaxSem: Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) [02:10:40] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [02:11:59] (03PS2) 10MaxSem: Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) [02:13:20] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [02:14:30] (03CR) 10jerkins-bot: [V: 04-1] Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) (owner: 10MaxSem) [02:35:37] RECOVERY - Puppet errors on deployment-mx is OK: OK: Less than 1.00% above the threshold [0.0] [02:40:40] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [02:53:22] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [03:08:03] PROBLEM - Puppet errors on deployment-aqs02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [03:18:27] PROBLEM - Puppet errors on deployment-poolcounter04 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [03:48:02] RECOVERY - Puppet errors on deployment-aqs02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:48:29] RECOVERY - Puppet errors on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:06:00] PROBLEM - Puppet errors on deployment-memc04 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [04:36:03] RECOVERY - Puppet errors on deployment-memc04 is OK: OK: Less than 1.00% above the threshold [0.0] [04:39:12] 10Release-Engineering-Team (Kanban), 10Wikimedia-Extension-setup, 10User-notice: Enable Collection by default on most sites - https://phabricator.wikimedia.org/T73416#3438383 (10Liuxinyu970226) [04:41:34] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0] [04:51:36] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [04:51:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [05:14:21] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [05:21:42] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [05:25:41] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [05:35:41] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [05:49:25] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:04:34] PROBLEM - Puppet errors on deployment-tmh01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [06:32:44] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [06:39:33] RECOVERY - Puppet errors on deployment-tmh01 is OK: OK: Less than 1.00% above the threshold [0.0] [06:52:43] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [06:53:05] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [07:01:24] RECOVERY - Puppet staleness on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [3600.0] [07:02:42] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [07:04:11] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [07:42:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [07:46:35] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [07:56:58] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [07:59:12] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:21:37] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [08:31:59] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [08:39:29] PROBLEM - Puppet errors on deployment-kafka04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [09:19:29] RECOVERY - Puppet errors on deployment-kafka04 is OK: OK: Less than 1.00% above the threshold [0.0] [09:38:50] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [09:44:40] 10Release-Engineering-Team (Kanban), 10CirrusSearch, 10Discovery, 10Discovery-Search, and 2 others: Figure out why browser tests can't create suggestion box - https://phabricator.wikimedia.org/T162966#3438668 (10zeljkofilipin) For upgrading to Chrome 59 see {T170032}. [10:21:42] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [10:23:45] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [10:28:54] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [10:38:54] PROBLEM - Puppet errors on deployment-cache-upload04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [10:41:42] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [10:46:27] PROBLEM - Puppet errors on deployment-elastic07 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [10:52:42] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [11:03:55] RECOVERY - Puppet errors on deployment-cache-upload04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:23:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [11:41:11] RECOVERY - Puppet errors on deployment-imagescaler01 is OK: OK: Less than 1.00% above the threshold [0.0] [11:42:46] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [11:51:25] RECOVERY - Puppet errors on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0] [11:54:44] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [11:59:03] PROBLEM - Puppet errors on deployment-memc05 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [12:01:05] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [12:02:09] PROBLEM - Puppet errors on deployment-imagescaler01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [12:36:03] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [12:36:41] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [12:38:17] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [12:44:44] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [12:55:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [13:04:02] RECOVERY - Puppet errors on deployment-memc05 is OK: OK: Less than 1.00% above the threshold [0.0] [13:19:44] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:26:39] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [13:35:48] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [13:38:18] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [13:39:52] PROBLEM - Puppet errors on deployment-ms-fe02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [13:46:39] Yippee, build fixed! [13:46:40] Project selenium-VisualEditor » firefox,beta,Linux,BrowserTests build #459: 09FIXED in 2 min 38 sec: https://integration.wikimedia.org/ci/job/selenium-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/459/ [14:03:40] PROBLEM - Puppet errors on deployment-tin is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [14:23:26] PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [14:24:44] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:24:52] RECOVERY - Puppet errors on deployment-ms-fe02 is OK: OK: Less than 1.00% above the threshold [0.0] [14:29:43] PROBLEM - Puppet errors on deployment-puppetdb01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [14:33:42] Project selenium-WikiLove » firefox,beta,Linux,BrowserTests build #454: 04FAILURE in 1 min 41 sec: https://integration.wikimedia.org/ci/job/selenium-WikiLove/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/454/ [14:44:42] RECOVERY - Puppet errors on deployment-puppetdb01 is OK: OK: Less than 1.00% above the threshold [0.0] [14:58:42] RECOVERY - Puppet errors on deployment-tin is OK: OK: Less than 1.00% above the threshold [0.0] [15:06:00] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [15:25:44] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [15:26:00] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [15:33:05] PROBLEM - Puppet errors on integration-slave-docker-1000 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [15:40:42] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [16:13:10] PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:15:21] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [16:20:34] PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [16:25:51] PROBLEM - Puppet errors on deployment-ircd is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:42:44] RainbowSprinkles gerrit stable-2.14 has had 90 more fixes since 2.14.1 was released heh. That is alot :) [16:45:05] i also added syntax highlighting for php, erb and puppet for 2.14.2 in polygerrit. I like the syntax highlighting of <% @ %> [16:45:24] i had the red highlighting for the php tags though makes it look like a syntax error. [16:47:35] PROBLEM - Puppet errors on deployment-restbase01 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0] [16:48:46] 10Gerrit, 10Release-Engineering-Team (Kanban), 10Regression, 10Upstream: Cannot log into Gerrit as of recent upgrade - https://phabricator.wikimedia.org/T152640#3439598 (10Paladox) [16:48:50] 10Gerrit, 10Release-Engineering-Team (Backlog), 10Patch-For-Review: Update gerrit to 2.14.1 - https://phabricator.wikimedia.org/T156120#3439597 (10Paladox) [16:50:23] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:06:33] RainbowSprinkles do we want to give gerrit2 full sudo rights? [17:06:41] i see scap makes gerrit2 a sudo user [17:06:44] -r--r----- 1 root root 71 Jul 7 00:16 scap_gerrit2 [17:06:48] No why on earth would it need full sudo? [17:06:55] We doint [17:06:59] scap did that [17:08:07] scap::target should only grant the deploy user the ability to restart the service in sudoers [17:08:27] hmm [17:08:28] i see [17:08:29] # This file is managed by Puppet! [17:08:29] gerrit2 ALL=(gerrit2) NOPASSWD: ALL [17:10:52] RECOVERY - Puppet errors on deployment-ircd is OK: OK: Less than 1.00% above the threshold [0.0] [17:11:05] i see it for gerrit-migrate which i prevously picked for the scap user there too [17:12:36] RECOVERY - Puppet errors on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [0.0] [17:15:07] that's not full sudo rights. the stuff inside () specifies users and groups [17:15:35] oh [17:15:57] ah i see [17:16:03] thanks [17:42:07] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [18:07:00] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [18:12:00] (03PS3) 10MaxSem: Restore ArticleCreationWorkflow [integration/config] - 10https://gerrit.wikimedia.org/r/365198 (https://phabricator.wikimedia.org/T170354) [18:12:04] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [18:20:04] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [18:33:55] 10Release-Engineering-Team: jsduck custom tags error when trying to run docs for a skin - https://phabricator.wikimedia.org/T170703#3439898 (10Jdlrobson) [18:35:05] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [18:42:02] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [18:46:23] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [18:57:05] 10Continuous-Integration-Config, 10Release-Engineering-Team: jsduck custom tags error when trying to run docs for a skin - https://phabricator.wikimedia.org/T170703#3439950 (10greg) [19:01:59] PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [19:04:18] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:21:21] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:25:08] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #9: 04FAILURE in 36 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/9/ [19:29:18] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0] [19:35:13] Project selenium-MinervaNeue » firefox,beta,Linux,BrowserTests build #9: 04FAILURE in 46 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/9/ [19:42:00] RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0] [19:42:22] PROBLEM - Puppet errors on integration-puppetmaster01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [19:51:04] PROBLEM - Puppet errors on deployment-mediawiki05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0] [19:53:42] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [20:16:02] !log cpan[1]> install LWP::UserAgent on tin [20:16:07] Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL [20:22:22] RECOVERY - Puppet errors on integration-puppetmaster01 is OK: OK: Less than 1.00% above the threshold [0.0] [20:31:07] RECOVERY - Puppet errors on deployment-mediawiki05 is OK: OK: Less than 1.00% above the threshold [0.0] [20:33:05] PROBLEM - Puppet errors on deployment-db03 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [20:41:49] Yippee, build fixed! [20:41:50] Project selenium-Echo » chrome,beta,Linux,BrowserTests build #455: 09FIXED in 48 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/455/ [20:41:50] Yippee, build fixed! [20:41:51] Project selenium-Echo » firefox,beta,Linux,BrowserTests build #455: 09FIXED in 48 sec: https://integration.wikimedia.org/ci/job/selenium-Echo/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/455/ [20:42:59] PROBLEM - Puppet errors on deployment-mediawiki06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [20:48:42] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [20:52:41] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [21:03:05] RECOVERY - Puppet errors on deployment-db03 is OK: OK: Less than 1.00% above the threshold [0.0] [21:07:04] TabbyCat: if you didn't see, everything should be cleaned up now :) [21:07:22] awesome harej many many thanks [21:07:25] I should probably write a post-mortem for the Phabricator ticket [21:07:40] I'll check after I clean the unblock requests backlog [21:07:48] that's easier than to fix those pages [21:07:58] just add "declined" :P [21:09:03] The script that was run is really problematic if a given page_title appears in both namespace n and namespace n+1 [21:09:07] i.e. a namespace and its talk [21:09:22] essentially it "fixes" one but leaves the others to languish in obscurity [21:09:55] There may have been maybe... 2 or 3 pages? That I couldn't fix. But I am not particularly worried about them. [21:11:34] that script is really old, maybe it is not thought to work with our current wiki versions? [21:11:55] i keep thinking that maybe those scripts should leave on-wiki traces, such as move log entries [21:17:58] RECOVERY - Puppet errors on deployment-mediawiki06 is OK: OK: Less than 1.00% above the threshold [0.0] [21:19:45] PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [21:26:46] PROBLEM - Puppet errors on deployment-jobrunner02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0] [21:42:38] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [21:50:46] PROBLEM - Puppet errors on deployment-fluorine02 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [21:59:43] RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0] [22:02:58] hey greg-g'ers fyi we did drop the change rate for nodepool a yesterday (?) when it was hammering us pretty good and it hasn't been raised yet, change aversion on friday mainly [22:03:01] just fyi [22:03:41] word [22:03:42] PROBLEM - Puppet errors on deployment-elastic06 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [22:06:47] RECOVERY - Puppet errors on deployment-jobrunner02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:08:01] PROBLEM - Puppet errors on deployment-stream is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [22:18:00] RECOVERY - Puppet errors on deployment-stream is OK: OK: Less than 1.00% above the threshold [0.0] [22:23:39] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:25:43] RECOVERY - Puppet errors on deployment-fluorine02 is OK: OK: Less than 1.00% above the threshold [0.0] [22:58:39] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [23:03:42] RECOVERY - Puppet errors on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0] [23:10:18] PROBLEM - Puppet errors on deployment-sentry01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [23:17:32] greg-g RainbowSprinkles thcipriani: hey.. just giving you a heads up, when we've sorted the current train I'd like to talk about the next train with regards to the Minerva skin as not keen not to break production. Can jump in a hangout or chat on IRC Monday. [23:19:46] 10Continuous-Integration-Config, 10VisualEditor, 10User-Ryasmeen: Intermittent failures of mwext-qunit-jessie - https://phabricator.wikimedia.org/T163123#3440757 (10Jdforrester-WMF) [23:24:41] PROBLEM - Puppet errors on deployment-secureredirexperiment is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [23:25:05] jdlrobson: So the train for next week, the option that is easiest conceptually...is structure master (between now and Tues) in a way that allows the two to peacefully coexist. [23:25:59] Then we'd branch like normal [23:26:18] RainbowSprinkles: so wmf9 is golden. wmf7 was the problematic one [23:26:23] I'm on train duty next week [23:26:38] So are masters ok too? [23:26:40] So we can enable MinervaNeue skin and then merge https://gerrit.wikimedia.org/r/#/c/365418/ [23:27:18] thcipriani: do we want to do wmf.9 for group 2 on Monday? [23:27:27] MinervaNeue when enabled should peacefully coexist with MobileFrontend without that patch [23:27:51] all wikis on Monday was my plan [23:27:51] Let's make sure nothing lands in master that breaks that until we're live 😊 [23:27:59] thcipriani: ack [23:28:51] So brief plan: all wikis to wmf.9 on Monday, we'll retry enabling after [23:28:53] 10Release-Engineering-Team, 10MinervaNeue: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3440783 (10Jdlrobson) [23:29:04] Then if that's good, wmf10 should be easy [23:29:27] Is the current wmf.9 what we want when enabling? Or are there further back ports? [23:31:54] 10Release-Engineering-Team, 10MinervaNeue: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3440788 (10Jdlrobson) Interestingly these PHP unit tests fail locally for me when not using hhvm. I can only assume that when RelatedArticles phpuni... [23:34:07] 10Release-Engineering-Team, 10MinervaNeue: RelatedPages PHP unit tests are failing during to an issue with MinervaNeue - https://phabricator.wikimedia.org/T170624#3440791 (10pmiazga) @Jdlrobson: I can check it on Monday [23:34:41] RECOVERY - Puppet errors on deployment-secureredirexperiment is OK: OK: Less than 1.00% above the threshold [0.0] [23:40:11] 10Release-Engineering-Team (Kanban), 10Phabricator (2016-10-05), 10Regression, 10Upstream: Regression: Ability to comment on phame posts not working - https://phabricator.wikimedia.org/T144338#3440797 (10Paladox) Confirmed that it regressed. I cannot comment on any blog posts now. I forgot what @mmodell d... [23:46:01] RainbowSprinkles: no further back ports that i know of [23:46:08] we'll know quickly in logstash if there are [23:46:17] So what's in git matches what's in prod now? [23:51:45] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [23:55:19] RECOVERY - Puppet errors on deployment-sentry01 is OK: OK: Less than 1.00% above the threshold [0.0]