[00:22:08] RECOVERY - Puppet errors on deployment-mathoid is OK: OK: Less than 1.00% above the threshold [0.0] [00:23:18] PROBLEM - Puppet errors on deployment-kafka01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0] [00:23:56] PROBLEM - Puppet errors on deployment-apertium02 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0] [01:03:19] RECOVERY - Puppet errors on deployment-kafka01 is OK: OK: Less than 1.00% above the threshold [0.0] [01:03:57] RECOVERY - Puppet errors on deployment-apertium02 is OK: OK: Less than 1.00% above the threshold [0.0] [03:41:20] PROBLEM - Puppet errors on deployment-logstash2 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0] [04:07:53] Project selenium-MultimediaViewer » safari,beta,OS X 10.9,BrowserTests build #561: 04FAILURE in 11 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=BrowserTests/561/ [04:21:20] RECOVERY - Puppet errors on deployment-logstash2 is OK: OK: Less than 1.00% above the threshold [0.0] [04:21:33] Project selenium-MultimediaViewer » firefox,beta,Linux,BrowserTests build #561: 04FAILURE in 25 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/561/ [07:32:41] 10Release-Engineering-Team, 10MediaWiki-Platform-Team, 10Patch-For-Review, 10Performance-Team (Radar): Support multi-instance hosts on mediawiki-config - https://phabricator.wikimedia.org/T178553#3717970 (10Marostegui) Thanks @Reedy for checking this! :-) Would you be able to quickly check it on beta? (I h... [09:02:02] PROBLEM - Puppet errors on deployment-trending01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [09:45:06] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718040 (10hoo) [10:22:52] 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-General: Decide whether we want the package-lock.json to commit or ignore - https://phabricator.wikimedia.org/T179229#3718074 (10Aklapper) [10:35:59] PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0] [11:11:00] RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0] [12:33:22] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3715032 (10ema) >>! In T179156#3717847, @BBlack wrote: > For future reference by another... [13:45:02] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718221 (10BBlack) Does Echo have any kind of push notification going on, even in light... [13:46:14] Im going to see how the default sized profile image will be like [13:46:38] and possibly submit a change upstream to allow gerrit's size to reize the avatar if the plugin says too :) [13:49:47] Also this will most likly require 2.14+ [13:50:00] as i doint use buck so i carn't test for 2.13 [14:02:20] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0] [14:02:20] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718225 (10BBlack) Now that I'm digging deeper, it seems there are one or more projects... [14:42:22] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [15:34:05] PROBLEM - Free space - all mounts on deployment-kafka01 is CRITICAL: CRITICAL: deployment-prep.deployment-kafka01.diskspace.root.byte_percentfree (<100.00%) [17:25:59] 10Release-Engineering-Team (Watching / External), 10ORES, 10Operations, 10Scoring-platform-team, and 4 others: 503 spikes and resulting API slowness starting 18:45 October 26 - https://phabricator.wikimedia.org/T179156#3718297 (10Legoktm) >>! In T179156#3718221, @BBlack wrote: > Does Echo have any kind of... [18:33:19] PROBLEM - Puppet errors on deployment-conf03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:39:22] PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0] [18:52:54] PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [18:54:14] PROBLEM - Puppet errors on deployment-ores-redis-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0] [19:08:09] Project selenium-MinervaNeue » chrome,beta,Linux,BrowserTests build #179: 04FAILURE in 19 min: https://integration.wikimedia.org/ci/job/selenium-MinervaNeue/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=BrowserTests/179/ [19:13:19] RECOVERY - Puppet errors on deployment-conf03 is OK: OK: Less than 1.00% above the threshold [0.0] [19:32:54] RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0] [19:34:14] RECOVERY - Puppet errors on deployment-ores-redis-01 is OK: OK: Less than 1.00% above the threshold [0.0] [21:20:09] twentyafterfour i've managed to create the plugin. im now getting this in the logs (the plugin may not work properly yet) [21:20:11] [2017-10-29 21:16:59,461] [HTTP-67] WARN org.apache.http.client.protocol.ResponseProcessCookies : Invalid cookie header: "Set-Cookie: WMF-Last-Access=29-Oct-2017;Path=/;HttpOnly;secure;Expires=Thu, 30 Nov 2017 12:00:00 GMT". Invalid 'expires' attribute: Thu, 30 Nov 2017 12:00:00 GMT [21:28:37] woo, using the default image works [21:28:38] bazel-genfiles/plugins/phabricator-avatar/phabricator-avatar.jar [21:28:40] woop[s [21:28:41] woops [21:29:05] though it's slow, most likly because it is parsing json [21:29:06] https://phab.wmfusercontent.org/file/data/ijvxjlgmrwy7iepap5au/PHID-FILE-jfhyojz5a6selxprnxqb/user1.png [21:31:43] twentyafterfour i wonder should we setup a repo for users to add there own image? It would be quicker to parse gerrit's side, not to mention we could do this ${username} [21:32:00] so the image would be called for example Paladox.png [21:32:01] :) [21:44:17] 10Gerrit: Create avatar plugin for gerrit that uses phab's conduit to get users profile image - https://phabricator.wikimedia.org/T179212#3718433 (10Paladox) I've created the plugin now. I am currently having an issue when it fetches the users image, but using the default image i provide is fine :). Does anyone... [21:46:24] paladox: on the weekends, please just update tasks with findings (summaries there of). Pinging people on the weekend should be reserved for timely needs, not just FYIs. [21:46:39] ok sorry. [22:15:07] 10Gerrit: Create avatar plugin for gerrit that uses phab's conduit to get users profile image - https://phabricator.wikimedia.org/T179212#3718466 (10demon) Who discussed this? When? What if a Gerrit user doesn't have a Phab profile (or if it's not connected. or has multiple)? What if they don't want the same ava... [22:15:25] 10Gerrit: Create avatar plugin for gerrit that uses phab's conduit to get users profile image - https://phabricator.wikimedia.org/T179212#3718467 (10demon) Also: please publish the code somewhere. [22:20:12] 10Gerrit: Create avatar plugin for gerrit that uses phab's conduit to get users profile image - https://phabricator.wikimedia.org/T179212#3718473 (10Paladox) >>! In T179212#3718466, @demon wrote: > Who discussed this? When? What if a Gerrit user doesn't have a Phab profile (or if it's not connected. or has multi... [22:22:50] PROBLEM - Puppet errors on deployment-urldownloader is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0] [22:28:51] 10Gerrit: Create avatar plugin for gerrit that uses phab's conduit to get users profile image - https://phabricator.wikimedia.org/T179212#3718474 (10Paladox) This is what it looks like {F10506923} {F10506926} [23:02:49] RECOVERY - Puppet errors on deployment-urldownloader is OK: OK: Less than 1.00% above the threshold [0.0]