[00:35:03] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[00:35:15] <tgr>	 greg-g: I ran out of steam, filed T185055
[00:35:15] <stashbot>	 T185055: Stack overflow when Redis is down - https://phabricator.wikimedia.org/T185055
[00:35:26] <tgr>	 but basically, MediaWiki can't connect to Redis
[00:38:41] <greg-g>	 tgr: thoughts on who to rope in? ctrl+f redis on https://www.mediawiki.org/wiki/Developers/Maintainers  doesn't help :)
[00:40:05] <tgr>	 greg-g: it's a kind of object cache, so AaronSchulz is the official maintainer
[00:40:16] <tgr>	 then again, the error is probably not in MediaWiki
[00:41:47] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3832934 (10EddieGP) From the Deployments page right now ( https://wikitech.wikimedia.org/w/index.php?title=Deployments&oldid=1780713 ), I...
[00:44:22] <tgr>	 I can connect to the redis instance locally but not from the appservers, so chances are nutcracker has some issue, no clue who is familiar with that
[00:44:32] <tgr>	 I guess I should at least try restarting
[00:45:31] <tgr>	 (nope)
[00:46:25] <greg-g>	 :(
[00:50:46] <tgr>	 in any case, it would be someone from ops
[00:51:11] <tgr>	 or maybe performance? when ori was here, he was the best person to ask
[00:51:48] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3905018 (10greg) Services always had that window, nothing changed there. Should we address it? Probably. It hasn't been an issue yet, as s...
[02:42:05] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator: Test phabricator translations on phab.wmflabs.org - https://phabricator.wikimedia.org/T184459#3905179 (10mmodell) I've tested this locally and it's working well.
[02:42:38] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator: Test phabricator translations on phab.wmflabs.org - https://phabricator.wikimedia.org/T184459#3905180 (10mmodell)
[04:22:11] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator: Test phabricator translations on phab.wmflabs.org - https://phabricator.wikimedia.org/T184459#3905229 (10mmodell) Currently the "limited" translations are hidden except when phabricator is in developer mode. This needs to be changed to make them usable.
[05:00:39] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<55.56%)
[06:55:39] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:01:29] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Language-2017-Oct-Dec, 10Language-2018-Jan-Mar: [betacluster] Special:Translate does not allow to translate article content - multiple "Uncaught SyntaxError: Unexpected token < in JSON at position 0" - https://phabricator.wikimedia.org/T180841#3771141 (10KartikMistry) Thanks...
[07:02:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10MediaWiki-extensions-Translate: [betacluster] "Uncaught TypeError: Cannot read property 'changeSettings' of null" when clicking on any option of row tux-message-selector - https://phabricator.wikimedia.org/T185038#3905362 (10KartikMistry)
[07:32:23] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Wikimedia-Incident: Address proximity of service deployments to train deployments problem - https://phabricator.wikimedia.org/T182733#3905381 (10demon) Actually Wednesday is the day most likely to go sideways. This bug is poorly titled I think. I believe the main motivati...
[09:36:58] <wikibugs>	 (03PS1) 10Hashar: Migrate unicodejs publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404643
[09:37:19] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate unicodejs publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404643 (owner: 10Hashar)
[09:39:13] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate unicodejs publish jobs to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404643 (owner: 10Hashar)
[09:39:36] <wikibugs>	 (03PS1) 10Hashar: Migrate unicodejs-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404644
[09:44:10] <wikibugs>	 (03PS2) 10Hashar: Migrate unicodejs-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404644
[09:44:40] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate unicodejs-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404644 (owner: 10Hashar)
[09:46:03] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate unicodejs-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404644 (owner: 10Hashar)
[10:01:10] <wikibugs>	 (03PS1) 10Hashar: Migrate oojs-core-publish oojs-core-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404645
[10:09:01] <wikibugs>	 (03PS2) 10Hashar: Migrate oojs-core-publish oojs-core-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404645
[10:09:03] <wikibugs>	 (03PS1) 10Hashar: Migrate visualeditor-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404646
[10:09:41] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate oojs-core-publish oojs-core-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404645 (owner: 10Hashar)
[10:09:47] <wikibugs>	 (03CR) 10Hashar: [C: 032] Migrate visualeditor-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404646 (owner: 10Hashar)
[10:11:36] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate oojs-core-publish oojs-core-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404645 (owner: 10Hashar)
[10:11:38] <wikibugs>	 (03Merged) 10jenkins-bot: Migrate visualeditor-coverage to Docker [integration/config] - 10https://gerrit.wikimedia.org/r/404646 (owner: 10Hashar)
[11:13:26] <wikibugs>	 10Beta-Cluster-Infrastructure: PHP fatal error: Stack overflow - https://phabricator.wikimedia.org/T185088#3905736 (10zhuyifei1999)
[11:43:51] <wikibugs>	 10Release-Engineering-Team (Kanban), 10releng-201718-q1, 10MediaWiki-General-or-Unknown, 10Epic, and 5 others: Port Selenium tests from Ruby to Node.js - https://phabricator.wikimedia.org/T139740#3905804 (10zeljkofilipin)
[11:43:55] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Cache, 10MediaWiki-Vagrant, 10Performance-Team (Radar), 10User-zeljkofilipin: MediaWiki core Selenium tests fail when targeting Vagrant - https://phabricator.wikimedia.org/T180035#3905802 (10zeljkofilipin) 05Resolved>03Open This is happening again. I c...
[12:21:14] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#3905855 (10zeljkofilipin)
[12:21:37] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#3905865 (10zeljkofilipin) p:05Triage>03Low
[12:21:51] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin: Update page object pattern in Selenium tests - https://phabricator.wikimedia.org/T185094#3905855 (10zeljkofilipin)
[12:21:53] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MW-1.31-release-notes (WMF-deploy-2018-01-16 (1.31.0-wmf.17)), 10Patch-For-Review, 10User-zeljkofilipin: Q3 Selenium framework improvements - https://phabricator.wikimedia.org/T182421#3905867 (10zeljkofilipin)
[13:02:16] <zeljkof>	 addshore: apologies for the late reviews, I have left a few comments on your selenium tests for core
[13:02:32] <zeljkof>	 looks good in general, but I would recommend a few improvements (in comments)
[13:09:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10HHVM: HHVM 3.18.5+dfsg-1+wmf3 changes parse_url causing unit tests to fail - https://phabricator.wikimedia.org/T185024#3905953 (10hashar)
[13:10:59] <hashar>	 !log nodepool: updating snapshot to get hhvm +wmf4 for T185024  : nodepool image-update wmflabs-eqiad snapshot-ci-jessie
[13:11:05] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[13:11:06] <stashbot>	 T185024: HHVM 3.18.5+dfsg-1+wmf3 changes parse_url causing unit tests to fail - https://phabricator.wikimedia.org/T185024
[13:32:25] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Operations, 10HHVM: HHVM 3.18.5+dfsg-1+wmf3 changes parse_url causing unit tests to fail - https://phabricator.wikimedia.org/T185024#3906004 (10MoritzMuehlenhoff) I've built/uploaded new HHVM packages for jessie (stretch following soon) which disable the broken patc...
[14:05:13] <dcausse>	 zeljkof: so apprently the broser spotted one issue on the beta cluster
[14:05:21] <dcausse>	 s/broser/browser test/
[14:05:30] <zeljkof>	 yeah! :D
[14:05:46] <dcausse>	 how can I debug this kind of error: https://en.wikipedia.beta.wmflabs.org/w/api.php?action=opensearch&format=json&formatversion=2&search=main%20p&namespace=0&limit=10&suggest=true
[14:05:48] <dcausse>	 ?
[14:08:42] <dcausse>	 all apis seem broken on the beta cluster 
[14:09:47] <dcausse>	 I cannot even access ApiSandbox: An error occurred while loading information for API module "main": HTTP error:
[14:14:09] <dcausse>	 PHP fatal error: Stack overflow
[14:15:46] <dcausse>	 already reported in T185088 apparently
[14:15:47] <stashbot>	 T185088: PHP fatal error: Stack overflow - https://phabricator.wikimedia.org/T185088
[14:33:40] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<11.11%)
[14:45:26] <stephanebisson>	 I suddenly have tons of ExternalStore errors in mw-vagrant "Error: 1146 Table 'wiki.blobs' doesn't exist". It happens during provisioning but also when using features that depend on externalstore. Anybody has a clue about why this is happening?
[15:15:05] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Someday), 10Patch-For-Review: Update gerrit to 2.14.7 - https://phabricator.wikimedia.org/T156120#3906216 (10Paladox) This can be done now :).  Zuul has been updated to include the fix (deployed to prod)
[15:27:25] <wikibugs>	 (03CR) 10Zfilipin: [C: 032] Create selenium-CirrusSearch-jessie Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/398030 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin)
[15:27:45] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:28:39] <wikibugs>	 (03Merged) 10jenkins-bot: Create selenium-CirrusSearch-jessie Jenkins job [integration/config] - 10https://gerrit.wikimedia.org/r/398030 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin)
[15:29:33] <wikibugs>	 (03CR) 10Zfilipin: "selenium-CirrusSearch-jessie job deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/398030 (https://phabricator.wikimedia.org/T175179) (owner: 10Zfilipin)
[15:33:42] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:40:25] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10User-zeljkofilipin: Run Selenium tests in CI for extensions - https://phabricator.wikimedia.org/T164721#3906274 (10zeljkofilipin)
[15:40:29] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Discovery, 10Discovery-Search (Current work), 10MW-1.31-release-notes (WMF-deploy-2018-01-02 (1.31.0-wmf.15)), and 2 others: Create selenium-CirrusSearch-jessie daily Jenkins job - https://phabricator.wikimedia.org/T175179#3906270 (10zeljkofilipin) 05Open>03Resolve...
[15:58:50] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3906336 (10zeljkofilipin) Special:RecentChanges is broken on a clean mediawiki-vagrant VM wit...
[16:00:59] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3901120 (10awight) I think E:ORES should handle the corner case more gracefully.  Anyway, thi...
[16:11:11] <wikibugs>	 (03PS1) 10Pablo Grass (WMDE): Add selenium to AdvancedSearch experimental [integration/config] - 10https://gerrit.wikimedia.org/r/404702 (https://phabricator.wikimedia.org/T184608)
[16:17:56] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-extensions-ORES, 10ORES, 10Scoring-platform-team, 10User-zeljkofilipin: Special:RecentChanges broken on Jenkins slaves - https://phabricator.wikimedia.org/T184938#3906386 (10zeljkofilipin) Creating a page (via API) did not fix the problem ([[ https://gerri...
[16:18:41] <shinken-wm>	 RECOVERY - Free space - all mounts on integration-slave-jessie-1003 is OK: OK: integration.integration-slave-jessie-1003.diskspace.root.byte_percentfree (More than half of the datapoints are undefined) integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (More than half of the datapoints are undefined)
[16:22:15] <wikibugs>	 (03CR) 10Zfilipin: [C: 032] Add selenium to AdvancedSearch experimental [integration/config] - 10https://gerrit.wikimedia.org/r/404702 (https://phabricator.wikimedia.org/T184608) (owner: 10Pablo Grass (WMDE))
[16:23:22] <wikibugs>	 (03Merged) 10jenkins-bot: Add selenium to AdvancedSearch experimental [integration/config] - 10https://gerrit.wikimedia.org/r/404702 (https://phabricator.wikimedia.org/T184608) (owner: 10Pablo Grass (WMDE))
[16:24:30] <zeljkof>	 !log Reloading Zuul to deploy 5f757310f499a6a2cdf036dde3d258046377186f
[16:24:36] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[16:24:40] <shinken-wm>	 PROBLEM - Free space - all mounts on integration-slave-jessie-1003 is CRITICAL: CRITICAL: integration.integration-slave-jessie-1003.diskspace._srv.byte_percentfree (<100.00%)
[16:24:49] <wikibugs>	 (03CR) 10Zfilipin: "The change is deployed." [integration/config] - 10https://gerrit.wikimedia.org/r/404702 (https://phabricator.wikimedia.org/T184608) (owner: 10Pablo Grass (WMDE))
[16:36:07] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Scap, 10Patch-For-Review: Scap: TypeError: error: (not all arguments converted during string formatting); format string: (Passed unrecognized git_binary_manager {}); arguments: ((u'fat',)) - https://phabricator.wikimedia.org/T184882#3906442 (10thcipriani) p:05High>03...
[16:40:48] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T180749#3906458 (10thcipriani)
[16:41:07] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T183956#3906459 (10thcipriani) a:03thcipriani
[16:41:41] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release, 10Train Deployments: 1.31.0-wmf.17 deployment blockers - https://phabricator.wikimedia.org/T183956#3906462 (10thcipriani) p:05Triage>03Normal
[16:45:52] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10User-zeljkofilipin, 10Wikimedia-Incident: Create selenium-core-jessie daily Jenkins job - https://phabricator.wikimedia.org/T185011#3906484 (10greg)
[17:02:01] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-SendGrid: PHPUnit runner does not load required dependencies from composer.json (SendGrid extension) - https://phabricator.wikimedia.org/T185115#3906552 (10Reedy)
[17:06:47] <Sagan>	 now that the reboots are done, can somebody take a look at beta?
[17:07:06] <Sagan>	 I get this all the time:
[17:07:07] <Sagan>	 There seems to be a problem with your login session; this action has been canceled as a precaution against session hijacking. Go back to the previous page, reload that page and then try again. 
[17:07:15] <Sagan>	 and API returns always "NeedToken"
[17:09:57] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10Release-Engineering-Team (Kanban), 10releng-201718-q3: Migrate PHPUnit Mediawiki jobs from Nodepool to Docker - https://phabricator.wikimedia.org/T183512#3906565 (10greg) p:05Triage>03Normal
[17:13:43] <greg-g>	 Sagan: https://phabricator.wikimedia.org/T185055
[17:14:23] <Sagan>	 greg-g: ah, ok, thx
[17:21:56] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Zuul: Update zuul to upstream master - https://phabricator.wikimedia.org/T158243#3906601 (10greg) p:05High>03Normal Lowering priority to Normal as we now unblocked the gerrit upgrade.
[17:24:08] <shinken-wm>	 PROBLEM - English Wikipedia Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 503 Service Unavailable - string 'Wikipedia' not found on 'https://en.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 2219 bytes in 0.244 second response time
[17:24:23] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki04 is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'http://en.wikipedia.beta.wmflabs.org:80/wiki/Main_Page?debug=true' - 1861 bytes in 0.278 second response time
[17:24:47] <shinken-wm>	 PROBLEM - English Wikipedia Mobile Main page on beta-cluster is CRITICAL: HTTP CRITICAL: HTTP/1.1 500 Internal Server Error - string 'Wikipedia' not found on 'https://en.m.wikipedia.beta.wmflabs.org:443/wiki/Main_Page?debug=true' - 2445 bytes in 0.310 second response time
[17:26:58] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-SendGrid: PHPUnit runner does not load required dependencies from composer.json (SendGrid extension) - https://phabricator.wikimedia.org/T185115#3906648 (10Umherirrender) Change from `extension-unittests-generic` to `extension-unittests-composer` in test...
[17:36:45] <paladox>	 hmm https://en.wikipedia.beta.wmflabs.org/wiki/Main_Page?debug=true seems now.
[17:36:54] <paladox>	 seems now = seems down now.
[17:37:00] <paladox>	 Possibly this https://phabricator.wikimedia.org/T185055
[17:50:26] <wikibugs>	 (03CR) 10Umherirrender: "There is no mediawiki/extensions/WikiAdmin" [integration/config] - 10https://gerrit.wikimedia.org/r/404401 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42)
[17:53:20] <addshore>	 thcipriani: how was the train last night?
[17:53:23] <ottomata>	 did I break https://deployment.wikimedia.beta.wmflabs.org/? or was it already broken?
[17:53:26] <addshore>	 I vanished after group1
[17:53:43] <ottomata>	 i just applied some varnishkafka changes there
[17:53:51] <ottomata>	 on the 2 deployment-cache hosts
[17:53:57] <ottomata>	 but, it seems varnish is doing fine...?
[17:59:27] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3906777 (10demon) 05Open>03Resolved a:03Paladox
[18:02:03] <wikibugs>	 10Gerrit, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata: Clean up mediawiki/extensions/Wikibase ACL - https://phabricator.wikimedia.org/T183081#3842843 (10demon) The wikidata permissions are mostly redundant, considering they're all in the extension-Wikibase group.
[18:03:45] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[18:06:45] <greg-g>	 ottomata: sadly https://phabricator.wikimedia.org/T185055
[18:08:30] <ottomata>	 Oook wusn't me thank you :)
[18:14:20] <wikibugs>	 (03PS1) 10Phantom42: Use extension-unittests-composer for SendGrid extension testing [integration/config] - 10https://gerrit.wikimedia.org/r/404748 (https://phabricator.wikimedia.org/T185115)
[18:15:44] <wikibugs>	 10Continuous-Integration-Config, 10MediaWiki-extensions-SendGrid, 10Patch-For-Review: PHPUnit runner does not load required dependencies from composer.json (SendGrid extension) - https://phabricator.wikimedia.org/T185115#3906834 (10Phantom42) > Change from `extension-unittests-generic` to `extension-unittest...
[18:29:18] <wikibugs>	 (03PS1) 10Chad: WIP: Create docker image for linting refs/meta/config [integration/config] - 10https://gerrit.wikimedia.org/r/404753
[18:29:40] <no_justification>	 thcipriani: Whee, probably totally doesn't work ^
[18:29:52] <thcipriani>	 nice :)
[18:31:23] <wikibugs>	 (03CR) 10Chad: "Should actually check out proper ref, unless Zuul already does that? Will it even work with refs/meta/config?" [integration/config] - 10https://gerrit.wikimedia.org/r/404753 (owner: 10Chad)
[18:32:43] <wmf-insecte>	 Project mwext-phpunit-coverage-publish build #125: 04FAILURE in 42 min: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/125/
[18:37:27] <wikibugs>	 10Gerrit, 10Release-Engineering-Team, 10Patch-For-Review: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3906897 (10Krinkle) 05Resolved>03Open Re-opening as the file is currently not compliant with the license terms of the photo. The choice for this photo was presumably inspired...
[18:37:39] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3906899 (10Krinkle) p:05Triage>03High
[18:39:22] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[18:41:45] <addshore>	 thcipriani: fyi https://gerrit.wikimedia.org/r/#/c/404727/
[18:42:38] <thcipriani>	 addshore: oh good :)
[18:44:51] <thcipriani>	 addshore: oh, just realized I missed you question about train yesterday! It went fine after applying your patch, nothing huge is wrong afaict; although, evidently, you've found something
[18:45:50] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3906994 (10Paladox) @Krinkle oh, how do we do attribution to Adrian Dreßler please?
[18:47:13] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3895960 (10greg) From the "use this file" button on the file page on commons:   ``` <a title="By Adrian Dreßler [CC BY 2.0 (http://creativecommons.org/licenses/by/2.0)], via Wikimedia Commons" href="ht...
[18:47:51] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3907002 (10greg) Better:  ``` By Adrian Dreßler [<a href="http://creativecommons.org/licenses/by/2.0">CC BY 2.0</a>], <a href="https://commons.wikimedia.org/wiki/File%3ASea_and_sky.jpg">via Wikimedia C...
[18:51:36] <addshore>	 thcipriani: yup, thats the only thing afaik
[18:51:40] * addshore does a quick once over
[18:53:13] <addshore>	 thcipriani: https://gerrit.wikimedia.org/r/#/c/404758/ is the CP of that patch to .17
[18:53:43] <addshore>	 also thcipriani https://gerrit.wikimedia.org/r/#/c/404757/ is a CP of a patch from .16 to .17 that is not currently merged on master.master patch is https://gerrit.wikimedia.org/r/#/c/403422
[18:58:16] <wikibugs>	 (03CR) 10Phantom42: "Hm, thank you! I will take a look and see what options we have to fix the problem" [integration/config] - 10https://gerrit.wikimedia.org/r/404401 (https://phabricator.wikimedia.org/T175794) (owner: 10Phantom42)
[18:59:05] <thcipriani>	 addshore: I think that 2nd one was actually done yesterday for wmf.17
[18:59:08] * thcipriani double checks
[19:01:04] <thcipriani>	 huh, I guess not.
[19:01:33] <wmf-insecte>	 Yippee, build fixed!
[19:01:33] <wmf-insecte>	 Project mwext-phpunit-coverage-publish build #126: 09FIXED in 28 min: https://integration.wikimedia.org/ci/job/mwext-phpunit-coverage-publish/126/
[19:01:35] <thcipriani>	 I'll get those done now.
[19:01:37] <addshore>	 thcipriani: i didn't spot it on the branch :)
[19:01:39] <addshore>	 =]
[19:02:16] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3907063 (10Paladox) We will need to use javascript to inline this.  As we are not expecting javascript to auto add any html inlineHTML will be safe for us.  I just need to figure out where to add this...
[19:03:34] <thcipriani>	 hrm, or I will when zuul gets a move on.
[19:06:29] <wikibugs>	 10Gerrit, 10Release-Engineering-Team: Customize Gerrit's login UI - https://phabricator.wikimedia.org/T184778#3907068 (10Krinkle) You might want to consider using CSS `"content"` property in combination with a `:before` or `:after` selector to display text on the page without adding additional HTML elements. T...
[19:10:46] <paladox>	 Krinkle thankyou
[19:10:51] <wikibugs>	 (03Abandoned) 10Zoranzoki21: Run phan for CologneBlue and MonoBook skins [integration/config] - 10https://gerrit.wikimedia.org/r/404172 (https://phabricator.wikimedia.org/T179554) (owner: 10Zoranzoki21)
[19:13:15] <thcipriani>	 addshore: hrm, getting some phpunit error on https://gerrit.wikimedia.org/r/#/c/404757/
[19:14:30] <addshore>	 blergh, just eating now, will check in a sec
[19:14:40] <thcipriani>	 thanks
[19:16:36] <wikibugs>	 (03PS1) 10Umherirrender: Remove unneeded closing declaration comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/404767
[19:20:24] <paladox>	 no_justification the gelf logstash plugin supports tcp now. So it is now easier for me to test in mwv :).
[19:25:41] <wikibugs>	 (03CR) 10Zoranzoki21: [C: 031] Remove unneeded closing declaration comments [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/404767 (owner: 10Umherirrender)
[19:27:43] <tgr>	 stephanebisson: did you do a git pull on MediaWiki but not run update.php?
[19:27:56] <addshore>	 thcipriani: blergh, not sure what is happening
[19:28:26] <tgr>	 otherwise, might be a bug with the update hooks for the recent MCR patch, you should tell MW-Platform about it
[19:30:35] <stephanebisson>	 tgr: I believe I'm all up to date. We've investigated the external store situation a bit and the way it was configured in mw-vagrant (only there for Flow) is no longer supported in Core.
[19:31:27] <addshore>	 stephanebisson: what issue are you having? if it is MCR related I might be able to spot it easily
[19:31:42] <stephanebisson>	 addshore: I don't think it is
[19:32:01] <stephanebisson>	 It's Flow-ExternalStore mw-vagrant config.
[19:32:44] <stephanebisson>	 We (probably matt_flaschen) will redo how it is configured in puppet to be closer to production in the way the code sees it.
[19:34:59] <matt_flaschen>	 On Vagrant currently it also affects non-Flow (External Store for everything), but it should be below the level of MCR.
[19:42:13] <addshore>	 thcipriani: this makes no sense, why the hell is the cherry pick failing :(
[19:42:35] <matt_flaschen>	 addshore, stephanebisson, I think we will do it in two phases, first putting the blobs tables in the main database, then making sure you can not "cross the streams" in either direction (read the blobs table when you should be in the main DB, and vice-versa).
[19:43:17] <matt_flaschen>	 stephanebisson, instead of configuring a whole separate MySQL, I think it's probably enough to just use user permissions to prevent crossing.
[19:44:54] <thcipriani>	 addshore: looks like it expects there to be an exception thrown in RevisionStore but it ends up going all the way to title where it does the 'Cannot access property on non-object' thing
[19:45:18] <addshore>	 Yeh, hmmmmm
[19:45:34] <addshore>	 but the behaviour hasn't changed between the 2 branches
[19:48:28] <wikibugs>	 (03PS1) 10MaxSem: Archive ContributionReporting [integration/config] - 10https://gerrit.wikimedia.org/r/404778 (https://phabricator.wikimedia.org/T185062)
[19:49:09] <thcipriani>	 doesn't it change: https://gerrit.wikimedia.org/r/#/c/404757/1/includes/Storage/RevisionStore.php
[19:56:26] <addshore>	 thcipriani: I mean, it changes it, but that file is the same on .16 and .17, so I don't understand why the same patch would make something fail
[20:00:49] <wikibugs>	 (03PS1) 10Chad: branch.py: Add $wgVersion fixing in DefaultSettings [tools/release] - 10https://gerrit.wikimedia.org/r/404787
[20:01:07] <shinken-wm>	 PROBLEM - Host deployment-netbox is DOWN: CRITICAL - Host Unreachable (10.68.19.203)
[20:01:26] <wikibugs>	 (03CR) 10Chad: "Best I can tell, this is the last step that we need to take over from make-wmf-branch." [tools/release] - 10https://gerrit.wikimedia.org/r/404787 (owner: 10Chad)
[20:01:47] <thcipriani>	 addshore: I uhh somehow $row is neither false nor an object from $db->selectRow according to this test ¯\_(ツ)_/¯
[20:02:06] <addshore>	 thcipriani: i have no idea how this test passed on the other branch
[20:02:08] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca03 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[20:02:32] <addshore>	 thcipriani: I have fixed the test, 1 sec
[20:02:56] <shinken-wm>	 PROBLEM - Puppet errors on deployment-poolcounter04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[20:04:30] <thcipriani>	 yeah, I'm not sure why that's happening either :(
[20:04:56] <thcipriani>	 we do have a new hhvm without the patch that was giving us trouble yesterday as of today.
[20:05:04] <matt_flaschen>	 stephanebisson, T161190
[20:05:08] <hashar>	 paladox: openstack has created some mailing lists dedicated to Zuul!  http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-announce  && http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss
[20:05:11] <stashbot>	 T161190: Configure External Store on Vagrant with blobs tables in databases named after DB - https://phabricator.wikimedia.org/T161190
[20:05:16] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-1002 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[20:05:33] <paladox>	 hashar thanks :)
[20:06:28] <paladox>	 hashar i saw this http://lists.zuul-ci.org/pipermail/zuul-announce/2018-January/000000.html
[20:06:37] <paladox>	 does this mean they have stopped supporting 2.x now?
[20:06:38] <hashar>	 yup
[20:06:49] <addshore>	 thcipriani: https://gerrit.wikimedia.org/r/#/c/404757/ PS3 should be fixed
[20:06:53] <addshore>	 PS2 is just a large balls up
[20:06:53] <paladox>	 I wonder if we can apply some custom patches to add back jenkins support in 3.x.
[20:06:56] <stephanebisson>	 matt_flaschen: looks good, I don't know how best to isolate the dbs
[20:06:57] <wikibugs>	 (03CR) 1020after4: [C: 031] branch.py: Add $wgVersion fixing in DefaultSettings [tools/release] - 10https://gerrit.wikimedia.org/r/404787 (owner: 10Chad)
[20:06:59] <hashar>	 which also means patches we have for zuul2 are not going to ever be merged/released by them :D
[20:07:07] <paladox>	 oh
[20:07:31] <hashar>	 but we can keep them cherry picked in our .deb package (the patch-queue/debian/jessie branch ) so it is not a big deal 
[20:07:31] <paladox>	 I guess they still support the jenkins plugin though i forgot the name of it
[20:07:37] <hashar>	 for Jenkins I don't know really
[20:08:19] <hashar>	 I am not sure what he meant by "dropping jenkins support" since it is hidden behind gearman but who knwos really
[20:08:32] <wikibugs>	 10Continuous-Integration-Config, 10Proton, 10Readers-Web-Backlog, 10Patch-For-Review, 10Readers-Web-Kanban-Board: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3728431 (10mmodell) So what remains to be done here? html2pdf test is br...
[20:08:35] <hashar>	 but eventually we might want to migrate to v3 as well
[20:08:43] <thcipriani>	 addshore: thank you, I'll get both these out.
[20:08:57] <paladox>	 Yeh
[20:09:00] <paladox>	 hashar :) :)
[20:09:03] <paladox>	 it supports github too
[20:09:28] <paladox>	 Maybe i should start testing zuul v3.
[20:11:07] * paladox upstreams the new login ui :)
[20:16:05] <addshore>	 thcipriani: I'm walking to the shops now, ping me if anything pops up
[20:16:27] <thcipriani>	 addshore: will do, thanks again. I am also confused about that test :\
[20:20:51] <andrewbogott>	 In about 90 minutes (at 16:00 PST) I'm going to reboot some more WMCS hosts.  This will interrupt nodepool and CI for a few minutes and also break wikitech and Horizon.
[20:20:55] <andrewbogott>	 It should be quick.
[20:23:29] <addshore>	 thcipriani: I'll have to checkout the other branch and see what is happening
[20:23:39] <addshore>	 The code itself is fine afaik
[20:34:12] <greg-g>	 andrewbogott: fun
[20:36:16] <andrewbogott>	 yeah
[20:37:08] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca03 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:37:52] <shinken-wm>	 RECOVERY - Puppet errors on deployment-poolcounter04 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:40:16] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:58:04] <wikibugs>	 (03CR) 10Chad: [C: 032] branch.py: Add $wgVersion fixing in DefaultSettings [tools/release] - 10https://gerrit.wikimedia.org/r/404787 (owner: 10Chad)
[20:59:47] <wikibugs>	 (03Merged) 10jenkins-bot: branch.py: Add $wgVersion fixing in DefaultSettings [tools/release] - 10https://gerrit.wikimedia.org/r/404787 (owner: 10Chad)
[21:00:52] <wikibugs>	 (03PS1) 10Chad: branch.py: Use pygerrit2 not pygerrit, also env for py3 [tools/release] - 10https://gerrit.wikimedia.org/r/404804
[21:00:54] <wikibugs>	 (03CR) 10Chad: [C: 032] branch.py: Use pygerrit2 not pygerrit, also env for py3 [tools/release] - 10https://gerrit.wikimedia.org/r/404804 (owner: 10Chad)
[21:01:30] <wikibugs>	 (03Merged) 10jenkins-bot: branch.py: Use pygerrit2 not pygerrit, also env for py3 [tools/release] - 10https://gerrit.wikimedia.org/r/404804 (owner: 10Chad)
[21:01:35] <paladox>	 no_justification that script looks like it wont work
[21:01:38] <paladox>	 it uses auth=HTTPDigestAuth(CONFIG['username'], CONFIG['password']))
[21:02:04] <paladox>	 digest was switched off.
[21:05:37] <no_justification>	 Auth works
[21:05:42] <no_justification>	 I'm just making bad requests rn
[21:10:18] <paladox>	 oh i see.
[21:15:14] <no_justification>	 requests.exceptions.HTTPError: 400 Client Error: Bad Request for url: https://gerrit.wikimedia.org/r/a/projects/mediawiki%2Fcore/branches/wmf%2F1.99.0-wmf.99
[21:15:19] <no_justification>	 Hmm, bad request
[21:16:10] <paladox>	 i guess it's because it's trying to auth using /a/
[21:16:37] <paladox>	 needs to be updated to use basic auth instead of digest.
[21:16:43] <paladox>	 oh never mind
[21:17:04] <paladox>	 heh i get a not found.
[21:17:28] <paladox>	 no_justification are you trying to create the branch or get info for that branch?
[21:18:07] <no_justification>	 I swapped it in my local change
[21:18:10] <no_justification>	 Create the branch
[21:18:50] <paladox>	 hmm
[21:18:55] <paladox>	 i see only this branch
[21:18:55] <paladox>	 	wmf/1.31.0-wmf.17
[21:18:56] <wikibugs>	 (03PS1) 10Chad: branch.py: Fix all kinds of egregious errors [tools/release] - 10https://gerrit.wikimedia.org/r/404807
[21:18:58] <wikibugs>	 (03CR) 10Chad: [C: 032] branch.py: Fix all kinds of egregious errors [tools/release] - 10https://gerrit.wikimedia.org/r/404807 (owner: 10Chad)
[21:19:32] <wikibugs>	 (03Merged) 10jenkins-bot: branch.py: Fix all kinds of egregious errors [tools/release] - 10https://gerrit.wikimedia.org/r/404807 (owner: 10Chad)
[21:20:27] <wikibugs>	 (03PS1) 10Chad: Add some docs to shut up pylint [tools/release] - 10https://gerrit.wikimedia.org/r/404808
[21:20:33] <wikibugs>	 (03CR) 10Chad: [C: 032] Add some docs to shut up pylint [tools/release] - 10https://gerrit.wikimedia.org/r/404808 (owner: 10Chad)
[21:20:35] <wikibugs>	 (03CR) 10Chad: [V: 032 C: 032] Add some docs to shut up pylint [tools/release] - 10https://gerrit.wikimedia.org/r/404808 (owner: 10Chad)
[21:37:34] <paladox>	 hashar heh zuul requires python 3 for v3.
[21:37:47] <hashar>	 paladox: yup
[21:38:04] <paladox>	 though reading the spec did not really mention the removal of jenkins support except it said zuul was primarly devloped for the migration from jenkins.
[21:38:15] <hashar>	 at least jessie has python 3.4
[21:38:44] <no_justification>	 branch.py requires py3.5+ :)
[21:38:49] <paladox>	 heh :)
[21:39:16] <hashar>	 you can grep jenkins in the zuulv3 branch
[21:39:25] <paladox>	 ok 
[21:39:31] <hashar>	 git log --grep=enkins zuulv3
[21:39:49] <wikibugs>	 (03PS1) 10Chad: Make config useful [tools/release] - 10https://gerrit.wikimedia.org/r/404829
[21:40:18] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Make config useful [tools/release] - 10https://gerrit.wikimedia.org/r/404829 (owner: 10Chad)
[21:41:52] <hashar>	 paladox: we will see :)
[21:42:00] <paladox>	 heh :)
[21:42:45] <paladox>	 hmm, it maches 2016 or lower
[21:42:54] <paladox>	 but i see no removal of jenkins support :).
[21:43:10] <hashar>	 ;D
[21:43:19] <hashar>	 anyway I gotta sleep
[21:43:31] <hashar>	 kudos for the gerrit login page
[21:44:03] <paladox>	 your welcome :).
[21:46:35] <wikibugs>	 (03PS2) 10Chad: Make config useful [tools/release] - 10https://gerrit.wikimedia.org/r/404829
[21:57:39] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<11.11%)
[22:02:07] <no_justification>	 deployment-redis01 is trusty. and we wonder why redis is broken
[22:02:23] <shinken-wm>	 PROBLEM - App Server Main HTTP Response on deployment-mediawiki06 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:03:26] <shinken-wm>	 PROBLEM - Puppet errors on deployment-eventlogging04 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:04:19] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis06 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[22:04:45] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cache-text04 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[22:05:16] <shinken-wm>	 PROBLEM - Puppet errors on deployment-memc07 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[22:05:50] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca02 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:05:52] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cpjobqueue is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[22:05:56] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-k8s-1002 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:05:59] <shinken-wm>	 PROBLEM - Puppet errors on deployment-etcd-01 is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [0.0]
[22:06:01] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[22:06:55] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cassandra3-01 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:07:02] <shinken-wm>	 PROBLEM - Puppet errors on deployment-videoscaler01 is CRITICAL: CRITICAL: 37.50% of data above the critical threshold [0.0]
[22:07:04] <shinken-wm>	 PROBLEM - Puppet errors on deployment-cumin is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[22:07:16] <shinken-wm>	 RECOVERY - App Server Main HTTP Response on deployment-mediawiki06 is OK: HTTP OK: HTTP/1.1 200 OK - 46843 bytes in 2.750 second response time
[22:08:17] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1007 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[22:08:23] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[22:08:23] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[22:08:40] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1005 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[22:08:41] <apergos>	 uh wat
[22:08:44] <shinken-wm>	 PROBLEM - Puppet errors on deployment-changeprop is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[22:08:54] <shinken-wm>	 PROBLEM - Puppet errors on deployment-parsoid09 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[22:09:29] <shinken-wm>	 PROBLEM - Puppet errors on deployment-memc06 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[22:09:35] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[22:09:55] <shinken-wm>	 PROBLEM - Puppet errors on deployment-aqs01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[22:10:13] <Reedy>	 apergos: reboots mean horizon/wikitech are down
[22:10:21] <greg-g>	 oh righg
[22:10:22] <Reedy>	 So probably means puppetmaster issues temporarily
[22:10:24] <shinken-wm>	 PROBLEM - Puppet errors on deployment-imagescaler02 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[22:10:40] <apergos>	 ah reboots, ok then
[22:10:42] <shinken-wm>	 PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 50.00% of data above the critical threshold [0.0]
[22:12:05] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mira is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[22:12:07] <shinken-wm>	 PROBLEM - Puppet errors on deployment-db04 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[22:13:15] <shinken-wm>	 PROBLEM - Puppet errors on deployment-elastic05 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[22:13:38] <shinken-wm>	 PROBLEM - Puppet errors on deployment-puppetmaster02 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[22:13:52] <shinken-wm>	 PROBLEM - Puppet errors on deployment-sca01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[22:14:09] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[22:14:52] <shinken-wm>	 PROBLEM - Puppet errors on deployment-kafka-jumbo-2 is CRITICAL: CRITICAL: 60.00% of data above the critical threshold [0.0]
[22:16:07] <no_justification>	 Reedy: Yeah error message sounds as much for that
[22:16:19] <no_justification>	 500 from puppetmaster hitting an http timeout in a function
[22:23:03] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Release-Engineering-Team (Someday), 10Reading-Infrastructure-Team-Backlog, and 3 others: Setup trending service CI - https://phabricator.wikimedia.org/T151469#3907746 (10Pchelolo) 05Open>03declined The trending service has been...
[22:30:57] <wikibugs>	 (03CR) 10Chad: [C: 032] Make config useful [tools/release] - 10https://gerrit.wikimedia.org/r/404829 (owner: 10Chad)
[22:31:28] <wikibugs>	 (03Merged) 10jenkins-bot: Make config useful [tools/release] - 10https://gerrit.wikimedia.org/r/404829 (owner: 10Chad)
[22:37:12] <wikibugs>	 10Continuous-Integration-Config, 10Release-Engineering-Team (Kanban), 10Proton, 10Readers-Web-Backlog, and 2 others: Set up Jenkins for chromium-render and chromium-render-deploy repositories - https://phabricator.wikimedia.org/T179552#3907796 (10greg) Adding our kanban, just so it's not lost. :)
[22:38:39] <shinken-wm>	 RECOVERY - Puppet errors on deployment-puppetmaster02 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:39:19] <shinken-wm>	 RECOVERY - Puppet errors on deployment-redis06 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:40:13] <shinken-wm>	 RECOVERY - Puppet errors on deployment-memc07 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:41:09] <wikibugs>	 (03PS1) 10Chad: makerelease.py: Simplify file output [tools/release] - 10https://gerrit.wikimedia.org/r/404876
[22:41:11] <wikibugs>	 (03CR) 10Chad: [C: 032] makerelease.py: Simplify file output [tools/release] - 10https://gerrit.wikimedia.org/r/404876 (owner: 10Chad)
[22:41:42] <wikibugs>	 (03Merged) 10jenkins-bot: makerelease.py: Simplify file output [tools/release] - 10https://gerrit.wikimedia.org/r/404876 (owner: 10Chad)
[22:42:01] <shinken-wm>	 RECOVERY - Puppet errors on deployment-videoscaler01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:43:16] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1007 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:43:22] <shinken-wm>	 RECOVERY - Puppet errors on deployment-redis05 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:43:26] <shinken-wm>	 RECOVERY - Puppet errors on deployment-eventlogging04 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:43:40] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-docker-1005 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:43:52] <shinken-wm>	 RECOVERY - Puppet errors on deployment-parsoid09 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:44:10] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator, 10Wikimedia-Logstash: Improve error reporting / integration between Kibana and Phabricator - https://phabricator.wikimedia.org/T185155#3907836 (10mmodell)
[22:44:45] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cache-text04 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:44:59] <shinken-wm>	 RECOVERY - Puppet errors on deployment-aqs01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:45:44] <shinken-wm>	 RECOVERY - Puppet errors on deployment-ms-be04 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:45:53] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cpjobqueue is OK: OK: Less than 1.00% above the threshold [0.0]
[22:45:56] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-k8s-1002 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:46:00] <shinken-wm>	 RECOVERY - Puppet errors on deployment-etcd-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:46:02] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mediawiki07 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:46:56] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cassandra3-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:47:04] <shinken-wm>	 RECOVERY - Puppet errors on deployment-cumin is OK: OK: Less than 1.00% above the threshold [0.0]
[22:48:13] <shinken-wm>	 RECOVERY - Puppet errors on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:48:23] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:48:45] <shinken-wm>	 RECOVERY - Puppet errors on deployment-changeprop is OK: OK: Less than 1.00% above the threshold [0.0]
[22:49:12] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0]
[22:49:30] <shinken-wm>	 RECOVERY - Puppet errors on deployment-memc06 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:49:34] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[22:52:07] <shinken-wm>	 RECOVERY - Puppet errors on deployment-mira is OK: OK: Less than 1.00% above the threshold [0.0]
[22:52:09] <shinken-wm>	 RECOVERY - Puppet errors on deployment-db04 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:04:16] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-docker-1007 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[23:06:56] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-k8s-1002 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[23:09:24] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-03 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[23:10:34] <shinken-wm>	 PROBLEM - Puppet errors on saucelabs-01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[23:15:11] <shinken-wm>	 PROBLEM - Puppet errors on integration-slave-jessie-android is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[23:15:47] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca02 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:16:17] <apergos>	 meh again?
[23:18:52] <shinken-wm>	 RECOVERY - Puppet errors on deployment-sca01 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:19:50] <shinken-wm>	 RECOVERY - Puppet errors on deployment-kafka-jumbo-2 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:20:24] <shinken-wm>	 RECOVERY - Puppet errors on deployment-imagescaler02 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:30:06] <wikibugs>	 (03PS1) 10Chad: makerelease: Simplify a few things [tools/release] - 10https://gerrit.wikimedia.org/r/404883
[23:30:09] <wikibugs>	 (03CR) 10Chad: [C: 032] makerelease: Simplify a few things [tools/release] - 10https://gerrit.wikimedia.org/r/404883 (owner: 10Chad)
[23:37:55] <wikibugs>	 (03Merged) 10jenkins-bot: makerelease: Simplify a few things [tools/release] - 10https://gerrit.wikimedia.org/r/404883 (owner: 10Chad)
[23:40:36] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator: Test phabricator translations on phab.wmflabs.org - https://phabricator.wikimedia.org/T184459#3908123 (10mmodell)
[23:44:09] <kaldari>	 Is jenkins-bot having problems? https://gerrit.wikimedia.org/r/#/c/404623
[23:44:35] <Reedy>	 https://integration.wikimedia.org/zuul/
[23:44:50] <Reedy>	 mwext-testextension-hhvm-jessie looks stuck
[23:45:34] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-01 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:49:22] <shinken-wm>	 RECOVERY - Puppet errors on saucelabs-03 is OK: OK: Less than 1.00% above the threshold [0.0]
[23:50:10] <shinken-wm>	 RECOVERY - Puppet errors on integration-slave-jessie-android is OK: OK: Less than 1.00% above the threshold [0.0]
[23:53:03] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Phabricator: Test phabricator translations on phab.wmflabs.org - https://phabricator.wikimedia.org/T184459#3908200 (10mmodell) 05Open>03Resolved I tested briefly on phab.wmflabs.org and more thoroughly on my local phabricator test install. This is ready to go to produ...