[00:02:07] <paladox>	 no_justification can i setup the avatars repo in gerrit (the one where we will store the images). Yep i know we are in no rush :)
[00:02:20] <no_justification>	 I'm also thinking we should make it configurable
[00:02:29] <no_justification>	 Actually, no
[00:02:33] <no_justification>	 Let's not overcomplicate this
[00:02:35] <no_justification>	 It Just Works
[00:02:36] <no_justification>	 heheheheheh
[00:02:39] <no_justification>	 I'm gone for the day
[00:02:40] <no_justification>	 l8r
[00:02:44] <paladox>	 ok
[00:03:04] * paladox setups the repo then :)
[00:03:09] <paladox>	 (not the plugin one)
[00:03:51] <paladox>	 no_justification ^^ :)
[00:07:10] <wikibugs>	 (03PS4) 10MaxSem: Prohibit PHP's vanilla execution [tools/codesniffer] - 10https://gerrit.wikimedia.org/r/423030
[00:09:01] <paladox>	 !log created new repo "All-Avatars" which will be used to host avatars used by gerrit. Setting owner as Gerrit Managers will allow merging in the repo fro all users soon :)
[00:09:03] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[00:14:45] <paladox>	 no_justification it seems we have the problem with user names being upper or lowercase
[00:14:53] * paladox has that problem on gerrit.git.wmflabs.org
[00:19:39] <paladox>	 no_justification https://gerrit.wikimedia.org/r/423584 :)
[00:21:19] <no_justification>	 Ooooh, what if custom repo....
[00:21:23] <no_justification>	 You store in All-Users
[00:21:27] <no_justification>	 On their ref
[00:21:29] <no_justification>	 As like avatar.png
[00:21:56] <wikibugs>	 10Gerrit: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183#4099199 (10Paladox) @demon has created https://github.com/demon/avatars-ondisk. I have done this fix https://github.com/demon/avatars-ondisk/pull/1  It works locally.  We will be hosting avatars at https://gerrit.wikimedia.org/r/plugi...
[00:22:03] <paladox>	 no_justification https://gerrit.wikimedia.org/r/plugins/gitiles/All-Avatars/
[00:22:19] <paladox>	 no_justification i am not sure if that is possible though
[00:22:23] <paladox>	 to store on there ref though
[00:23:09] <no_justification>	 Oh, it's not something nice life refs/users/Chad
[00:23:18] <no_justification>	 It's the sha1 of my user :\
[00:23:21] <no_justification>	 Or a UUID
[00:23:23] <no_justification>	 Or something silly
[00:23:47] <paladox>	 yeh
[00:24:09] <paladox>	 no_justification i thought your user id was an emojie?
[00:24:17] <no_justification>	 On-wiki
[00:24:18] <no_justification>	 Hehe
[00:24:20] <paladox>	 oh
[00:24:26] <no_justification>	 I don't think ref names can use unicode
[00:24:27] <no_justification>	 :p
[00:24:35] <paladox>	 notedb can
[00:24:36] <paladox>	 i think
[00:24:51] <no_justification>	 $ git checkout -b 😂
[00:24:51] <no_justification>	 Switched to a new branch '😂'
[00:24:55] <no_justification>	 TODAY I LEARNED
[00:24:55] <paladox>	 lol
[00:25:00] <no_justification>	 ALL MY BRANCHES WILL NOW BE EMOJI
[00:25:10] <paladox>	 lol
[00:25:56] <paladox>	 no_justification i wonder what default image should we use?
[00:25:57] <paladox>	 heh
[00:26:46] <no_justification>	 https://commons.wikimedia.org/wiki/File:Stick_Figure.svg
[00:26:48] <no_justification>	 Totally. That.
[00:26:54] <paladox>	 lols
[00:28:45] <paladox>	 no_justification https://gerrit.git.wmflabs.org/r/c/3/
[00:35:46] <paladox>	 stick men heh
[00:35:54] <paladox>	 https://gerrit.git.wmflabs.org/r/?polygerrit=1/c/3/
[00:36:05] <paladox>	 https://gerrit.git.wmflabs.org/r/c/3/?polygerrit=1
[00:40:33] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4099247 (10Tgr) Code quality is important to developers but barely relevant for user...
[00:50:13] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-mediawiki04 is CRITICAL: CRITICAL: deployment-prep.deployment-mediawiki04.diskspace.root.byte_percentfree (<11.11%)
[00:55:34] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4099282 (10Legoktm) >>! In T178349#4099247, @Tgr wrote: > Code quality is important...
[00:56:53] <no_justification>	 paladox: tbh....we can use avatars-external for this
[00:56:56] <no_justification>	 Skip our own plugin
[00:57:24] <no_justification>	 1) git repo with people's avatars in it
[00:57:24] <no_justification>	 2) have puppet clone that to /var/www/html/avatars and it goes out via apache
[01:00:12] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-mediawiki04 is OK: OK: All targets OK
[01:05:24] <paladox>	 Oh
[01:05:29] <paladox>	 no_justification: yeh 
[01:05:51] <paladox>	 Hmm though it doesn’t use default no_justification  or does it?
[01:22:27] <paladox>	 no_justification: we may have to use your plugin as it defaults to a default image
[01:43:16] <wikibugs>	 10Release-Engineering-Team, 10Epic, 10Tracking: [EPIC] Provide pre-merge reports on patchsets (tracking) - https://phabricator.wikimedia.org/T101542#4099380 (10Legoktm)
[01:43:19] <wikibugs>	 10Release-Engineering-Team, 10Epic, 10WMF-deploy-2015-06-30_(1.26wmf12): [EPIC] Encourage developers to increase code coverage - https://phabricator.wikimedia.org/T100294#4099381 (10Legoktm)
[01:43:24] <wikibugs>	 10Continuous-Integration-Infrastructure, 10phpunit-patch-coverage, 10Epic, 10MediaWiki-Platform-Team (MWPT-Q3-Jan-Mar-2018), and 2 others: Provide (pre-merge) code coverage reports on patchsets - https://phabricator.wikimedia.org/T101544#4099378 (10Legoktm) 05Open>03Resolved For the most part this is d...
[01:44:54] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Epic: Provide (pre-merge) performance reports on patchsets - https://phabricator.wikimedia.org/T101543#4099388 (10Legoktm)
[01:44:56] <wikibugs>	 10Release-Engineering-Team, 10Epic, 10Tracking: [EPIC] Provide pre-merge reports on patchsets (tracking) - https://phabricator.wikimedia.org/T101542#1342089 (10Legoktm)
[01:44:58] <wikibugs>	 10Continuous-Integration-Infrastructure: Preview generated documentation in test pipeline for review - https://phabricator.wikimedia.org/T72945#4099390 (10Legoktm)
[01:45:00] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Testing-Initiative: Jenkins: Set up perceptual diffs (visual regression testing) - https://phabricator.wikimedia.org/T64633#4099391 (10Legoktm)
[01:45:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Epic: Provide infrastructure to store files by project/branch post-merge to compare with pre-merge - https://phabricator.wikimedia.org/T101545#4099384 (10Legoktm) 05Open>03declined Declining given that the main motivation for this was {T101544}, which was implemen...
[02:23:30] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-fluorine02 is CRITICAL: CRITICAL: deployment-prep.deployment-fluorine02.diskspace._srv.byte_percentfree (<10.00%)
[04:09:12] <wikibugs>	 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint issue on Abusefilter are meaningless. Just show dots and "1 problem" - https://phabricator.wikimedia.org/T190072#4099467 (10Legoktm) p:05Triage>03Unbreak! This is a bit ridiculous...  ``` Running "stylelint:all" (stylelint) task  resou...
[04:09:35] <wikibugs>	 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint is just outputting dots and number of errors, making it impossible to fix - https://phabricator.wikimedia.org/T190072#4099471 (10Legoktm)
[04:17:14] <wikibugs>	 10Continuous-Integration-Config, 10AbuseFilter, 10Upstream: stylelint is just outputting dots and number of errors, making it impossible to fix - https://phabricator.wikimedia.org/T190072#4099472 (10Legoktm) Workaround: `./node_modules/.bin/stylelint resources/*.less` appears to work.
[07:08:32] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-fluorine02 is OK: OK: All targets OK
[07:39:15] <wikibugs>	 (03PS1) 10Hashar: Add php-apcu and php-tidy [integration/quibble] - 10https://gerrit.wikimedia.org/r/423614 (https://phabricator.wikimedia.org/T191233)
[07:47:35] <wikibugs>	 (03PS3) 10Hashar: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026
[07:47:37] <wikibugs>	 (03PS1) 10Hashar: docker: quibble 0.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/423617
[07:52:47] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team: Consider disabling differential - https://phabricator.wikimedia.org/T191182#4099819 (10TerraCodes) Wouldn't this break likes D1017 where they link to differential?
[08:07:10] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Add missing dependencies for MediaWiki tests to quibble image - https://phabricator.wikimedia.org/T191233#4099840 (10hashar) a:03hashar
[08:22:29] <shinken-wm>	 PROBLEM - Mediawiki Error Rate on graphite-labs is CRITICAL: CRITICAL: 20.00% of data above the critical threshold [10.0]
[08:51:55] <wikibugs>	 10Release-Engineering-Team (Kanban), 10MediaWiki-Core-Tests, 10MW-1.31-release-notes (WMF-deploy-2018-03-27 (1.31.0-wmf.27)), 10Patch-For-Review, 10User-zeljkofilipin: Replace nodemw with mwbot - https://phabricator.wikimedia.org/T181284#3785137 (10Pablo-WMDE) WikibaseLexeme change in https://gerrit.wiki...
[08:52:37] <wikibugs>	 (03PS1) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652
[08:54:53] <wikibugs>	 (03PS1) 10Phedenskog: WebPageTest: Remove the Windows test agent [integration/config] - 10https://gerrit.wikimedia.org/r/423653 (https://phabricator.wikimedia.org/T165626)
[08:58:50] <wikibugs>	 (03PS2) 10Hashar: Add missing MediaWiki core tests dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423614 (https://phabricator.wikimedia.org/T191233)
[09:07:39] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add missing MediaWiki core tests dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423614 (https://phabricator.wikimedia.org/T191233) (owner: 10Hashar)
[09:08:04] <wikibugs>	 (03Merged) 10jenkins-bot: Add missing MediaWiki core tests dependencies [integration/quibble] - 10https://gerrit.wikimedia.org/r/423614 (https://phabricator.wikimedia.org/T191233) (owner: 10Hashar)
[09:08:23] <wikibugs>	 10Continuous-Integration-Infrastructure, 10MediaWiki-extensions-Newsletter: Unit tests for Newsletter extension failing in Wikimedia CI (ApiNewsletterSubscribeTest) - https://phabricator.wikimedia.org/T191284#4099983 (10matmarex) p:05Triage>03Unbreak!
[09:28:15] <wikibugs>	 (03PS2) 10Hashar: docker: quibble 0.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/423617
[09:28:43] <wikibugs>	 (03CR) 10Hashar: [C: 032] docker: quibble 0.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/423617 (owner: 10Hashar)
[09:30:02] <wikibugs>	 (03Merged) 10jenkins-bot: docker: quibble 0.0.3 [integration/config] - 10https://gerrit.wikimedia.org/r/423617 (owner: 10Hashar)
[09:49:15] <hashar>	 !log building releng/quibble:0.0.3
[09:49:18] <stashbot>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL
[10:05:31] <wikibugs>	 (03PS4) 10Hashar: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026
[10:10:43] <wikibugs>	 (03PS5) 10Hashar: Experimental Quibble job [integration/config] - 10https://gerrit.wikimedia.org/r/423026
[10:18:17] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Quibble, 10Patch-For-Review: Add missing dependencies for MediaWiki tests to quibble image - https://phabricator.wikimedia.org/T191233#4100132 (10hashar) 05Open>03Resolved
[10:22:47] <wikibugs>	 (03PS1) 10QChris: Allow “Gerrit Managers” to import history [extensions/HoverGallery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423662
[10:22:49] <wikibugs>	 (03CR) 10QChris: [V: 031 C: 032] Allow “Gerrit Managers” to import history [extensions/HoverGallery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423662 (owner: 10QChris)
[10:22:52] <wikibugs>	 (03PS1) 10QChris: Import done. Revoke import grants [extensions/HoverGallery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423663
[10:22:54] <wikibugs>	 (03CR) 10QChris: [V: 031 C: 032] Import done. Revoke import grants [extensions/HoverGallery] (refs/meta/config) - 10https://gerrit.wikimedia.org/r/423663 (owner: 10QChris)
[11:30:13] <wikibugs>	 (03PS1) 10Hashar: Partial support composer as a packages source [integration/quibble] - 10https://gerrit.wikimedia.org/r/423669
[11:30:15] <wikibugs>	 (03PS1) 10Hashar: Add composer test and npm test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423670
[12:11:52] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-ores01 is CRITICAL: CRITICAL: deployment-prep.deployment-ores01.diskspace._srv.byte_percentfree (No valid datapoints found)deployment-prep.deployment-ores01.diskspace.root.byte_percentfree (<100.00%)
[12:16:42] <wikibugs>	 (03PS1) 10Hashar: git-changed-in-head did not detect renames [integration/jenkins] - 10https://gerrit.wikimedia.org/r/423676
[12:38:07] <wikibugs>	 (03PS2) 10WMDE-leszek: WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652
[12:38:40] <shinken-wm>	 PROBLEM - Puppet errors on deployment-ms-be04 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[12:39:16] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WikibaseLexeme: run node selenium tests as part of regular build [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek)
[12:42:34] <wikibugs>	 (03PS1) 10Hashar: Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677
[12:46:13] <godog>	 hi! I am on horizon but can't see "deployment-prep" in the list of my projects, expected?
[12:54:26] <shinken-wm>	 PROBLEM - Puppet errors on deployment-ms-be03 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[12:55:57] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4100436 (10MarcoAurelio) @greg should be able to fix @Ottomata's accesses there.
[12:57:05] <paladox>	 hashar looks like we will be getting avatars in gerrit soon T191183 :)
[12:57:06] <stashbot>	 T191183: Enable avatars in gerrit - https://phabricator.wikimedia.org/T191183
[12:59:53] <thedj>	 paladox: nice !
[13:00:00] <paladox>	 thedj :)
[13:00:14] <paladox>	 thedj it at least works on https://gerrit.git.wmflabs.org/r/?polygerrit=1
[13:01:10] <thedj>	 paladox: so, it's a gerrit plugin ?
[13:01:26] <thedj>	 or is it a local hosted gravatar or something ?
[13:01:37] <paladox>	 thedj yep, chad created a plugin that's similar to an existing one
[13:01:39] <paladox>	 nope
[13:01:44] <paladox>	 it's a repo hosted 
[13:02:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: deployment-prep down hosts - fix/remove? - https://phabricator.wikimedia.org/T191293#4100460 (10MarcoAurelio)
[13:02:36] <thedj>	 can't we pull directly from phabricator ?
[13:03:00] <thedj>	 oh gerrit doesn't know your phabricator name of course...
[13:03:04] <paladox>	 thedj https://gerrit.wikimedia.org/r/#/admin/projects/All-Avatars
[13:03:24] <paladox>	 thedj that would be too slow
[13:03:25] <paladox>	 as phabricator does not do /<image_name>
[13:03:34] <paladox>	 instead it uses some random generated id then /<image-name>
[13:03:47] <paladox>	 thedj well and theres that :)
[13:03:47] <thedj>	 dump script ? ;)
[13:04:08] <paladox>	 thedj dump script?
[13:04:55] <thedj>	 pseudocode: for all phabricator users with, wmf ldap name, fetch avatar, write to repo, run nightly :)
[13:05:27] <paladox>	 oh
[13:07:34] <paladox>	 https://gerrit.wikimedia.org/g/All-Avatars/+/refs/heads/master
[13:08:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Release-Engineering-Team, 10Puppet: Long-lived cherry-picks on deployment-puppetmaster02.deployment-prep.equiad.wmflabs - https://phabricator.wikimedia.org/T191294#4100482 (10MarcoAurelio)
[13:10:41] <paladox>	 thedj all users would be trusted when merging in https://gerrit.wikimedia.org/g/All-Avatars/+/refs/heads/master
[13:11:19] <paladox>	 i am thinking we should set object sizes to 10 or 20mb
[13:14:19] <wikibugs>	 (03PS2) 10Hashar: Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677
[13:15:35] <wikibugs>	 (03PS3) 10Hashar: Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677
[13:26:03] <Hauskatze>	 hashar: ntegration-slave-docker-1003 is having puppet errors
[13:26:07] <Hauskatze>	 *integration
[13:36:01] <wikibugs>	 (03PS4) 10Hashar: Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677 (https://phabricator.wikimedia.org/T158974)
[13:36:33] <shinken-wm>	 PROBLEM - Puppet errors on deployment-eventlog05 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[13:36:56] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team: Consider disabling differential - https://phabricator.wikimedia.org/T191182#4100526 (10EddieGP) I might have expressed myself badly by saying "disable". The proposal here is to stop reviewing code on differential. It isn't to technically disable differenti...
[13:53:42] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep access request - https://phabricator.wikimedia.org/T191296#4100564 (10fgiunchedi)
[13:56:49] <wikibugs>	 (03CR) 10Hashar: [C: 032] Partial support composer as a packages source [integration/quibble] - 10https://gerrit.wikimedia.org/r/423669 (owner: 10Hashar)
[13:56:53] <wikibugs>	 (03CR) 10Hashar: [C: 032] Add composer test and npm test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423670 (owner: 10Hashar)
[13:56:56] <wikibugs>	 (03CR) 10Hashar: [C: 032] Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677 (https://phabricator.wikimedia.org/T158974) (owner: 10Hashar)
[13:57:14] <wikibugs>	 (03Merged) 10jenkins-bot: Partial support composer as a packages source [integration/quibble] - 10https://gerrit.wikimedia.org/r/423669 (owner: 10Hashar)
[13:57:19] <wikibugs>	 (03Merged) 10jenkins-bot: Add composer test and npm test [integration/quibble] - 10https://gerrit.wikimedia.org/r/423670 (owner: 10Hashar)
[13:57:23] <wikibugs>	 (03Merged) 10jenkins-bot: Speed up php lint/style with git-changed-in-head [integration/quibble] - 10https://gerrit.wikimedia.org/r/423677 (https://phabricator.wikimedia.org/T158974) (owner: 10Hashar)
[14:11:46] <wikibugs>	 10Continuous-Integration-Infrastructure (shipyard), 10Operations, 10Operations-Software-Development, 10Patch-For-Review: New tool to track package updates/status for hosts and images (debmonitor) - https://phabricator.wikimedia.org/T167504#4100624 (10Volans)
[15:01:04] <wikibugs>	 (03PS1) 10Hashar: Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714
[15:01:06] <wikibugs>	 (03PS1) 10Hashar: WIP refactor clone [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715
[15:01:26] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] WIP refactor clone [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715 (owner: 10Hashar)
[15:18:04] <niedzielski>	 twentyafterfour: hey there! i'm trying to add a blog post to the web team's blog: https://phabricator.wikimedia.org/phame/blog/manage/9/. unfortunately, we can't figure out how to add a user to the "custom policy" (i asked an admin, joakino ). can you add me or tell us how to edit it?
[15:19:39] <joakino>	 i think i'm just allowed to post there, not an admin. phuedx requested we were added when the blog was created but niedzielski joined the team later
[15:22:15] <niedzielski>	 ah, i guess ideally we'd add another person as admin then so we can update the permissions to include the rest of the team. i think folks like jan, tilman, etc should have access
[15:27:44] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:33:41] <shinken-wm>	 PROBLEM - Puppet errors on deployment-redis02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:46:59] <wikibugs>	 10MediaWiki-Releasing, 10MediaWiki-Installer, 10MW-1.31-release: Expand the set of bundled extensions to achieve a default MediaWiki experience that's comparable to Wikimedia sites - https://phabricator.wikimedia.org/T178349#4101002 (10demon) Yes. I'm not saying that code quality / dev interest are the reaso...
[15:49:16] <wikibugs>	 (03PS2) 10Hashar: Split repo dependencies and actual cloning [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715
[15:49:18] <wikibugs>	 (03PS2) 10Hashar: Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714
[15:49:39] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Split repo dependencies and actual cloning [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715 (owner: 10Hashar)
[15:49:41] <wikibugs>	 (03CR) 10jerkins-bot: [V: 04-1] Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714 (owner: 10Hashar)
[16:01:21] <wikibugs>	 (03CR) 10WMDE-leszek: "An example of when it is needed for browser tests to have had a composer install/update run prior to running tests could be seen for examp" [integration/config] - 10https://gerrit.wikimedia.org/r/423652 (owner: 10WMDE-leszek)
[16:03:31] <paladox>	 no_justification http://gerrit-new.wmflabs.org/r/admin/repos/%25F0%259F%2598%2582
[16:03:33] <paladox>	 oh wow
[16:03:35] <paladox>	 lol
[16:03:52] <paladox>	 git clone http://gerrit-new.wmflabs.org/r/😂
[16:04:12] <paladox>	 and it worked checking it out
[16:05:54] <paladox>	 you can also have your full name as 😂 but username has to be text
[16:11:06] <wikibugs>	 (03PS3) 10Hashar: Split repo dependencies and actual cloning [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715
[16:11:08] <wikibugs>	 (03PS3) 10Hashar: Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714
[16:11:55] <wikibugs>	 (03CR) 10Hashar: [C: 032] Split repo dependencies and actual cloning [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715 (owner: 10Hashar)
[16:11:57] <wikibugs>	 (03CR) 10Hashar: [C: 032] Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714 (owner: 10Hashar)
[16:12:18] <wikibugs>	 (03Merged) 10jenkins-bot: Split repo dependencies and actual cloning [integration/quibble] - 10https://gerrit.wikimedia.org/r/423715 (owner: 10Hashar)
[16:12:23] <wikibugs>	 (03Merged) 10jenkins-bot: Port mw-create-composer-local.py [integration/quibble] - 10https://gerrit.wikimedia.org/r/423714 (owner: 10Hashar)
[16:19:33] <no_justification>	 I'm....surprised that works
[16:19:53] <paladox>	 no_justification it's notedb though
[16:19:59] <paladox>	 so basically it reading from a repo
[16:20:19] <no_justification>	 Not for DB reasons.
[16:20:23] <paladox>	 oh
[16:20:27] <no_justification>	 I'm surprised that (j)git allows it on disk!
[16:20:59] <paladox>	 heh
[16:30:02] <paladox>	 no_justification re the avatars, we have to use your plugin that supports a fallback to a default image :)
[16:30:16] <no_justification>	 I don't see why a fallback matters?
[16:30:21] <no_justification>	 Just leave it blank w/ no avatar?
[16:30:37] <paladox>	 no_justification in polygerrit, it does not behave like gwtui.
[16:30:42] <paladox>	 it will show a grey circle
[16:32:30] <no_justification>	 Ok, and?
[16:32:54] <no_justification>	 Is that a problem?
[16:33:24] <paladox>	 no_justification see https://phabricator.wikimedia.org/F16603433
[16:33:31] <paladox>	 notice the grey circle at the top
[16:33:53] <no_justification>	 Yes. I saw that yesterday. What is the problem? 
[16:34:33] <paladox>	 it is hard to find the login menu
[16:34:40] <paladox>	 if the circle is greyish
[16:34:43] <paladox>	 with no image in it
[16:35:17] <no_justification>	 Ahhh, OK now I understand 
[16:35:48] * no_justification has rather good vision, things like "hard to see" aren't obvious 
[16:36:41] <paladox>	 heh
[16:38:44] <no_justification>	 I just wish there was an easy way to inject preferences for a user. 
[16:39:01] <paladox>	 yeh
[16:39:08] <no_justification>	 It seems like I'd have to DIY and that feels error prone
[16:39:17] <wikibugs>	 10Scap, 10Scoring-platform-team, 10Patch-For-Review: [Blocked] Support git-lfs - https://phabricator.wikimedia.org/T180627#4101205 (10awight) Is there anything I should be doing to support this?
[16:39:20] <paladox>	 DIY?
[16:43:48] <greg-g>	 "do it yourself"
[16:44:19] <paladox>	 oh thanks
[16:50:53] <wikibugs>	 10Gerrit, 10Release-Engineering-Team (Kanban), 10Patch-For-Review: Install motd plugin - https://phabricator.wikimedia.org/T190810#4101265 (10demon) 05Open>03Resolved
[17:03:05] <paladox>	 no_justification wondering could you review https://github.com/demon/avatars-ondisk/pull/1 again please? :) :)
[17:04:36] <paladox>	 thanks :)
[17:09:11] <wikibugs>	 (03PS1) 10Florianschmidtwelzow: Whitelist HunterH [integration/config] - 10https://gerrit.wikimedia.org/r/423739
[17:09:23] <twentyafterfour>	 niedzielski: I added you to the edit policy... Can you see  https://phabricator.wikimedia.org/phame/blog/edit/9/
[17:11:46] <niedzielski>	 twentyafterfour: thanks! unfortunately, i logged out and back in again but i'm still blocked :|
[17:13:04] <twentyafterfour>	 niedzielski: how about now?
[17:14:04] <niedzielski>	 twentyafterfour: sorry, still the same even after logging out and back in again
[17:14:22] <twentyafterfour>	 FWIW you shouldn't have to log out and back in to test
[17:14:30] <twentyafterfour>	 hmm... 
[17:16:07] <twentyafterfour>	 niedzielski: try once more?
[17:17:08] <niedzielski>	 twentyafterfour: hooray!
[17:17:09] <niedzielski>	 it works!
[17:17:23] <niedzielski>	 thank you so so much twentyafterfour !
[17:18:41] <twentyafterfour>	 niedzielski: You're welcome!
[17:21:47] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: ensure minikube running on integration agents - https://phabricator.wikimedia.org/T188936#4101375 (10dduvall)
[17:21:49] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Permission denied upon `minikube start` as non-root user - https://phabricator.wikimedia.org/T191211#4101373 (10dduvall) 05Open>03Invalid After chatting with @thcipriani it's clear that `minikube --vm-driver=none` is not really meant to work when run...
[17:32:12] <awight>	 Why would I be getting these errors while deploying from tin?  > Timeout, server ores1002.eqiad.wmnet not responding.
[17:32:27] <awight>	 The target machine is fine, I can ssh to it from my local box.
[17:33:09] <mutante>	 ores1002 not in dsh groups
[17:33:38] <awight>	 O_O
[17:33:51] <mutante>	 i am not saying this is "that explains it".. just i see this difference:
[17:34:03] <mutante>	 hieradata/common/scap/dsh.yaml:      - ores1001.eqiad.wmnet
[17:34:13] <mutante>	 while ores1002 is not in that file
[17:34:37] <mutante>	 now if you ask "why is it trying to connect there if it's not in the groups" then i cant explain that yet either
[17:35:25] <awight>	 hehe
[17:35:52] <mutante>	 ores-canaries are 1001 and 2001
[17:35:53] <awight>	 Is it supposed to be getting a list of hosts from conftool?
[17:35:56] <mutante>	 1002 isnt mentioned at all
[17:36:15] <mutante>	 what is 1002? not-a-canary?
[17:36:44] <mutante>	 yea, true, you are right awight
[17:36:50] <mutante>	 it gets the list from conftool
[17:36:54] <awight>	 We have two clusters, 1001-9 and 2001-9, and have been deploying to them pretty regularly
[17:36:56] <awight>	 this is strange.
[17:36:59] <mutante>	 and ores1001 is in it because that is a canary
[17:37:33] <mutante>	 there is a referene in that file to conftool
[17:38:44] <mutante>	 https://config-master.wikimedia.org/pybal/eqiad/ores
[17:40:55] <mutante>	 https://phabricator.wikimedia.org/T142621
[17:41:22] <mutante>	 i wonder how we have "ores1001/1002 have not been properly decomed" tickets while they also exist
[17:41:32] <thcipriani>	 hrm looks like /usr/bin/git submodule update --init --recursive --jobs 38 never completed...
[17:41:48] <thcipriani>	 on ores1002
[17:42:20] <mutante>	 oh.. ok.. in that case... "it's always suckmodules(tm)"
[17:42:55] <awight>	 T142621 is 2 years old, and is a different machine now.   Is this task still relevant?
[17:42:56] <stashbot>	 T142621: investigate ores1002 - not in racktables but shows up on switch - https://phabricator.wikimedia.org/T142621
[17:43:13] <awight>	 lol submodules are amazeballs for job security
[17:44:01] <mutante>	 awight: different machines should not have the same names :/  but there was probably a $reason to hack it
[17:44:13] <awight>	 argh
[17:45:44] <awight>	 My understanding is that I’m supposed to be relying on dsh manifests rather than the scap/ host lists
[17:45:50] <awight>	 so I’m not sure what to do at this point
[17:46:04] <thcipriani>	 so it seems like ssh may have timed out on 1002 after starting submodule update
[17:47:24] <thcipriani>	 if you want to use the machine dsh list rather than the scap host lists you can use the full path the the machine dsh lists, or you can remove the ones under ./scap and scap will fall back to using the ones under /etc/dsh/groups/[blah]
[17:49:03] <awight>	 ty, I’ll try that
[17:55:38] <wikibugs>	 (03PS1) 10Chad: deploy notes: fix a bunch of pylint errors [tools/release] - 10https://gerrit.wikimedia.org/r/423753
[17:58:27] <wikibugs>	 (03CR) 10Thcipriani: [C: 032] deploy notes: fix a bunch of pylint errors [tools/release] - 10https://gerrit.wikimedia.org/r/423753 (owner: 10Chad)
[17:58:56] <wikibugs>	 (03Merged) 10jenkins-bot: deploy notes: fix a bunch of pylint errors [tools/release] - 10https://gerrit.wikimedia.org/r/423753 (owner: 10Chad)
[18:03:05] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline, 10Patch-For-Review: ensure minikube running on integration agents - https://phabricator.wikimedia.org/T188936#4101538 (10dduvall) a:05dduvall>03thcipriani
[18:08:04] <wikibugs>	 10Beta-Cluster-Infrastructure: deployment-prep access request - https://phabricator.wikimedia.org/T191296#4101564 (10thcipriani) 05Open>03Resolved a:03thcipriani @fgiunchedi you should have access and be a projectadmin now.
[18:09:09] <paladox>	 no_justification https://github.com/golang/go/wiki/GerritBot
[18:09:29] <paladox>	 i got that link from https://groups.google.com/forum/#!topic/repo-discuss/Fbrl76bPdS4 (bottom)
[18:11:32] <paladox>	 https://github.com/golang/build/tree/master/cmd/gerritbot
[18:14:32] <paladox>	 no_justification also i think we can deploy your avatar plugin now :)
[18:17:26] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4101600 (10greg) 05Open>03stalled p:05Triage>03Low Brain dump of facts/where things are: * There is no push to get more people on Differential. That stopped long...
[18:20:48] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: install helm on integration agents - https://phabricator.wikimedia.org/T188934#4101608 (10dduvall) a:05dduvall>03thcipriani Left open to ensure the right invocation of `helm init` in puppet.
[18:21:10] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Verify functionality of the 'production' image in the context of an isolated k8s deployment - https://phabricator.wikimedia.org/T183165#4101616 (10dduvall)
[18:21:13] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: install kubectl on integration agents - https://phabricator.wikimedia.org/T188933#4101615 (10dduvall) 05Open>03Resolved
[18:21:45] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: modify service-pipeline to include helm install/helm test - https://phabricator.wikimedia.org/T188935#4101618 (10dduvall) p:05Triage>03Normal a:03dduvall
[18:22:32] <halfak>	 Hey folks.  We're alerting for th ORES extension. 
[18:22:45] <paladox>	 twentyafterfour ^^
[18:22:52] <halfak>	 It looks like something changed at 14:39 UTC
[18:23:07] <halfak>	 Suddenly, no more jobs were completing successfully. 
[18:23:11] <halfak>	 The service is working fine. 
[18:23:15] <halfak>	 No changes around that time. 
[18:23:23] <halfak>	 See https://grafana.wikimedia.org/dashboard/db/ores-extension?orgId=1
[18:24:18] <halfak>	 "jobrunner.pop.ORESFetchScoreJob.ok.mw****.rate" is what fell to zero
[18:24:45] <halfak>	 awight, ^ I'm adding some notes.  Can you add yours too?
[18:24:53] <wikibugs>	 (03PS1) 10Hashar: Cover quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423757
[18:25:08] <awight>	 Willdo
[18:25:26] <awight>	 halfak: Where are you adding notes?
[18:25:26] <thcipriani>	 is this for prod? if so it looks like https://tools.wmflabs.org/sal/log/AWKL8pOgBEfgIt1j_daj happened at 14:39, not sure if it's related
[18:25:33] <wikibugs>	 (03CR) 10Hashar: [C: 032] Cover quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423757 (owner: 10Hashar)
[18:25:57] <wikibugs>	 (03Merged) 10jenkins-bot: Cover quibble.zuul [integration/quibble] - 10https://gerrit.wikimedia.org/r/423757 (owner: 10Hashar)
[18:26:38] <paladox>	 halfak awight ^^
[18:27:15] <halfak>	 thcipriani, yeah prod
[18:27:36] <halfak>	 Looks like "jobrunner.pop.ORESFetchScoreJob.failure.mw****.rate" is zero
[18:27:44] <halfak>	 So maybe the jobs are just not happening. 
[18:27:56] <awight>	 thcipriani: That looks promising.  Our job is included in some of these patches, e.g. https://gerrit.wikimedia.org/r/#/c/423511/1/scap/vars.yaml
[18:28:06] <halfak>	 aha!
[18:28:30] <paladox>	 halfak https://gerrit.wikimedia.org/r/#/c/423710/1/wmf-config/jobqueue.php
[18:28:34] <paladox>	 awight ^^
[18:29:35] <awight>	 paladox: That particular patch doesn’t seem to change the assigned value, fwiw
[18:29:41] <paladox>	 ok
[18:29:53] <thcipriani>	 halfak: awight maybe services folks have more info and could help either mobrovac or Pchelolo 
[18:30:01] <awight>	 paladox: ooh actually, it does, I was wrong
[18:30:07] <paladox>	 oh
[18:31:30] <halfak>	 https://grafana.wikimedia.org/dashboard/db/ores-extension?orgId=1&panelId=1&fullscreen
[18:31:35] <halfak>	 ^ Now shows failed rate. 
[18:31:37] <halfak>	 At zero
[18:31:43] <halfak>	 (Or practically zero
[18:31:45] <halfak>	 )
[18:32:28] <halfak>	 https://grafana.wikimedia.org/dashboard/db/ores-extension?orgId=1&panelId=5&fullscreen
[18:32:59] <halfak>	 shows that we were erroring out on failed jobs that were trickling in while no successful jobs were happening. 
[18:33:05] <halfak>	 Thus 100% failures. 
[18:36:50] <halfak>	 Maybe these things are still happening and it's just that we're not logging it effectively. 
[18:39:22] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mx is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[18:41:53] <awight>	 Pchelolo: mobrovac: We need some help, please see backscroll.
[18:42:13] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101697 (10dduvall)
[18:43:41] <halfak>	 Looks like ORES filters are still working. 
[18:43:44] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101716 (10dduvall) p:05Triage>03Normal a:03dduvall
[18:43:48] <halfak>	 I think this might just be a logging/metrics bug. 
[18:43:56] <halfak>	 awight, I think we should downgrade our concerns. 
[18:44:04] <halfak>	 https://en.wikipedia.org/wiki/Special:RecentChanges?hidebots=1&hidecategorization=1&hideWikibase=1&limit=500&days=7&enhanced=1&damaging__likelybad_color=c4&damaging__verylikelybad_color=c5&urlversion=2
[18:44:57] * halfak reverts some vandalism just to make sure
[18:46:56] <awight>	 Good find, the specific metric is jobrunner.pop.ORESFetchScoreJob.ok.mw****.rate which easily could have changed
[18:55:43] <halfak>	 awight, I think it might be time to file a passive-aggressive task. 
[18:55:51] <halfak>	 And assign it to services 
[18:56:08] <awight>	 I’ve been livetweeting on T190327
[18:56:09] <stashbot>	 T190327: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327
[18:56:29] <halfak>	 Oh that's better 
[18:56:32] <awight>	 :) https://phabricator.wikimedia.org/T190327#4101798
[18:56:56] <halfak>	 Oh!  looks like Pchelolo got it for us :) 
[18:57:47] <awight>	 Are you seeing any graphs recover?  I’m not sure where the metric was changed.
[18:57:59] <halfak>	 Oh wait.  I'm confused.  He said, "I've updated the metric, the job was running all the time. I'll change your metrics for you."
[18:58:09] <halfak>	 But no I don't see the grafana fixed 
[19:00:00] <awight>	 I think he’s still working on it
[19:01:17] <halfak>	 Oh gotcha. 
[19:01:30] * halfak goes back to writing about social power dynamics in algorithm design. 
[19:02:12] <awight>	 I can see how that might make you paranoid
[19:06:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Puppet: Error: Could not find class role::kafka::jumbo::mirror for deployment-kafka0[45] - https://phabricator.wikimedia.org/T191154#4095542 (10thcipriani) >>! In T191154#4097292, @Ottomata wrote: > Hm actually, I don't seem to have access to the deployment-prep project in Hori...
[19:25:09] <shinken-wm>	 PROBLEM - SSH on integration-slave-docker-1012 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[19:28:40] <halfak>	 thanks Pchelolo!
[19:30:33] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101948 (10dduvall)
[19:30:51] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101697 (10dduvall) Added a third option.
[19:40:02] <shinken-wm>	 RECOVERY - SSH on integration-slave-docker-1012 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0)
[19:47:24] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Release Pipeline: Come up with a decent method of declaring helm chart path/version in service repo - https://phabricator.wikimedia.org/T191327#4101991 (10demon) Option 3 is what I was learning towards in my initial work.
[20:15:32] <wikibugs>	 (03PS1) 1020after4: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776
[20:17:00] <wikibugs>	 (03PS2) 1020after4: deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776
[20:30:57] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#4102152 (10mmodell)
[20:39:00] <shinken-wm>	 PROBLEM - Puppet errors on deployment-mediawiki07 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[20:50:20] <wikibugs>	 (03PS1) 10Hashar: Move logging coloring [integration/quibble] - 10https://gerrit.wikimedia.org/r/423784
[20:50:22] <wikibugs>	 (03PS1) 10Hashar: Also run 'npm test' for skins/extensions [integration/quibble] - 10https://gerrit.wikimedia.org/r/423785
[21:20:53] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102361 (10bd808) >>! In T191182#4101600, @greg wrote: > * We need to take into account Toolforge users who want one-click creation of a repository for their tools. This...
[21:23:35] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102377 (10mmodell) >>! In T191182#4102361, @bd808 wrote: > I'm not sure that I agree with this. In the Wikimedia movement, "tools" means a really broad range of things....
[21:24:54] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102379 (10Paladox) ^^, we should keep differential open for users who want to use it.
[21:32:26] <wikibugs>	 10Release-Engineering-Team (Kanban), 10Patch-For-Review, 10Release, 10Train Deployments: 1.31.0-wmf.28 deployment blockers - https://phabricator.wikimedia.org/T183967#4102410 (10mmodell)
[21:32:41] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102411 (10HappyDog) > Trying to fit all work by all people into a single workflow seems counterproductive to a goal of mine which is to make technical contributions easi...
[21:37:54] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102443 (10TerraCodes) One advantage of using all the same thing is that I don't have to try to figure out where something is because I know which site it's on.
[21:58:03] <wikibugs>	 10Phabricator, 10Operations: Phabricator is really slow - https://phabricator.wikimedia.org/T191361#4102487 (10Paladox)
[21:58:15] <wikibugs>	 10Phabricator, 10Operations: Phabricator is really slow - https://phabricator.wikimedia.org/T191361#4102497 (10Paladox) p:05Triage>03High
[22:03:27] <wikibugs>	 (03PS3) 1020after4: deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776
[22:04:17] <wikibugs>	 (03PS4) 1020after4: deploy-promote: add support for 'testwikis' group and $PHABTASK environment var [tools/release] - 10https://gerrit.wikimedia.org/r/423776
[22:06:45] <wikibugs>	 10Phabricator, 10Operations: Phabricator is really slow - https://phabricator.wikimedia.org/T191361#4102530 (10Paladox) Possibly related T182832
[22:12:31] <wikibugs>	 10Release-Engineering-Team (Watching / External), 10Wikipedia-Android-App-Backlog, 10Wikipedia-iOS-App-Backlog: Create end-to-end automated test for Wikipedia native app(s) - https://phabricator.wikimedia.org/T90177#4102578 (10LGoto) 05Open>03declined Closed as part of board grooming process.
[22:15:41] <wikibugs>	 10Phabricator, 10Operations: Phabricator is really slow - https://phabricator.wikimedia.org/T191361#4102588 (10Paladox) Takes 3.13s to load.  But it is very slow.  Even submitting this comment is taking a few secs whereas before it was really fast.
[22:17:38] <shinken-wm>	 PROBLEM - SSH on integration-slave-docker-1014 is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[22:18:21] <wikibugs>	 10Phabricator, 10Operations: Phabricator is loading really slowly - https://phabricator.wikimedia.org/T191361#4102595 (10Paladox)
[22:20:31] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4102601 (10Dzahn) ``` [phab1001:~] $ sudo apache-status...
[22:20:39] <wikibugs>	 10Phabricator, 10Operations: Phabricator is loading really slowly - https://phabricator.wikimedia.org/T191361#4102602 (10Paladox)
[22:22:29] <shinken-wm>	 RECOVERY - SSH on integration-slave-docker-1014 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u4 (protocol 2.0)
[22:26:40] <wikibugs>	 10Phabricator, 10Operations: Phabricator is loading really slowly - https://phabricator.wikimedia.org/T191361#4102623 (10Paladox) 05Open>03Resolved <mutante> restarting Apache on phab1001 - T182832
[22:30:21] <wikibugs>	 10Phabricator, 10Release-Engineering-Team (Kanban), 10Operations, 10Patch-For-Review, and 2 others: Apache on phab1001 is gradually leaking worker processes which are stuck in "Gracefully finishing" state - https://phabricator.wikimedia.org/T182832#4102628 (10Paladox) This happened again even with the cron...
[22:36:57] <wikibugs>	 10Differential, 10Developer-Relations, 10Differential-Beta, 10Mobile-App-Goals, and 3 others: [Differential] Update repo configuration to enable Differential - https://phabricator.wikimedia.org/T134505#4102686 (10LGoto) 05Open>03declined Closed as part of board grooming process.
[23:10:16] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102820 (10EddieGP) In an ideal world, it'd be possible to contribute to scap through github, send patches for restbase to gerrit, and upload a puppet patch to differenti...
[23:28:51] <wikibugs>	 10Gerrit, 10Phabricator, 10Release-Engineering-Team (Someday): Consider disabling differential - https://phabricator.wikimedia.org/T191182#4102841 (10bd808) >>! In T191182#4102411, @HappyDog wrote: > Do I take it from this that in practice most users work in silos, i.e. they work on one project, with an opti...