[00:12:11] <wikibugs>	 3Phabricator: Phabricator's "Create Task" blows away a task-in-progress - https://phabricator.wikimedia.org/T88255#1007198 (10Jhawkinson) 3NEW
[00:17:47] <shinken-wm>	 RECOVERY - Puppet failure on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [0.0]  
[00:19:03] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic07 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[00:21:13] <shinken-wm>	 RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[00:23:53] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[00:37:47] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<33.33%)  
[00:38:05] <wikibugs>	 3Beta-Cluster, operations: Convert git-sync-upstream script from bash to python - https://phabricator.wikimedia.org/T88238#1007212 (10yuvipanda) @Joe I think bash is a terrible language when you aren't just executing a sequence of commands (with or without piping). My personal guideline is to rewrite to python w...
[02:51:46] <wikibugs>	 3Beta-Cluster, operations: Set up an alert for unmerged changes in deployment-prep - https://phabricator.wikimedia.org/T87616#1007244 (10yuvipanda)
[02:51:47] <wikibugs>	 3Beta-Cluster, operations: Convert git-sync-upstream script from bash to python - https://phabricator.wikimedia.org/T88238#1007242 (10yuvipanda) 5Open>3declined On second thoughts, a lot of the complexity in that bash script can go away, so I'm just going to do that instead.  Still think complicated bash scr...
[02:57:00] <wikibugs>	 3Beta-Cluster, operations: Convert git-sync-upstream script from bash to python - https://phabricator.wikimedia.org/T88238#1007246 (10yuvipanda) (simplification done in https://gerrit.wikimedia.org/r/188019) and https://gerrit.wikimedia.org/r/#/c/188018/
[03:11:33] <shinken-wm>	 PROBLEM - Puppet failure on deployment-salt is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]  
[03:16:34] <shinken-wm>	 RECOVERY - Puppet failure on deployment-salt is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:18:30] <shinken-wm>	 PROBLEM - Puppet failure on deployment-logstash1 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]  
[03:30:09] <wikibugs>	 3Phabricator: Where is Differential? - https://phabricator.wikimedia.org/T88257#1007275 (10Negative24) 3NEW
[03:34:20] <wikibugs>	 3Phabricator: Where is Differential? - https://phabricator.wikimedia.org/T88257#1007290 (10Jdforrester-WMF)
[03:34:24] <wikibugs>	 3Code-Review, Engineering-Community: Plan to migrate code review from Gerrit to Phabricator - https://phabricator.wikimedia.org/T18#1007291 (10Jdforrester-WMF)
[03:38:31] <shinken-wm>	 RECOVERY - Puppet failure on deployment-logstash1 is OK: OK: Less than 1.00% above the threshold [0.0]  
[03:39:30] <wmf-insecte>	 Yippee, build fixed!
[03:39:31] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #289: FIXED in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/289/
[03:47:03] <wmf-insecte>	 Yippee, build fixed!
[03:47:03] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #452: FIXED in 10 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/452/
[03:49:35] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #515: FAILURE in 38 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/515/
[04:17:54] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce build #272: FAILURE in 30 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-monobook-sauce/272/
[04:19:30] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #481: FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-chrome-sauce/481/
[04:29:07] <wmf-insecte>	 Project browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce build #288: FAILURE in 40 min: https://integration.wikimedia.org/ci/job/browsertests-Flow-en.wikipedia.beta.wmflabs.org-linux-firefox-monobook-sauce/288/
[04:39:31] <grrrit-wm>	 (03PS1) 10Jforrester: mwext-VisualEditor-qunit-karma: Make non-voting until it works [integration/config] - 10https://gerrit.wikimedia.org/r/188024 
[04:40:52] <grrrit-wm>	 (03CR) 10Jforrester: "No idea if this is the way to make this work, but…" [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[05:44:37] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #443: FAILURE in 32 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/443/
[05:54:35] <wmf-insecte>	 Project beta-scap-eqiad build #39897: FAILURE in 40 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39897/
[05:59:30] <wikibugs>	 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#1007440 (10yuvipanda)
[05:59:31] <wikibugs>	 3Beta-Cluster: Unify labs and prod roles for role::deployment::deployment_servers - https://phabricator.wikimedia.org/T86885#1007439 (10yuvipanda) 5Open>3Resolved
[06:37:43] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK  
[06:38:37] <wmf-insecte>	 Yippee, build fixed!
[06:38:38] <wmf-insecte>	 Project beta-scap-eqiad build #39899: FIXED in 24 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39899/
[06:58:10] <wikibugs>	 3Phabricator: Show percentage of teams migrated to Phabricator for project management - https://phabricator.wikimedia.org/T434#1007504 (10Qgil)
[06:58:13] <wikibugs>	 3Phabricator: Clearer instructions in the Phabricator login page - https://phabricator.wikimedia.org/T545#1007506 (10Qgil)
[07:13:46] <wikibugs>	 3Code-Review, Engineering-Community: How to prioritize code review of patches submitted by volunteers - https://phabricator.wikimedia.org/T78768#1007558 (10Qgil) a:3Qgil
[07:13:50] <wikibugs>	 3Phabricator, Engineering-Community, Wikimedia-Git-or-Gerrit: Migrate Gerrit project ownership request system (+2 rights) to Phabricator - https://phabricator.wikimedia.org/T86639#1007564 (10Qgil) a:3Qgil
[07:16:45] <wikibugs>	 3Phabricator, Engineering-Community, Wikimedia-Git-or-Gerrit: Migrate Gerrit project ownership request system (+2 rights) to Phabricator - https://phabricator.wikimedia.org/T86639#1007573 (10Nemo_bis) >>! In T86639#973256, @Qgil wrote: > Can we just reuse #Wikimedia-Git-or-Gerrit for this process, asking people...
[07:26:00] <shinken-wm>	 PROBLEM - Puppet failure on deployment-elastic06 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]  
[07:37:17] <shinken-wm>	 PROBLEM - Puppet failure on deployment-elastic05 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[07:48:46] <shinken-wm>	 PROBLEM - Puppet failure on deployment-elastic07 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[08:01:58] <wikibugs>	 3Engineering-Community, Code-Review: Plan to migrate code review from Gerrit to Phabricator - https://phabricator.wikimedia.org/T18#1007597 (10Qgil) Just adding a reminder about the need to discuss this plan in March if we want to work on this during the April-June quarter.
[08:23:27] <wikibugs>	 3MediaWiki-extensions-ContentTranslation, Continuous-Integration, ContentTranslation-Deployments: Enable Debian CI tests on all Apertium packages - https://phabricator.wikimedia.org/T87607#1007715 (10Arrbee)
[08:28:59] <shinken-wm>	 PROBLEM - Puppet failure on deployment-fluoride is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]  
[08:29:25] <shinken-wm>	 PROBLEM - Puppet failure on deployment-jobrunner01 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]  
[08:32:50] <wikibugs>	 3Wiki-Release-Team, Engineering-Community, Release-Engineering: Clean #Wiki-Release-Team project - https://phabricator.wikimedia.org/T88263#1007725 (10Qgil) 3NEW
[08:54:02] <shinken-wm>	 RECOVERY - Puppet failure on deployment-fluoride is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:54:28] <shinken-wm>	 RECOVERY - Puppet failure on deployment-jobrunner01 is OK: OK: Less than 1.00% above the threshold [0.0]  
[08:56:17] <wikibugs>	 3Project-Creators, Phabricator, Engineering-Community: Analytics-Volunteering and Wikidata's Need-Volunter tags - https://phabricator.wikimedia.org/T88266#1007773 (10Qgil) 3NEW
[09:28:18] <wikibugs>	 3Beta-Cluster: Remove all ::beta roles in puppet - https://phabricator.wikimedia.org/T86644#1007858 (10hashar)
[09:40:46] <grrrit-wm>	 (03CR) 10Hashar: "The proposed patch stops triggering Zend based tests on patchset proposal but they are still triggered on code-review +2. The idea is to s" [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar)
[09:41:48] <grrrit-wm>	 (03Abandoned) 10Hashar: yamllint: allow !!python/tuples [integration/jenkins] - 10https://gerrit.wikimedia.org/r/183858 (owner: 10Hashar)
[09:47:20] <vikasyaligar>	 hashar: Hello !
[09:47:54] <vikasyaligar>	 hashar: Wanted to know whether zeljko is going to come online today ? 
[09:50:21] <wikibugs>	 3Phabricator: phab-01 is broken: HTTPFutureCURLResponseStatus - https://phabricator.wikimedia.org/T88272#1007918 (10Qgil) 3NEW
[09:50:22] <hashar>	 vikasyaligar: I have no idea
[09:50:50] <hashar>	 vikasyaligar: we were in SF last week, I  am not sure when he is coming back home
[09:51:08] <hashar>	 vikasyaligar: he might have attended FOSDEM in Bruxelles as well, so he might be en route back home :D
[09:52:12] <vikasyaligar>	 hashar: ah ! OK 
[09:53:57] <vikasyaligar>	 hashar: I am not able to join #wikimedia-qa and also I am sure if zeljko hangouts in this channel or not 
[09:54:21] <hashar>	 vikasyaligar: #wikimedia-qa has been closed and redirects to this #wikimedia-releng
[09:55:10] <vikasyaligar>	 hashar: OK ! great :) 
[09:55:20] <vikasyaligar>	 hashar: thank you for the information :) 
[10:12:37] <grrrit-wm>	 (03PS2) 10Hashar: Set beta-scap-$datacenter timeout to 45 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/186406 (owner: 10Greg Grossmeier)
[10:14:01] <grrrit-wm>	 (03PS3) 10Hashar: Set beta-scap-$datacenter timeout to 45 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/186406 (owner: 10Greg Grossmeier)
[10:15:10] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Job updated:" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/186406 (owner: 10Greg Grossmeier)
[10:21:19] <grrrit-wm>	 (03Merged) 10jenkins-bot: Set beta-scap-$datacenter timeout to 45 minutes [integration/config] - 10https://gerrit.wikimedia.org/r/186406 (owner: 10Greg Grossmeier)
[10:30:42] <wikibugs>	 3Phabricator, Engineering-Community, Analytics-Tech-community-metrics: Monthly report of total / active Phabricator users - https://phabricator.wikimedia.org/T1003#1007960 (10Nemo_bis) Trying to get some comparable numbers pre and post-migration. It seems that the sum of MediaWiki (including VisualEditor and Mob...
[10:34:22] <wikibugs>	 3Phabricator, Engineering-Community, Wikimedia-Git-or-Gerrit: Migrate Gerrit project ownership request system (+2 rights) to Phabricator - https://phabricator.wikimedia.org/T86639#1007961 (10Qgil) Yes, in the new instructions we can encourage people to use thumbs up/down tokens in addition to whatever comments t...
[10:45:51] <grrrit-wm>	 (03PS2) 10Hashar: Add Unicodesnowman to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/184578 (owner: 10Gergő Tisza)
[10:53:16] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Add Unicodesnowman to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/184578 (owner: 10Gergő Tisza)
[10:54:14] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add Unicodesnowman to the CI whitelist [integration/config] - 10https://gerrit.wikimedia.org/r/184578 (owner: 10Gergő Tisza)
[10:54:40] <grrrit-wm>	 (03CR) 10Hashar: "Deployed!" [integration/config] - 10https://gerrit.wikimedia.org/r/184578 (owner: 10Gergő Tisza)
[10:58:47] <grrrit-wm>	 (03PS2) 10Hashar: Whitelist UltrasonicNXT (Adam Carter) [integration/config] - 10https://gerrit.wikimedia.org/r/186019 (owner: 10Legoktm)
[11:10:20] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Whitelist UltrasonicNXT (Adam Carter) [integration/config] - 10https://gerrit.wikimedia.org/r/186019 (owner: 10Legoktm)
[11:11:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Whitelist UltrasonicNXT (Adam Carter) [integration/config] - 10https://gerrit.wikimedia.org/r/186019 (owner: 10Legoktm)
[11:15:35] <grrrit-wm>	 (03CR) 10Hashar: "Otherwise looks fine :)" [integration/config] - 10https://gerrit.wikimedia.org/r/186141 (https://phabricator.wikimedia.org/T63641) (owner: 10Sn1per)
[11:17:03] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] mwext-VisualEditor-qunit-karma: Make non-voting until it works (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[11:19:16] <grrrit-wm>	 (03PS2) 10Hashar: mediawiki/selenium to support rspec [integration/config] - 10https://gerrit.wikimedia.org/r/185550 
[11:20:13] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] mediawiki/selenium to support rspec [integration/config] - 10https://gerrit.wikimedia.org/r/185550 (owner: 10Hashar)
[11:21:13] <grrrit-wm>	 (03Merged) 10jenkins-bot: mediawiki/selenium to support rspec [integration/config] - 10https://gerrit.wikimedia.org/r/185550 (owner: 10Hashar)
[11:21:41] <grrrit-wm>	 (03CR) 10Hashar: "Deployed! happy rspec :-]" [integration/config] - 10https://gerrit.wikimedia.org/r/185550 (owner: 10Hashar)
[11:23:18] <grrrit-wm>	 (03Abandoned) 10Hashar: Fixed gem build warnings [selenium] - 10https://gerrit.wikimedia.org/r/172518 (owner: 10Zfilipin)
[11:25:06] <grrrit-wm>	 (03Abandoned) 10Hashar: regenerate rubocop todo [selenium] - 10https://gerrit.wikimedia.org/r/179460 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:14] <grrrit-wm>	 (03Abandoned) 10Hashar: change compact style module definition to nested (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176261 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:18] <grrrit-wm>	 (03Abandoned) 10Hashar: add basic documentation comments (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176275 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:25] <grrrit-wm>	 (03Abandoned) 10Hashar: add empty lines (per rubocop style) [selenium] - 10https://gerrit.wikimedia.org/r/176276 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:29] <grrrit-wm>	 (03Abandoned) 10Hashar: remove extra blank lines (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176277 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:33] <grrrit-wm>	 (03Abandoned) 10Hashar: use guard clause per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176279 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:37] <grrrit-wm>	 (03Abandoned) 10Hashar: disable cop in this instance as rvm requires no space [selenium] - 10https://gerrit.wikimedia.org/r/176282 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:50] <grrrit-wm>	 (03Abandoned) 10Hashar: remove redundant selfs (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176340 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:53] <grrrit-wm>	 (03Abandoned) 10Hashar: use english special globals per rubocop [selenium] - 10https://gerrit.wikimedia.org/r/176342 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:25:57] <grrrit-wm>	 (03Abandoned) 10Hashar: disable global cop around globals that can't easily be removed [selenium] - 10https://gerrit.wikimedia.org/r/176344 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:01] <grrrit-wm>	 (03Abandoned) 10Hashar: remove comma after last item of hash (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/179461 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:05] <grrrit-wm>	 (03Abandoned) 10Hashar: fullstop with method name (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/179462 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:08] <grrrit-wm>	 (03Abandoned) 10Hashar: remove space inside empty braces (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/179463 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:16] <grrrit-wm>	 (03Abandoned) 10Hashar: use &: style instead of block (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/179464 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:19] <grrrit-wm>	 (03Abandoned) 10Hashar: fix rubocop string literal offenses [selenium] - 10https://gerrit.wikimedia.org/r/176252 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:23] <grrrit-wm>	 (03Abandoned) 10Hashar: mark unused argument (rubocop fix) [selenium] - 10https://gerrit.wikimedia.org/r/176253 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:28] <grrrit-wm>	 (03Abandoned) 10Hashar: remove non applicable private (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176254 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:46] <grrrit-wm>	 (03Abandoned) 10Hashar: rubocop line length fixes [selenium] - 10https://gerrit.wikimedia.org/r/176255 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:52] <grrrit-wm>	 (03Abandoned) 10Hashar: add MediaWiki standard rubocop config [selenium] - 10https://gerrit.wikimedia.org/r/176256 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:26:57] <grrrit-wm>	 (03Abandoned) 10Hashar: change block comments to # comments (per rubocop) [selenium] - 10https://gerrit.wikimedia.org/r/176260 (https://phabricator.wikimedia.org/T75898) (owner: 10Stan)
[11:28:15] <grrrit-wm>	 (03PS3) 10Hashar: AbuseFilter requires the AntiSpoof extension [integration/config] - 10https://gerrit.wikimedia.org/r/185634 (https://phabricator.wikimedia.org/T84859) (owner: 1001tonythomas)
[11:29:05] <grrrit-wm>	 (03PS4) 10Hashar: mwconf: allow empty lines in extensions_load.txt [integration/jenkins] - 10https://gerrit.wikimedia.org/r/183884 
[11:29:50] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] mwconf: allow empty lines in extensions_load.txt [integration/jenkins] - 10https://gerrit.wikimedia.org/r/183884 (owner: 10Hashar)
[11:31:09] <grrrit-wm>	 (03Merged) 10jenkins-bot: mwconf: allow empty lines in extensions_load.txt [integration/jenkins] - 10https://gerrit.wikimedia.org/r/183884 (owner: 10Hashar)
[11:32:11] <grrrit-wm>	 (03CR) 10Hashar: "deployed with git-deploy" [integration/jenkins] - 10https://gerrit.wikimedia.org/r/183884 (owner: 10Hashar)
[11:35:11] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Jobs refreshed and passing :-)" [integration/config] - 10https://gerrit.wikimedia.org/r/185634 (https://phabricator.wikimedia.org/T84859) (owner: 1001tonythomas)
[11:40:27] <grrrit-wm>	 (03PS2) 10Hashar: Add AntiSpoof to the shared mw job [integration/config] - 10https://gerrit.wikimedia.org/r/187903 
[11:42:15] <grrrit-wm>	 (03Merged) 10jenkins-bot: AbuseFilter requires the AntiSpoof extension [integration/config] - 10https://gerrit.wikimedia.org/r/185634 (https://phabricator.wikimedia.org/T84859) (owner: 1001tonythomas)
[11:54:47] <wikibugs>	 3Code-Review, Engineering-Community: How to prioritize code review of patches submitted by volunteers - https://phabricator.wikimedia.org/T78768#1008065 (10Qgil) http://korma.wmflabs.org/browser/gerrit_review_queue.html has a graph about **Age of open changesets by affilation (monthly snapshot)**. According to t...
[12:14:42] <wmf-insecte>	 Project beta-scap-eqiad build #39936: FAILURE in 41 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39936/
[12:24:52] <wikibugs>	 3Beta-Cluster, operations: Minimize differences between beta and production (Tracking) - https://phabricator.wikimedia.org/T87220#1008115 (10yuvipanda)
[12:24:53] <wikibugs>	 3Beta-Cluster, operations: Set up an alert for unmerged changes in deployment-prep - https://phabricator.wikimedia.org/T87616#1008111 (10yuvipanda) 5Open>3Resolved YESSSS http://shinken.wmflabs.org/service/deployment-salt/Long%20lived%20cherry-picks%20on%20puppetmaster  alerts if there has been more than 1 c...
[12:33:21] <wikibugs>	 3Beta-Cluster, operations: Move scap puppet code into a module - https://phabricator.wikimedia.org/T87221#1008119 (10yuvipanda) It is in a module now (yay!), but needs some more tweaks before it can replace beta::scap
[12:34:56] <wmf-insecte>	 Yippee, build fixed!
[12:34:57] <wmf-insecte>	 Project beta-scap-eqiad build #39938: FIXED in 1 min 0 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39938/
[12:36:23] <wikibugs>	 3Continuous-Integration: AbuseFilter requires the AntiSpoof extension - https://phabricator.wikimedia.org/T84859#1008120 (1001tonythomas) 5Open>3Resolved a:301tonythomas
[12:46:44] <wikibugs>	 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1008133 (10yuvipanda) 3NEW
[13:02:10] <shinken-wm>	 RECOVERY - Puppet failure on deployment-elastic05 is OK: OK: Less than 1.00% above the threshold [0.0]  
[13:21:31] <wikibugs>	 3Beta-Cluster: Add mx-records for beta.wmflabs.org - https://phabricator.wikimedia.org/T88215#1008166 (10yuvipanda) Looks like mx records are maintained in LDAP.   ```root@virt1000:/home/yuvipanda# ldapsearch -x -D cn=proxyagent,ou=profile,dc=wikimedia,dc=org -b ou=hosts,dc=wikimedia,dc=org  -W dc=tools Enter LD...
[13:30:34] <wikibugs>	 3Beta-Cluster: Add mx-records for beta.wmflabs.org - https://phabricator.wikimedia.org/T88215#1008174 (10yuvipanda) Aaaand bam ```○ → dig   -t mx beta.wmflabs.org @virt1000.wikimedia.org  ; <<>> DiG 9.8.3-P1 <<>> -t mx beta.wmflabs.org @virt1000.wikimedia.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<...
[13:31:52] <wikibugs>	 3Beta-Cluster: Add mx-records for beta.wmflabs.org - https://phabricator.wikimedia.org/T88215#1008175 (10yuvipanda) 5Open>3Resolved a:3yuvipanda Done :) @andrewbogott is there a way to do this from wikitech?
[13:49:19] <wikibugs>	 3Beta-Cluster: PHP notices on eval.php on deployment-bastion - https://phabricator.wikimedia.org/T88283#1008186 (10yuvipanda) 3NEW
[14:44:35] <wikibugs>	 3Echo, MediaWiki-General-or-Unknown, VisualEditor, Release-Engineering: Get JQuery error "a is undefined" running browser tests locally for Firefox - https://phabricator.wikimedia.org/T87446#1008250 (10Cmcmahon) It's not exactly fixed.
[14:48:18] <wikibugs>	 3Echo, MediaWiki-General-or-Unknown, VisualEditor, Release-Engineering: Get JQuery error "a is undefined" running browser tests locally for Firefox - https://phabricator.wikimedia.org/T87446#1008251 (10Cmcmahon) It's not exactly fixed. Elena discovered that by not firing an explicit javascript 'onfocus' event, t...
[15:09:42] <wikibugs>	 3Quality-Assurance, Mobile-Web: Issue with Chrome driver with resizing window - https://phabricator.wikimedia.org/T88288#1008283 (10Jdlrobson) 3NEW
[15:22:06] <grrrit-wm>	 (03PS1) 10Hashar: Ensure extensions are loaded in a consistent order [integration/config] - 10https://gerrit.wikimedia.org/r/188052 
[15:22:21] <grrrit-wm>	 (03PS2) 10Hashar: Ensure extensions are loaded in a consistent order [integration/config] - 10https://gerrit.wikimedia.org/r/188052 
[15:22:58] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Ensure extensions are loaded in a consistent order [integration/config] - 10https://gerrit.wikimedia.org/r/188052 (owner: 10Hashar)
[15:25:01] <wikibugs>	 3Code-Review, Wikimedia-General-or-Unknown: Implement a sane code-review process for MediaWiki JS/CSS pages on Wikimedia sites - https://phabricator.wikimedia.org/T71445#1008323 (10Anomie) >>! In T71445#1005858, @Krinkle wrote: > Using Git does raise the barrier for contribution a little.  This is a concern, tho...
[15:27:38] <wikibugs>	 3Beta-Cluster: PHP notices on eval.php on deployment-bastion - https://phabricator.wikimedia.org/T88283#1008326 (10hashar)
[15:27:41] <wikibugs>	 3Beta-Cluster, Release-Engineering: Warnings when running `mwscript eval.php` on deployment-prep - https://phabricator.wikimedia.org/T88093#1008327 (10hashar)
[15:29:09] <grrrit-wm>	 (03Merged) 10jenkins-bot: Ensure extensions are loaded in a consistent order [integration/config] - 10https://gerrit.wikimedia.org/r/188052 (owner: 10Hashar)
[15:44:21] <grrrit-wm>	 (03PS2) 10Hashar: zuul: test 'recheck' behavior [integration/config] - 10https://gerrit.wikimedia.org/r/184967 
[15:44:26] <grrrit-wm>	 (03PS3) 10Hashar: zuul: test 'recheck' behavior [integration/config] - 10https://gerrit.wikimedia.org/r/184967 
[15:44:31] <grrrit-wm>	 (03PS2) 10Hashar: zuul: test check/test behavior [integration/config] - 10https://gerrit.wikimedia.org/r/184968 
[15:45:27] <grrrit-wm>	 (03CR) 10Hashar: "Thanks for the review QChris, will follow up on this later on :-}" [integration/config] - 10https://gerrit.wikimedia.org/r/184886 (https://phabricator.wikimedia.org/T64429) (owner: 10Hashar)
[16:25:52] <wikibugs>	 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1008399 (10greg) @chad ? :)
[16:26:42] <yuvipanda>	 greg-g: heh, I have so few meetings these days that I get excited for each of them :D
[16:26:49] <wikibugs>	 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1008402 (10greg) See also: {T87676}
[16:28:33] <James_F>	 greg-g: This edit signs your team up to supporting the LTS model; not sure if you want to weigh in? https://www.mediawiki.org/w/index.php?title=MediaWiki_1.19&diff=0&oldid=1227441 
[16:31:06] <greg-g>	 yuvipanda: I can help change that if you want :)
[16:31:12] <yuvipanda>	 greg-g: nooo
[16:31:12] <yuvipanda>	 :)
[16:32:20] <yuvipanda>	 greg-g: we should give volunteers sudo! tonythomas could've used sudo today on deployment-prep
[16:32:29] <yuvipanda>	 since we are not going to do commercial ssl certs there anyway...
[16:33:36] <greg-g>	 yuvipanda: https://phabricator.wikimedia.org/T71269
[16:34:30] <wikibugs>	 3Beta-Cluster: Give all members of the deployment-prep project sudo - https://phabricator.wikimedia.org/T71269#1008444 (10yuvipanda) @greg any objections?
[16:41:01] <wikibugs>	 3Beta-Cluster: Give all members of the deployment-prep project sudo - https://phabricator.wikimedia.org/T71269#1008462 (10greg) doit, but let it be known that this means we'll continually hit weird ssl-related issues that aren't seen in prod (and thus annoying as heck, and we'll (RelEng) need help diagnosing the...
[16:41:40] <grrrit-wm>	 (03CR) 10Jforrester: mwext-VisualEditor-qunit-karma: Make non-voting until it works (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[16:46:05] <greg-g>	 James_F: I didn't miss your note on that, I'm just mulling
[16:47:42] * ^demon|away stabs Comcast a few times
[16:48:06] <^demon|away>	 Thank you for finally deciding I'm worthy of one of your precious precious IP addresses and issuing me one.
[16:48:18] * ^demon|away has been sitting here all morning with 0.0.0.0 and no sign of the DHCP gods.
[16:53:29] <yuvipanda>	 greg-g: I wrote a couple of emails in response to yours
[16:55:00] <greg-g>	 yuvipanda: I saw :) added the points to the bottom of the etherpad
[16:59:47] <James_F>	 greg-g: Sure. :-)
[17:00:17] <Reedy>	 hmmm, meeeting...
[17:01:11] <greg-g>	 ^d: now that you have an IP, will you be able to make it?
[17:01:42] <JohnFLewis>	 Reedy: don't you just turn up and let people praise you for being awesome at those meetings?
[17:01:50] <^d>	 I am joining, uno momento por favor
[17:02:45] <wikibugs>	 3Beta-Cluster: Setup puppet exported resources to collect ssh host keys for beta - https://phabricator.wikimedia.org/T72792#1008489 (10bd808) >>! In T72792#1005979, @yuvipanda wrote: > I no longer see this cherry-picked on beta, and jenkins seems to be just as happy?  This is "fixed" for beta using a local hack...
[17:07:37] <wikibugs>	 3Wikimedia-Logstash, Release-Engineering: Icinga monitoring for elasticsearch doesn't notice OOM conditions - https://phabricator.wikimedia.org/T76090#1008508 (10bd808) p:5Triage>3Normal
[17:11:59] <grrrit-wm>	 (03CR) 10Cmcmahon: [C: 04-1] "See inline" (031 comment) [integration/config] - 10https://gerrit.wikimedia.org/r/186451 (owner: 10Jdlrobson)
[17:22:12] <chrismcmahon>	 wow bd808 I'm looking at all the stuff new in vagrant since the last time I used it, that's impressive
[17:22:44] <bd808>	 it gets better all the time :)
[17:23:01] <bd808>	 we have a bunch of things waiting for review right now too
[17:23:02] <greg-g>	 Got to admit it's getting better
[17:23:57] <wikibugs>	 3Phabricator: request for deletion: 'shell' project - https://phabricator.wikimedia.org/T87623#1008532 (10Aklapper) Any other comments? If not I'll go ahead and archive that tag.
[17:23:58] <marxarelli>	 yeah, and if i can get a merge from the fakefs maintainers, we'll have /some/ test coverage
[17:24:32] <bd808>	 real developers test on other people's workstations
[17:24:52] <bd808>	 +2 and look for complaints. works every time ;)
[17:27:16] <marxarelli>	 ha! that made me think of the budweisser ad that ran yesterday: "real men drinks suds"
[17:30:16] <chrismcmahon>	 marxarelli: would it be OK to set up some pair-programming time later this week (and maybe into the future)? I have a Ruby thing I haven't figured out, and I'd like to start applying EAL places. 
[17:37:34] <chrismcmahon>	 I even updated the documentation :-)
[17:38:21] <marxarelli>	 chrismcmahon: yeah, let's schedule something
[17:38:40] <marxarelli>	 tomorrow would work for me
[17:39:20] <chrismcmahon>	 marxarelli: I think I have to do VE stuff tomorrow. How about Thursday? I know I'll be prepared by then. 
[17:39:30] <marxarelli>	 chrismcmahon: sounds good
[17:42:07] <wikibugs>	 3Phabricator: request for deletion: 'shell' project - https://phabricator.wikimedia.org/T87623#1008557 (10Aklapper) >>! In T87623#1002652, @Dzahn wrote: > Btw, what is a Site-request. Let's quesiton that as well :) Is that an alias for Team Platform Engineering?  Different task please. But from experience, there...
[17:53:26] <grrrit-wm>	 (03PS12) 10KartikMistry: WIP: Add generic npm-set-env to fix npm on */deploy repos [integration/config] - 10https://gerrit.wikimedia.org/r/184609 
[17:54:53] <wikibugs>	 3Beta-Cluster, operations: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#1008579 (10Joe) I have manually converted one host to use www-data, and took it out of rotation. A smoke test shows it working correctly. To make the conversion happen, I created a series of puppet patches...
[17:56:02] <wikibugs>	 3Flow, VisualEditor, Continuous-Integration: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#1008582 (10EBernhardson)
[17:58:23] <yuvipanda>	 greg-g: I didn't agree to another weekly meeting yet, did I?
[17:58:51] <greg-g>	 yuvipanda: no, not *yet*.
[17:58:58] <greg-g>	 :)
[18:01:04] <wikibugs>	 3Flow, VisualEditor, Continuous-Integration: Flow tests fails to run with VisualEditor installed - https://phabricator.wikimedia.org/T86920#1008598 (10EBernhardson)
[18:01:41] <grrrit-wm>	 (03PS4) 10Legoktm: Load extensions using wfLoadExtensions() if possible [integration/jenkins] - 10https://gerrit.wikimedia.org/r/184003 (https://phabricator.wikimedia.org/T86359) 
[18:01:51] <marxarelli>	 ok, heading in to the office
[18:03:41] <greg-g>	 https://phabricator.wikimedia.org/T88298
[18:03:44] <greg-g>	 will get to it in a bit
[18:08:29] <wikibugs>	 3MediaWiki-Core-Team, MediaWiki-Configuration, Continuous-Integration: Update jenkins for extension registration changes - https://phabricator.wikimedia.org/T86359#1008652 (10Legoktm)
[18:11:39] <wikibugs>	 3Phabricator: Clearer instructions in the Phabricator login page - https://phabricator.wikimedia.org/T545#1008664 (10Aklapper) I have no idea how much of a diff it would be to maintain this (@mmodell: any idea?), but moving the "Login or Register: MediaWiki" button above the "Login or Register with LDAP" one, ad...
[18:16:28] <wikibugs>	 3MediaWiki-extensions-ContentTranslation, Continuous-Integration, ContentTranslation-Deployments: Enable Debian CI tests on all Apertium packages - https://phabricator.wikimedia.org/T87607#1008678 (10KartikMistry)
[18:16:56] <wikibugs>	 3MediaWiki-extensions-ContentTranslation, Continuous-Integration, ContentTranslation-Deployments: Enable Debian CI tests on all Apertium packages - https://phabricator.wikimedia.org/T87607#1008679 (10KartikMistry) a:3KartikMistry
[18:17:05] <wikibugs>	 3Continuous-Integration, Apex: All jenkins tests attempts in mediawiki/skins/apex fail with "This change was unable to be automatically merged with the current state of the repository." - https://phabricator.wikimedia.org/T88302#1008680 (10matmarex) 3NEW
[18:18:54] <wikibugs>	 3Phabricator: Enabling Herald - https://phabricator.wikimedia.org/T630#1008690 (10Aklapper) I'd be interested in your specific situation and usecase then if the current situation makes following "impossible".
[18:32:51] <yuvipanda>	 ^d: twentyafterfour I think https://phabricator.wikimedia.org/T87003 is a pre-requisite before we create any instances
[18:32:57] <yuvipanda>	 because fuck 2g /vars
[18:33:23] <^d>	 I've got a lead on why the logs blew up.
[18:34:05] <yuvipanda>	 ^d: wheeee!
[18:34:50] <wikibugs>	 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1008769 (10Chad) apifeature usage started spamming the log, made it grow to almost 2G.   {P254}
[18:34:59] <wmf-insecte>	 Project beta-scap-eqiad build #39975: FAILURE in 57 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39975/
[18:37:23] <wikibugs>	 3Phabricator: Prompt for batch "Assign" action is incorrect - https://phabricator.wikimedia.org/T88155#1008777 (10Aklapper) > Enter "none" and blur the textbox.  I have no idea how to blur a textbox. If I click outside or tab out it either picks up the string or empties the input field again. Firefox 35.
[18:37:34] <wikibugs>	 3Beta-Cluster: /var/log full on deployment-elasticsearch* hosts - https://phabricator.wikimedia.org/T88280#1008779 (10Chad) {P255}  Which leads to red status, obvs.
[18:38:09] <^d>	 manybubbles: any clue what's going on here ^
[18:38:20] <^d>	 apifeatureusage shards seem...busted
[18:38:52] <manybubbles>	 boo
[18:39:28] <twentyafterfour>	 yuvipanda: agreed, fuck 2g /var
[18:40:03] <wikibugs>	 3Continuous-Integration, Apex: All jenkins tests attempts in mediawiki/skins/apex fail with "This change was unable to be automatically merged with the current state of the repository." - https://phabricator.wikimedia.org/T88302#1008787 (10Paladox) What about adding an ignor file .gitignor file to the project.
[18:40:04] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce build #290: FAILURE in 33 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_8.1-internet_explorer-11-sauce/290/
[18:41:37] <yuvipanda>	 ^d: manybubbles I uncherry-picked the apifeatureusage stuff earlier today
[18:41:40] <yuvipanda>	 did that fuck things up?
[18:41:55] <manybubbles>	 I'm not paying enough attention to be sure
[18:42:08] <wikibugs>	 3WMF-Legal, Phabricator: Policy for inappropriate avatars? - https://phabricator.wikimedia.org/T1180#1008795 (10Aklapper) I don't see anything really actionable here either... Personally I'd decline this.
[18:42:28] <^d>	 [2015-01-27 20:58:42,452][WARN ][indices.cluster          ] [deployment-elastic05] [apifeatureusage-2014.12.24][0] failed to start shard
[18:42:39] <^d>	 yuvipanda: First failure I see
[18:42:54] <yuvipanda>	 yeah, that's way before I did anything
[18:42:57] <yuvipanda>	 is that a tuesday
[18:43:03] <^d>	 Yes
[18:43:04] <yuvipanda>	 if that was last tuesday that was the great labs restart
[18:43:14] <^d>	 That'd explain it
[18:44:16] <^d>	 So I'd say just nuke the indexes and recreate.
[18:44:23] <^d>	 And things should quiet down
[18:46:17] <wikibugs>	 3Echo, MediaWiki-General-or-Unknown, Release-Engineering: Get JQuery error "a is undefined" running Echo browser tests locally for Firefox - https://phabricator.wikimedia.org/T87873#1008832 (10Mattflaschen) p:5Unbreak!>3High
[18:48:50] <wikibugs>	 3WMF-Legal, Phabricator: Policy for inappropriate avatars? - https://phabricator.wikimedia.org/T1180#1008854 (10Krenair) 5Open>3declined a:3Krenair Let's mark this declined unless #WMF-Legal wants to do something.
[18:49:58] <wikibugs>	 3Beta-Cluster, operations: Renumber apache user/group to uid=48 - https://phabricator.wikimedia.org/T78076#1008872 (10greg) Ideally before, yes.
[18:49:59] <wmf-insecte>	 Project browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #453: FAILURE in 11 min: https://integration.wikimedia.org/ci/job/browsertests-Core-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/453/
[18:51:19] <wikibugs>	 3MediaWiki-Core-Team, operations, Deployment-Systems, Release-Engineering: Update servers in scap rsync proxy pool - https://phabricator.wikimedia.org/T1342#1008884 (10bd808) 5Open>3Resolved
[18:52:30] <^d>	 yuvipanda: I dropped the affected indexes.
[18:52:37] <^d>	 Just means there will be a gap in beta stats.
[18:52:47] <yuvipanda>	 ^d: \o/
[18:52:56] <^d>	 I'll prune those logs and we should be fine
[18:52:57] <yuvipanda>	 well, I dropped apifeatureusage from the puppetmaster anyway
[18:52:59] <yuvipanda>	 cool
[18:53:04] <yuvipanda>	 ty ^d
[18:54:31] <manybubbles>	 sorry - got distracted by meeting.  everything "fixed" now?
[18:54:41] <^d>	 root@deployment-elastic06:/var/log# du -sh .
[18:54:41] <^d>	 249M	.
[18:55:02] <wmf-insecte>	 Yippee, build fixed!
[18:55:03] <wmf-insecte>	 Project beta-scap-eqiad build #39977: FIXED in 1 min 3 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/39977/
[18:55:33] <yuvipanda>	 cool
[18:55:35] <yuvipanda>	 but seriously
[18:55:38] <yuvipanda>	 fuck 2G /vars
[18:55:39] <yuvipanda>	 although
[18:55:40] <yuvipanda>	 I must admit
[18:55:41] <^d>	 yeah
[18:55:41] <^d>	 lol
[18:55:45] <yuvipanda>	 that if we go to unified 20G vars
[18:55:51] <yuvipanda>	 and a runaway log file fills up /
[18:55:56] <yuvipanda>	 I wonder if we can still ssh into it
[18:55:57] <greg-g>	 we'll just see the problems later :)
[18:55:58] <yuvipanda>	 or if it's toast
[18:56:05] <yuvipanda>	 we'll have alerting still
[18:56:12] <yuvipanda>	 and 10% warn on 20G is 2G empty space
[18:56:18] <yuvipanda>	 while 10% on 2G is only 200M
[18:56:32] <twentyafterfour>	 how about 8g var?
[18:56:41] <yuvipanda>	 let's kill separate /vars
[18:56:52] <yuvipanda>	 and just respond to alerts instead :)
[18:56:58] <greg-g>	 WWPD (What Would Prod Do?)
[18:57:03] <yuvipanda>	 heh
[18:57:07] <yuvipanda>	 they would respond to alerts!
[18:57:16] <yuvipanda>	 this kept happening with ocg, for example
[18:57:30] <Nikerabbit>	 hmm I think this issue also caused https://phabricator.wikimedia.org/T88118 
[18:58:08] <^d>	 Why does df still say partition is still like 98% full?
[18:58:21] <^d>	 du disagrees.
[18:58:44] <yuvipanda>	 ^d: ah, hahahahahaaaaaaaaaaaaaaa
[18:58:51] <yuvipanda>	 ^d: it means something is keeping file handles open
[18:58:57] <yuvipanda>	 df still counts that
[18:58:58] <twentyafterfour>	 ^ that
[18:58:59] <yuvipanda>	 du does not
[18:59:03] <^d>	 Ahh
[18:59:08] <^d>	 Probably have to kick elasticsearch
[18:59:11] <yuvipanda>	 yeah
[18:59:20] <twentyafterfour>	 if you rm a file that is open by some process it doesn't go away until the handle is closecd
[18:59:24] * ^d grabs his boots
[18:59:36] <^d>	 yeahhh, I didn't know df counted that.
[18:59:37] <^d>	 #til
[19:00:08] <^d>	  /dev/mapper/vd-log  2.0G   21M  1.8G   2% /var/log
[19:00:11] <^d>	 Much nicer
[19:07:01] <wmf-insecte>	 Project browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce build #465: FAILURE in 20 min: https://integration.wikimedia.org/ci/job/browsertests-VisualEditor-test2.wikipedia.org-linux-chrome-sauce/465/
[19:12:53] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-elastic05 is OK: OK: All targets OK  
[19:20:13] <wikibugs>	 3Staging: Setup staging-palladium as puppetmaster and saltmaster - https://phabricator.wikimedia.org/T88304#1008978 (10yuvipanda) I should also probably make some hiera setup to make it possible for every instance on one project to have salt and puppetmasters automatically set.
[19:20:42] <yuvipanda>	 ^d: should staging have jenkins access from the start?
[19:20:44] * yuvipanda votes no
[19:20:53] <yuvipanda>	 twentyafterfour: greg-g ^
[19:20:56] <greg-g>	 as in?
[19:21:00] <yuvipanda>	 as in
[19:21:07] <greg-g>	 Jenkins will need to run scap etc
[19:21:11] <yuvipanda>	 oh
[19:21:11] <yuvipanda>	 hmm
[19:21:12] <yuvipanda>	 right
[19:21:20] <^d>	 No it won't. I think we could do it from the tin side.
[19:21:40] <yuvipanda>	 we really need to explicitly think about how to get jenkins access.
[19:21:49] * greg-g lets you all figure it out, just make sure we can update code automagically :)
[19:21:50] <yuvipanda>	 contint module and the beta stuff need to *carefully* mingle
[19:21:55] <^d>	 Simple, we don't :)
[19:22:17] <yuvipanda>	 so let’s ignore jenkins for now, and then figure that out later?
[19:22:27] <yuvipanda>	 as in, figure out the auto update stuff later? :)
[19:22:47] <^d>	 I think it won't be that hard to port what jenkins does to doing it in some cron from tin/etc.
[19:22:56] <yuvipanda>	 it used to be a cron!
[19:23:00] <yuvipanda>	 I like crons
[19:23:01] <yuvipanda>	 :)
[19:23:03] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-elastic06 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [43200.0]  
[19:23:27] <greg-g>	 why not leave it in Jenkins? we get alerts and build history
[19:23:49] * greg-g said he would let you figure it out...
[19:23:59] <yuvipanda>	 I would like to clean up some of the contint puppet code too, if we want to do that
[19:24:02] <yuvipanda>	 and that’s somewhat… scary?
[19:24:07] <yuvipanda>	 it does scary things
[19:24:31] <^d>	 greg-g: Which are nice, but we don't use the same thing for alerts and such in prod.
[19:24:40] <yuvipanda>	 it feels a bit like scap / trebuchet, and I get this  feeling I should stay away from it :)
[19:24:49] <yuvipanda>	 also what ^d  said
[19:25:33] <greg-g>	 there just might be some knock on effects I'm not thinking of cleary (I'm "paying attention" in the ops meeting, like yuvi is)
[19:25:42] <yuvipanda>	 greg-g: ;)
[19:25:50] <yuvipanda>	 greg-g: forgot you were still here
[19:26:27] <greg-g>	 I'm everywhere
[19:26:59] <yuvipanda>	 :D
[19:27:54] <^d>	 greg-g: https://www.youtube.com/watch?v=HLCasyAh7ic
[19:28:06] * greg-g waits to watch that
[19:28:21] <yuvipanda>	 greg-g: no, put it on now and play it loudly!
[19:28:57] <grrrit-wm>	 (03PS1) 10Dduvall: Step definitions refactored for the EAL [selenium] - 10https://gerrit.wikimedia.org/r/188102 
[19:36:01] <shinken-wm>	 RECOVERY - Puppet failure on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [0.0]  
[19:38:03] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-elastic06 is OK: OK: Less than 1.00% above the threshold [3600.0]  
[19:39:36] <wikibugs>	 3Continuous-Integration, Apex: All jenkins tests attempts in mediawiki/skins/apex fail with "This change was unable to be automatically merged with the current state of the repository." - https://phabricator.wikimedia.org/T88302#1009091 (10hashar) 5Open>3Resolved a:3hashar Some fault happened on zuul merge...
[19:40:23] <shinken-wm>	 PROBLEM - Puppet failure on deployment-pdf01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]  
[19:44:11] <grrrit-wm>	 (03CR) 10Krinkle: "This doesn't make sense. Please verify ASAP." [integration/config] - 10https://gerrit.wikimedia.org/r/188052 (owner: 10Hashar)
[19:44:29] <yuvipanda>	 greg-g: progress!
[19:45:04] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-elastic07 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [43200.0]  
[20:10:21] <twentyafterfour>	 more progress
[20:11:14] <twentyafterfour>	 success criteria: once shinken-wm is reporting critical for every deployment-* host, we've won
[20:13:46] <wikibugs>	 3Phabricator: Fatal error (30 seconds timeout) upon certain maniphest search in a component when not logged in - https://phabricator.wikimedia.org/T87739#1009294 (10Aklapper) a:3mmodell @mmodell: Could you take a look at this?  Is this just about increasing some timeout setting somewhere, or is this more compl...
[20:25:43] <twentyafterfour>	 so how hard would it be to log every request time and status code seen by varnish? assuming varnish isn't the failure point, those logs should provide a fairly good picture of service availability and quality 
[20:28:40] <twentyafterfour>	 the ratio of success to error response codes would be a good 'percentage availability' metric and the average response times would give us a rough high-level performance metric 
[20:32:16] <twentyafterfour>	 http://vuksan.com/linux/ganglia/index.html#varnish_stats
[20:32:39] <wikibugs>	 3MediaWiki-extensions-ZeroPortal, MediaWiki-extensions-ZeroBanner, Continuous-Integration: Jenkins must not load ZeroPortal before ZeroBanner - https://phabricator.wikimedia.org/T88015#1009338 (10hashar) 5Resolved>3Open We should be able to load extensions in random order, for example by using scandir() whic...
[20:41:52] <grrrit-wm>	 (03CR) 10Hashar: "When using 'check experimental' on the AntiSpoof extension, it ends up being loaded last and that has a side effect with some other extens" [integration/config] - 10https://gerrit.wikimedia.org/r/188052 (owner: 10Hashar)
[20:47:59] <wmf-insecte>	 Yippee, build fixed!
[20:48:00] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #444: FIXED in 29 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/444/
[20:50:28] <chrismcmalunch>	 hi ^d are you aware of anything strange going on for File searches for beta cluster enwiki? like maybe returning incomplete results or not finishing somehow? I'm seeing behavior in VE as if a search for media never completes, unlike on mw.o
[20:50:59] <^d>	 file searches are slow because NFS.
[20:52:58] <chrismcmahon>	 OK. Still looks like a bug of some sort, I'll keep poking around
[21:01:42] <grrrit-wm>	 (03PS3) 10Hashar: Whitelist pywikibot people with +2 rights [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) (owner: 10Mpaa)
[21:03:57] <grrrit-wm>	 (03CR) 10Hashar: [C: 031] "The previous patchsets sorted the entries which make it almost impossible to review the patch." [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) (owner: 10Mpaa)
[21:06:57] <grrrit-wm>	 (03CR) 10Hashar: "Thanks! I have updated the wiki doc https://www.mediawiki.org/w/index.php?title=Continuous_integration/Documentation_generation&diff=13899" [integration/config] - 10https://gerrit.wikimedia.org/r/187797 (owner: 10Krinkle)
[21:07:39] <wikibugs>	 3Phabricator, operations: Add @emailbot to #operations - https://phabricator.wikimedia.org/T87611#1009432 (10chasemp)
[21:08:49] <wikibugs>	 3Phabricator, operations: Add @emailbot to #operations - https://phabricator.wikimedia.org/T87611#995092 (10chasemp) @RobH, yes?  Anyone have objections to adding @emailbot to #operations so it can relay comments to private issues?
[21:09:14] <wikibugs>	 3Continuous-Integration: Migrate all jobs depending on Zuul git repos out of production slaves - https://phabricator.wikimedia.org/T86659#1009437 (10hashar)
[21:09:15] <wikibugs>	 3Continuous-Integration: Migrate JSDuck jobs to Ubuntu Trusty - https://phabricator.wikimedia.org/T86174#1009438 (10hashar)
[21:10:37] <grrrit-wm>	 (03CR) 10Mpaa: [C: 031] Whitelist pywikibot people with +2 rights [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) (owner: 10Mpaa)
[21:10:48] <wikibugs>	 3Continuous-Integration: Migrate all jobs depending on Zuul git repos out of production slaves - https://phabricator.wikimedia.org/T86659#973540 (10hashar)
[21:19:04] <grrrit-wm>	 (03PS2) 10Krinkle: mwext-VisualEditor-qunit-karma: Make non-voting until it works [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[21:20:43] <grrrit-wm>	 (03CR) 10Jforrester: [C: 031] mwext-VisualEditor-qunit-karma: Make non-voting until it works [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[21:21:41] <grrrit-wm>	 (03CR) 10Krinkle: [C: 032] mwext-VisualEditor-qunit-karma: Make non-voting until it works [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[21:23:37] <grrrit-wm>	 (03Merged) 10jenkins-bot: mwext-VisualEditor-qunit-karma: Make non-voting until it works [integration/config] - 10https://gerrit.wikimedia.org/r/188024 (owner: 10Jforrester)
[21:29:36] <wikibugs>	 3Phabricator: Drop 3 empty repos - https://phabricator.wikimedia.org/T88330#1009545 (10Chad)
[21:31:00] <wikibugs>	 3Phabricator: Drop 3 empty repos - https://phabricator.wikimedia.org/T88330#1009547 (10chasemp) 5Open>3Resolved done dude
[21:33:53] <wikibugs>	 3Continuous-Integration: Add VisualEditor to WikiHiero CI dependencies - https://phabricator.wikimedia.org/T88331#1009560 (10Jdforrester-WMF)
[21:39:14] <Krinkle>	 !log Deployed I94f65b56368 and reloading Zuul
[21:39:17] <qa-morebots>	 Logged the message, Master
[21:39:55] <wikibugs>	 3Continuous-Integration: Add VisualEditor to Jenkins dependencies for WikiHiero - https://phabricator.wikimedia.org/T88331#1009617 (10Krinkle)
[21:53:38] <grrrit-wm>	 (03CR) 10Krinkle: [C: 031] "Maybe just remove the lines entirely? We have version control for old code. No need to keep them commented out. They still run per-merge i" [integration/config] - 10https://gerrit.wikimedia.org/r/187904 (owner: 10Hashar)
[21:55:13] <wikibugs>	 3Release-Engineering: Release MW 1.24.2 and 1.23.9 tarballs - https://phabricator.wikimedia.org/T88120#1009644 (10Aklapper)
[22:05:25] <grrrit-wm>	 (03CR) 10John Vandenberg: [C: 031] Whitelist pywikibot people with +2 rights [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) (owner: 10Mpaa)
[22:08:51] <wikibugs>	 3Beta-Cluster: Migrate beta cluster jobrunner to HAT - https://phabricator.wikimedia.org/T87214#1009706 (10mmodell)
[22:10:49] <spagewmf>	 bd808: https://www.mediawiki.org/wiki/Download_from_Git#Fetch_external_libraries recommends two options, which is one too many :)
[22:10:49] <grrrit-wm>	 (03CR) 10Ricordisamoa: [C: 031] Whitelist pywikibot people with +2 rights [integration/config] - 10https://gerrit.wikimedia.org/r/186611 (https://phabricator.wikimedia.org/T87413) (owner: 10Mpaa)
[22:12:19] <spagewmf>	 bd808: which one does MW-Vagrant use?
[22:12:27] <bd808>	 composer
[22:19:30] <wikibugs>	 3Continuous-Integration, Analytics-EventLogging: Set up jsduck test job for EventLogging - https://phabricator.wikimedia.org/T88343#1009758 (10Krinkle) 3NEW
[22:24:05] <wikibugs>	 3Continuous-Integration: mediawiki-phpunit-hhvm fails sometimes with Zuul-cloner: IOError: Unable to obtain lock for .git/HEAD.lock - https://phabricator.wikimedia.org/T87591#1009767 (10Krinkle)
[22:24:25] <wikibugs>	 3Continuous-Integration: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1009769 (10Krinkle)
[22:24:32] <wikibugs>	 3Continuous-Integration: Jenkins: Jobs should not be affected by .git/index.lock of previous run - https://phabricator.wikimedia.org/T49638#1009770 (10Krinkle)
[22:24:43] <wikibugs>	 3Continuous-Integration: mediawiki-phpunit-hhvm fails sometimes with Zuul-cloner: IOError: Unable to obtain lock for .git/HEAD.lock - https://phabricator.wikimedia.org/T87591#994738 (10Krinkle)
[22:26:31] <wikibugs>	 3Continuous-Integration: mediawiki-phpunit-hhvm fails sometimes with Zuul-cloner: IOError: Unable to obtain lock for .git/HEAD.lock - https://phabricator.wikimedia.org/T87591#1009783 (10Krinkle)
[22:26:33] <wikibugs>	 3Continuous-Integration: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#1009784 (10Krinkle)
[22:26:56] <wikibugs>	 3Continuous-Integration: Zuul-cloner failing to acquire .git/config lock sometimes - https://phabricator.wikimedia.org/T86730#975087 (10Krinkle) https://integration.wikimedia.org/ci/job/mediawiki-extensions-zend/1938/console  ``` 22:09:41 ERROR:zuul.Repo:Unable to initialize repo for https://gerrit.wikimedia.org...
[22:31:38] <spagewmf>	 bd808: thx. `composer install --no-dev` creates a smaller core/vendor: no monolog, Plancake, README.md, ruflin that are in vendor.git.  Are those only needed in WMF's complex cluster?
[22:31:46] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#1009801 (10RobH)
[22:31:51] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957717 (10RobH)
[22:32:38] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957717 (10RobH)
[22:33:16] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957717 (10RobH)
[22:33:36] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#1009823 (10RobH) 5Open>3Resolved As the three days has long since passed, I'm merging https://gerrit.wikimedia.org/r/#/c/183062 live...
[22:33:43] <legoktm[NE]>	 spagewmf: monolog is an optional core dependency, plancake is for the BounceHandler extension. vendor also has elastica for cirrussearch
[22:34:30] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#1009827 (10RobH) 5Resolved>3Open
[22:34:44] <wikibugs>	 3Continuous-Integration, Ops-Access-Requests: Make sure relevant RelEng people have access to gallium (Chris M, Dan, Mukunda, Zeljko) - https://phabricator.wikimedia.org/T85936#957717 (10RobH)
[22:36:40] <wikibugs>	 3Beta-Cluster: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1009837 (10Cmcmahon) I mentioned this to Chad, who pointed out that media/file storage on beta cluster is still on an NFS filesystem so searching media files always has be...
[22:36:56] <wikibugs>	 3Phabricator: Registration for phab-01.wmflabs.org broken - https://phabricator.wikimedia.org/T88346#1009838 (10Tgr) 3NEW
[22:37:59] <wikibugs>	 3Beta-Cluster, VisualEditor: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1009854 (10Cmcmahon)
[22:40:41] <wikibugs>	 3Code-Review, Engineering-Community: How to prioritize code review of patches submitted by volunteers - https://phabricator.wikimedia.org/T78768#1009869 (10Tgr) >>! In T78768#1008065, @Qgil wrote: > http://korma.wmflabs.org/browser/gerrit_review_queue.html has a graph about **Age of open changesets by affilation...
[22:48:20] <wikibugs>	 3Beta-Cluster, VisualEditor: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1009916 (10greg) >>! In T87676#1009837, @Cmcmahon wrote: > so searching media files always has been and always will be slow.   (Until we setup a swift machin...
[23:10:44] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree.value (<11.11%)  
[23:13:30] <wikibugs>	 3Beta-Cluster, VisualEditor: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1010002 (10Cmcmahon) >>! In T87676#1009916, @greg wrote: >>>! In T87676#1009837, @Cmcmahon wrote: >> so searching media files always has been and always will...
[23:14:51] <wikibugs>	 3Beta-Cluster, VisualEditor: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1010004 (10Jdforrester-WMF) >>! In T87676#1010002, @Cmcmahon wrote: > True, but I am still concerned (albeit in a hand-wavy non-specific way) about non-Wikip...
[23:15:35] <wikibugs>	 3Beta-Cluster, VisualEditor: Searching for images in VisualEditor with Firefox is painfully slow in Beta Cluster - https://phabricator.wikimedia.org/T87676#1010005 (10Jdforrester-WMF) The timeout is 100s, and according to @Mooeypoo hasn't changed in the past year or so. The new interface uses roughly the same AP...