[00:18:47] <matanya>	 greg-g: have a moment ?
[00:19:52] <greg-g>	 matanya: on IRC yeah, I'm wfh today
[00:20:04] <matanya>	 yeah, irc works
[00:20:27] <matanya>	 i wanted to ask to push recent wmf branches to the fdc wiki
[00:20:39] <matanya>	 i am not sure to what group it belongs, if at all
[00:20:56] <greg-g>	 you mean it isn't being updated along with the train?
[00:21:07] <matanya>	 but it is missing some nice features the entire cluster got in the past months 
[00:21:13] <matanya>	 yes, i fear so
[00:21:25] * greg-g is confuszored
[00:21:32] <matanya>	  	1.27.0-wmf.6 (796411d)
[00:21:32] <matanya>	 10:24, 17 November 2015
[00:21:49] <greg-g>	 that's today :)
[00:21:50] <matanya>	 in special:version
[00:22:18] * greg-g is now more confused
[00:22:22] <matanya>	 yes, but for example, the split between echo  and notifications is not there
[00:22:49] <greg-g>	 oh, same on meta, that might be a settings issue?
[00:23:11] <matanya>	 ah, didn't think of that
[00:24:04] <matanya>	 greg-g: thanks, i'll compare the diff's and submit patches for the most annoying misses 
[00:24:10] * greg-g nods
[00:24:11] <greg-g>	 np
[00:30:39] <legoktm>	 matanya: the messages pane doesn't show up until someone leaves you a message.
[00:31:04] <matanya>	 legoktm: thanks, i meant gui-wise
[00:31:32] <legoktm>	 yes, the second tab won't show up until someone writes something on your talk page
[00:31:42] <legoktm>	 tab/icon/badge/flyout/popup
[00:31:49] <legoktm>	 whatever you call it :P
[00:32:12] <matanya>	 confirmed, thanks
[00:33:07] * matanya goes to write some recommendation text instead.  
[00:35:00] <greg-g>	 legoktm: doohicky
[01:29:27] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.94 ms
[01:57:18] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[03:34:24] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[03:37:14] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 1.00 ms
[03:42:17] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[03:53:47] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Graphoid: Graphoid does not deploy sync to deployment-sca01 - https://phabricator.wikimedia.org/T118929#1812944 (10Yurik) 3NEW
[03:54:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Graphoid: Graphoid does not deploy sync to deployment-sca01 - https://phabricator.wikimedia.org/T118929#1812951 (10Yurik)
[04:28:22] <grrrit-wm>	 (03PS1) 10Tim Starling: Note results of libdc1394 bug investigation [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/253827 
[04:44:38] <grrrit-wm>	 (03CR) 10Tim Starling: [C: 032 V: 032] Note results of libdc1394 bug investigation [integration/uprightdiff] - 10https://gerrit.wikimedia.org/r/253827 (owner: 10Tim Starling)
[05:29:40] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #606: 04FAILURE in 27 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/606/
[06:39:26] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK
[08:33:27] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce build #786: 04FAILURE in 23 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-os_x_10.9-safari-sauce/786/
[08:43:59] <grrrit-wm>	 (03CR) 10Ori.livneh: [C: 031] Configure thumbor/exif-optimizer [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[09:00:24] <hashar>	 zeljkof: I am around more or less
[09:00:50] <zeljkof>	 hashar: more or less? :)
[09:02:36] <zeljkof>	 hashar: can not join the hangout?
[09:05:15] <hashar>	 nature's call
[09:05:15] <hashar>	  :D
[09:20:45] <grrrit-wm>	 (03PS2) 10Hashar: Added building the gem to `rake test` (CI entry point for Ruby) [selenium] - 10https://gerrit.wikimedia.org/r/252693 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:30:48] <grrrit-wm>	 (03PS1) 10Hashar: Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/253858 (https://phabricator.wikimedia.org/T114860) 
[09:30:56] <grrrit-wm>	 (03CR) 10Hashar: "restored with https://gerrit.wikimedia.org/r/253858" [integration/config] - 10https://gerrit.wikimedia.org/r/252690 (https://phabricator.wikimedia.org/T114860) (owner: 10Zfilipin)
[09:31:08] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/253858 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar)
[09:32:02] <grrrit-wm>	 (03Merged) 10jenkins-bot: Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/253858 (https://phabricator.wikimedia.org/T114860) (owner: 10Hashar)
[09:32:15] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.48 ms
[09:32:56] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252698 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:33:29] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [selenium] - 10https://gerrit.wikimedia.org/r/252693 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:33:38] <grrrit-wm>	 (03Merged) 10jenkins-bot: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252698 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:38:52] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "rake-jessie pass \O/" [selenium] - 10https://gerrit.wikimedia.org/r/252693 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:47:29] <wmf-insecte>	 Project beta-scap-eqiad build #79039: 04FAILURE in 2 min 10 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79039/
[09:48:00] <grrrit-wm>	 (03Merged) 10jenkins-bot: Added building the gem to `rake test` (CI entry point for Ruby) [selenium] - 10https://gerrit.wikimedia.org/r/252693 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[09:51:29] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[09:54:47] <shinken-wm>	 PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[10:07:34] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1813263 (10zeljkofilipin)
[10:19:28] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.55 ms
[10:32:15] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.99 ms
[10:42:18] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[10:47:40] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[11:11:30] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms
[11:12:16] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 1.40 ms
[11:32:38] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[11:44:25] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[12:47:07] <wmf-insecte>	 Project browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce build #702: 04FAILURE in 1 min 7 sec: https://integration.wikimedia.org/ci/job/browsertests-PdfHandler-test2.wikipedia.org-linux-firefox-sauce/702/
[12:55:55] <shinken-wm>	 PROBLEM - Host deployment-mediawiki02 is DOWN: CRITICAL - Host Unreachable (10.68.16.127)
[12:58:00] <shinken-wm>	 RECOVERY - Host deployment-mediawiki02 is UP: PING OK - Packet loss = 0%, RTA = 1.07 ms
[13:01:06] <wmf-insecte>	 Project browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #852: 04FAILURE in 29 min: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/852/
[13:06:29] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.90 ms
[13:15:12] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[13:34:44] <zeljkof>	 hashar: any idea why rake-jessie ran for this commit? https://gerrit.wikimedia.org/r/#/c/236038/
[13:35:00] <hashar>	 zeljkof: because of zuul/layout.yaml ?
[13:35:02] <zeljkof>	 it was merged in september
[13:35:04] <hashar>	 have you looked at it?
[13:35:14] <hashar>	 oh
[13:35:22] <zeljkof>	 I mean, why would we run a job for already merged commit
[13:35:22] <hashar>	 probably manually enqueued to verify it is working properly
[13:35:30] <zeljkof>	 I see
[13:35:35] <hashar>	 I sometime trigger rechecks from the command line against the last change that got merged
[13:35:47] <hashar>	 on gallium that is with "zuul enqueue"
[13:36:15] <zeljkof>	 nevermind, I have just stumbled upon it and got confused
[13:38:19] <hashar>	 zeljkof: http://docs.openstack.org/infra/zuul/client.html#enqueue :D
[13:38:29] <hashar>	 so you can do:
[13:38:37] <zeljkof>	 hashar: thanks, was not aware of that
[13:38:52] <hashar>	 zuul enqueue --trigger gerrit --pipeline test --project mediawiki/core --change 12345,42
[13:39:14] <hashar>	 which would inject in Zuul scheduler the exact same thing as if 12345,42 entered the test pipeline (i.e. a new patch)
[13:40:23] <zeljkof>	 cool
[13:48:44] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 1.42 ms
[13:49:26] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.87 ms
[13:59:56] <grrrit-wm>	 (03PS2) 10Hashar: Configure thumbor/exif-optimizer [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[14:00:31] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Configure thumbor/exif-optimizer [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[14:01:23] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Configure thumbor/exif-optimizer [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[14:05:30] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "gate failed because of a network error preventing access to Jenkins API." [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[14:06:20] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[14:06:22] <grrrit-wm>	 (03Merged) 10jenkins-bot: Configure thumbor/exif-optimizer [integration/config] - 10https://gerrit.wikimedia.org/r/253668 (https://phabricator.wikimedia.org/T111722) (owner: 10Gilles)
[14:08:09] <hashar>	 gilles: merci d'avoir pris le temps de configurer le CI pour thumblor :-)
[14:08:26] <gilles>	 hashar: merci pour les reviews!
[14:19:27] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.88 ms
[14:22:41] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[14:22:48] <mobrovac>	 hashar: pkoi je vois pas les changements que j'ai cherry-pické sur deployment-bastion?
[14:23:00] <mobrovac>	 puppet agent -tv aide pas sur deployment-restbase0x
[14:23:01] <mobrovac>	 :(
[14:24:14] <hashar>	 mobrovac: cherry pické quoi ou ? 
[14:24:29] <hashar>	 pour puppet ca doit être sur integration-puppetmaster  dans /var/lib/git/operations/puppet
[14:24:41] <mobrovac>	 ah ok
[14:24:44] <mobrovac>	 sigh
[14:25:01] <mobrovac>	 hashar: https://wikitech.wikimedia.org/wiki/Nova_Resource:Deployment-prep/How_code_is_updated dit pas ça :P
[14:25:05] <hashar>	 si c'est pour mediawiki /  /srv/mediawiki-staging est maintenu par un job jenkins qui nettoie tout toutes les 10 minutes
[14:25:18] <mobrovac>	 non non, c'est RESTBase
[14:25:23] <hashar>	 aahhh
[14:25:36] <hashar>	 RESTBase no clue
[14:25:57] <hashar>	 y avait eu l'expérimentation y a deux semaines pour utiliser  scap deploy pour restbase
[14:26:10] <mobrovac>	 j'essaie integration-puppetmaster
[14:26:12] <hashar>	 mais je t'avoue qu'ajaourd'hui je ne sais pas comment restbase est poussé/mise à jour sur beta :(
[14:26:35] <mobrovac>	 yup, mais ça c'est pour valider https://gerrit.wikimedia.org/r/#/c/253895/ avant d'aller en prod
[14:27:13] <mobrovac>	 euh? sudo su me domande le mot de passe
[14:27:36] <mobrovac>	 hashar: ^ ?
[14:27:47] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-restbase01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[14:27:51] <hashar>	 ah puppet
[14:27:59] <hashar>	 donc faut le cherry picker sur integration-puppetmaster en effet
[14:28:11] <wmf-insecte>	 Project browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce build #694: 04FAILURE in 2 min 10 sec: https://integration.wikimedia.org/ci/job/browsertests-Echo-en.wikipedia.beta.wmflabs.org-linux-firefox-sauce/694/
[14:28:24] <mobrovac>	 j'y suis, mais je peux pas devenir root là-bas
[14:28:30] <mobrovac>	 pkoi?
[14:28:35] <hashar>	 hashar@integration-puppetmaster:~$ sudo su -
[14:28:36] <hashar>	 root@integration-puppetmaster:~# 
[14:28:46] <mobrovac>	 nope
[14:28:50] <mobrovac>	 tjr mot de passe
[14:29:08] <mobrovac>	 ah!
[14:29:12] <mobrovac>	 ok ok
[14:29:21] <hashar>	 tu dois pas avoir les droits ...
[14:29:22] <hashar>	 grr
[14:29:44] <hashar>	 ouep 
[14:29:47] <mobrovac>	 attends
[14:29:53] <mobrovac>	 c'est koi ça?
[14:29:56] <hashar>	 !log granting root access to mobrovac on beta
[14:30:07] <mobrovac>	 integration-puppetmaster.integration.eqiad.wmflabs == integration-puppetmaster.eqiad.wmflabs
[14:30:17] <hashar>	 oh
[14:30:18] <hashar>	 man
[14:30:21] <mobrovac>	 mais integration-puppetmaster.deployment-prep.eqiad.wmflabs existe pas
[14:30:29] <hashar>	 lets switch to english because my french is crap for work related business
[14:30:31] <hashar>	 so yeah
[14:30:34] <mobrovac>	 haha
[14:30:42] <hashar>	 we have a couple puppet master, one for beta and one for integration
[14:30:44] <hashar>	 and I keep mixing up each of them 
[14:30:56] <hashar>	 so for beta it is deployment-puppetmaster.deployment-prep.eqiad.wmflabs 
[14:31:12] <hashar>	 User mobrovac may run the following commands on deployment-puppetmaster:
[14:31:14] <hashar>	     (ALL) NOPASSWD: ALL
[14:31:14] <hashar>	 mobrovac: sorry :-/
[14:31:18] <mobrovac>	 right, so i cherry-picked on the good one in the first place
[14:31:31] <mobrovac>	 but still can't see the changes on deployment-restbase0x
[14:31:35] <mobrovac>	 hm hm hm
[14:31:42] * mobrovac goes to dig
[14:31:42] <hashar>	 yeah
[14:32:03] <hashar>	 sometime I have to kick puppetmaster process
[14:32:12] <hashar>	 for it to catch up with cherry picks
[14:33:29] <mobrovac>	 ah no, i got it now, the config in beta is managed by scap
[14:33:31] <mobrovac>	 it's a symlink
[14:33:36] <mobrovac>	 so it doesn't change
[14:33:37] <mobrovac>	 uf
[14:33:38] <mobrovac>	 kk
[14:33:42] <mobrovac>	 thnx hashar!
[14:33:42] <hashar>	 mess mess
[14:33:45] <mobrovac>	 yeah
[14:36:24] <wmf-insecte>	 Project browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce build #327: 04FAILURE in 8 min 24 sec: https://integration.wikimedia.org/ci/job/browsertests-MobileFrontend-SmokeTests-linux-chrome-sauce/327/
[14:56:49] <wikibugs>	 5Continuous-Integration-Scaling, 6operations, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1813655 (10hashar) The two last puppet patches has let the zuul-merger on scandium to reach out gallium AND let the slaves git clone from scandium git...
[14:57:19] <grrrit-wm>	 (03CR) 10Hashar: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/173830 (owner: 10Hashar)
[14:57:37] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] tests: factor out code to find defined project [integration/config] - 10https://gerrit.wikimedia.org/r/173830 (owner: 10Hashar)
[15:04:43] <hashar>	 andrewbogott: zuul-merger pooled on scandium thanks for the follow up patche
[15:04:48] <hashar>	 I am monitoring it 
[15:04:49] <hashar>	 :-D
[15:05:00] <andrewbogott>	 working so far?
[15:11:29] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[15:12:21] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.67 ms
[15:15:36] <wikibugs>	 5Continuous-Integration-Scaling, 6operations, 5Patch-For-Review: install/deploy scandium as zuul merger (ci) server - https://phabricator.wikimedia.org/T95046#1813684 (10hashar) More unpuppetized / badly pauperized stuff:  ``` stderr: 'Cloning into '/srv/ssd/zuul/git/mediawiki/core'... Warning: Identity file...
[15:16:54] <hashar>	 !log stopping zuul-merger on scandium. Lacks ssh private key to reach gerrit
[15:18:04] <icinga-wm>	 PROBLEM - zuul_merger_service_running on scandium is CRITICAL: PROCS CRITICAL: 0 processes with regex args ^/usr/share/python/zuul/bin/python /usr/bin/zuul-merger
[15:19:30] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.83 ms
[15:27:06] <dcausse>	 I'd like to set a config for wikipedia sites only, I see that I can add a 'wikipedia' key in mediawiki-config InitialiseSettings.php (like wgFavicon).
[15:28:02] <dcausse>	 And I have a test that uses SiteConfiguration but these vars are not resolved properly, I was wondering where this resolution is made
[15:55:31] <shinken-wm>	 RECOVERY - Puppet failure on deployment-eventlogging03 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:00:11] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1813816 (10hashar) https://gerrit.wikimedia.org/r/253040 got merged at ` Wed Nov 18 09:22:43 2015 UTC`.  T...
[16:03:14] <hashar>	 dcausse: it is based on the dblist files
[16:03:23] <hashar>	 dcausse: can't remember offhand how the resolution is done though :(
[16:03:56] <hashar>	 probably somewhere in multiversion/MWWikiversions.ph
[16:04:18] <dcausse>	 hashar: thanks, I'll continue to search, resolution by suffix seams to work in my case but not with sitename :/
[16:04:23] <hashar>	 wmf-config/CommonSettings.php:          $dblist = MWWikiversions::readDbListFile( $tag );
[16:04:23] <hashar>	 wmf-config/CommonSettings.php:          if ( in_array( $wgDBname, $dblist ) ) {
[16:04:50] <hashar>	 bah barely make sense
[16:04:58] <hashar>	 but CommonSettings.php has some code
[16:05:02] <hashar>	 reading the dblist files
[16:05:18] <dcausse>	 what's the meaning of tag?
[16:05:31] <hashar>	 and adding matching .dblist files to a $wikiTags variable which is passed as a parameter to the wgConf object
[16:06:48] <hashar>	 that is mediawiki/core includes/SiteConfiguration.php
[16:10:30] <dcausse>	 hashar: thanks!
[16:26:31] <bd808>	 thcipriani: I can't land D48. I'm getting "Exception: You do not have permission to push to this repository." using ssh://vcs@git-ssh.wikimedia.org/diffusion/MSCA/scap.git as my origin
[16:27:21] <thcipriani>	 bd808: blerg. I'm guessing for whatever reason it's limited to RelEng team (but I'm not sure about that) twentyafterfour ostriches ^
[16:30:29] <bd808>	 thcipriani: *nod* that makes some sense. Feel free to land it yourself if you guys are ready. It looks like the associated puppet change merged
[16:31:29] <thcipriani>	 bd808: kk, I can land it, would be nice to know how to change these permissions though. Blindly clicking around interfaces isn't yielding anything that looks useful so far.
[16:32:55] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: PING CRITICAL - Packet loss = 100%
[16:37:31] <twentyafterfour>	 bd808: thanks I'll give you push permissions ;)
[16:38:01] <shinken-wm>	 PROBLEM - Puppet failure on deployment-tmh01 is CRITICAL: CRITICAL: 55.56% of data above the critical threshold [0.0]
[16:38:07] * bd808 runs wild with scap changes
[16:38:49] <Krenair>	 Do the permissions not match gerrit?
[16:41:00] <twentyafterfour>	 Krenair: we haven't migrated permissions in any way, the scap repo is a one-off which we are using to test out differential more thoroughly
[16:42:56] <thcipriani>	 bd808: haven't started any sort of landing for D48, FYI, so now that you have permissions, feel free. I Need to upgrade my arcanist and wanted to wait until I was done deploying.
[16:43:19] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[16:45:13] <bd808>	 twentyafterfour: I'm still getting the same error. What should my git remote look like? I have "ssh://vcs@git-ssh.wikimedia.org/diffusion/MSCA/scap.git"
[16:45:39] <twentyafterfour>	 bd808: I just added you maybe you tried before I hit save?
[16:45:45] <twentyafterfour>	 your remote looks correct
[16:46:07] <ostriches>	 Krenair: No, permissions do not match gerrit. Gerrit permissions are not exactly a model to follow :)
[16:46:08] <bd808>	 victory!
[16:46:14] <ostriches>	 If anything we should be *more* permissive.
[16:46:53] <twentyafterfour>	 ostriches: with scap we probably shouldn't be very permissive, given how critical it is
[16:46:57] <Krenair>	 ostriches, well, yes
[16:47:05] <bd808>	 thcipriani, twentyafterfour: it's merged now.
[16:47:17] <ostriches>	 twentyafterfour: Well, since we're planning to package it I'm not too worried.
[16:47:18] <twentyafterfour>	 bd808: awesome thanks!
[16:47:25] <Krenair>	 ostriches, point is, the set of people able to do things in the gerrit repo should probably have been granted the equivalent
[16:47:55] <ostriches>	 No, it started as "just repo admins" and then "just releng" and then I'll probably widen it to "anyone"
[16:47:56] * twentyafterfour doesn't even know who had commit access in gerrit
[16:48:22] <Krenair>	 allow anyone to push to the repository?
[16:48:23] <twentyafterfour>	 ostriches: there is a per-repository setting for "can push" and there are also project based policies
[16:48:31] <ostriches>	 I know.
[16:48:42] <ostriches>	 I think anyone with an account should be able to push to it, sure.
[16:49:03] <grrrit-wm>	 (03CR) 10Paladox: "Can now be merged." [integration/config] - 10https://gerrit.wikimedia.org/r/253376 (owner: 10Paladox)
[16:49:05] <Krenair>	 to the master branch?
[16:49:08] <twentyafterfour>	 I'm not entirely opposed to that but anyone can get an account...
[16:49:19] <twentyafterfour>	 so that's pretty much wide-open to anything goes
[16:49:21] <paladox>	 hashar: Please merge https://gerrit.wikimedia.org/r/#/c/253376/ and https://gerrit.wikimedia.org/r/#/c/253165/ they can be merged now.
[16:49:38] * twentyafterfour thinks we might want a bit more restriction than that.
[16:49:47] <hashar>	 unreviewed, in my review queue. Maybe others here can process them though :)
[16:49:48] <shinken-wm>	 PROBLEM - Puppet failure on mira is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[16:49:59] <ostriches>	 Maybe we could have a group like we did with Code Review.
[16:50:08] <ostriches>	 Let anyone who's a member add any other member.
[16:50:19] <ostriches>	 And those people can generally push to *any* repo unless it's otherwise ACL'd
[16:50:22] <ostriches>	 (ie: puppet)
[16:50:46] <twentyafterfour>	 so circle of trust, essentially?
[16:50:49] <twentyafterfour>	 that makes sense
[16:50:50] <ostriches>	 Yep
[16:51:40] <twentyafterfour>	 if we use packages and audits then each team can set up audits on the code that they care about (so that we catch any dangerous rogue commits before they get deployed anywhere)
[16:51:57] <shinken-wm>	 PROBLEM - Puppet failure on deployment-mediawiki03 is CRITICAL: CRITICAL: 30.00% of data above the critical threshold [0.0]
[16:52:09] <twentyafterfour>	 audit really is a nice tool and it eliminates the need for strict push controls, generally
[16:55:46] <ostriches>	 And really, the number of repos that actually need strict push controls is limited (puppet, dns)
[16:56:21] <ostriches>	 Most other things go through (or should go through) a packaging process which would be the extra set of control you need.
[17:03:09] <Krenair>	 ostriches, deployment branches?
[17:03:22] <ostriches>	 Deployment branches should die in a fire.
[17:03:25] <ostriches>	 Terrible pattern
[17:03:31] <ostriches>	 At least the way we do them for MW
[17:03:35] <ostriches>	 + exts
[17:03:38] <Krenair>	 what do you suggest we replace them with?
[17:03:50] <ostriches>	 A shared repo for deployment that's a "fork" of MW.
[17:03:57] <ostriches>	 *That* repo would have all the permissions
[17:04:07] <ostriches>	 And extensions/core can just be left alone and permissive.
[17:08:42] <greg-g>	 there's a task for that :)
[17:08:54] <greg-g>	 we were just talking about it during antoine and I's 1:1
[17:13:29] <ostriches>	 Krenair: Anyway, if we did per-branch push permissions we'd set them up in Herald anyway. The "Can push" permission would remain permissive.
[17:18:37] <thcipriani>	 !log updating scap in beta cluster to deploy https://phabricator.wikimedia.org/D48
[17:25:13] <thcipriani>	 so far, off to a bad start: 2/10 minions completed fetch :(
[17:34:57] <thcipriani>	 !log scap update in beta failing, mediawikis failing git fetch, unable to find needed commit, removing directory and refetching
[17:40:02] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Release-Engineering-Team: Rebuild deployment master - https://phabricator.wikimedia.org/T117504#1814180 (10demon) a:5demon>3None
[17:41:12] <marxarelli>	 thcipriani: are you scap3-ing?
[17:41:40] <thcipriani>	 marxarelli: trebucheting scap3 on deployment-prep
[17:41:51] <marxarelli>	 ahhh
[17:41:54] <marxarelli>	 right, minions
[17:42:00] <thcipriani>	 corrupt git checkout on most of the minions.
[17:43:40] <thcipriani>	 it's weird, with mira in deployment prep, I just had to login and run the salt-call commands manually and everything worked fine: sudo salt-call deploy.fetch 'scap/scap' then deploy.checkout 'scap/scap'
[17:46:58] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212#1814203 (10Nikerabbit) The proposed end goal might cause more SWAT deploys will be used instead, working against the original goal.  I am also cautious whether staying lo...
[17:49:18] <shinken-wm>	 RECOVERY - Host integration-labsvagrant is UP: PING OK - Packet loss = 0%, RTA = 0.58 ms
[17:50:07] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team: Take heat off day before the weekly branch-cut? - https://phabricator.wikimedia.org/T118212#1814216 (10greg) We can easily cut the next branch before we deploy it (say, on Friday for the upcoming Tuesday train start) but we'll need yet-another-Beta-Cluster to te...
[18:16:20] <thcipriani>	 bd808: ostriches so now I'm still seeing permissions issues with sync-masters, but now it has to do tasks.merge_cdb_updates https://integration.wikimedia.org/ci/job/beta-scap-eqiad/79087/console
[18:16:22] <wikibugs>	 10Beta-Cluster-Infrastructure, 6operations, 7HHVM, 5Patch-For-Review: Convert work machines (tin, terbium) to Trusty and hhvm usage - https://phabricator.wikimedia.org/T87036#1814315 (10dduvall)
[18:19:40] <bd808>	 thcipriani: hmmm... need help digging into that?
[18:20:20] <ostriches>	 Bleh.
[18:21:08] <thcipriani>	 bd808: I can start looking into it, I just tracked down what was happening, poked you guys in case it was something obvious.
[18:21:35] <bd808>	 that directory is 0755 l10nupdate:wikidev
[18:25:20] <bd808>	 do we usually run the cdb update task as the l10nupdate user? /me looks
[18:25:45] <thcipriani>	 that's what I was just digging for. I know there's a sudoer rule to allow mwdeploy to run some things as l10nupdate
[18:26:50] <bd808>	 thcipriani: I think that's the problem. look for other calls to scap-rebuild-cdbs
[18:27:10] <thcipriani>	 okie doke
[18:28:05] * ostriches mutters something about doing everything as root
[18:28:15] <bd808>	 well.. no. It looks we call that script as mwdeploy
[18:28:41] <bd808>	 main.RebuildCdbs actually asserts that it it run as mwdeploy
[18:29:15] <Krinkle>	 greg-g: 'staging' sounds interesting. Is there a task or page about that?
[18:31:07] <greg-g>	 Krinkle: not really, but summary: yet-another-test-cluster that has two goals: 1) be yet another cluster and 2) clean up the hacks that make up beta cluster
[18:31:26] <bd808>	 thcipriani: this is probably related to the differences between staging and the deploy dir
[18:31:28] <greg-g>	 Krinkle: staging isn't really on our near-team roadmap right now (it's not a simple thing) :)
[18:31:51] <bd808>	 in staging we want l10nupdate to be able to mess with the cdb files to do the nightly update with mater message changes
[18:32:07] <greg-g>	 Krinkle: there's https://phabricator.wikimedia.org/tag/staging/ but that's too "down in the weeds"
[18:32:22] <bd808>	 but in the deploy dir (which is what scap normally operates on) everything is mwdeploy:mwdeploy
[18:32:45] * greg-g is in the SoS and then the monthly eng managers meeting, slow to respond now
[18:33:09] <bd808>	 *master message changes
[18:33:09] <thcipriani>	 bd808: that makes sense, mostly :)
[18:34:16] * bd808 tries to find the difference between syncing from master->master and just updating the "normal" master
[18:36:19] <bd808>	 thcipriani: on the deploy master we run tasks.update_localization_cache
[18:36:37] <bd808>	 that has the sudo as l10nupdate bits in it
[18:36:55] <bd808>	 That process creates the l10n json files
[18:37:29] <thcipriani>	 seems like we only do the merge_cdb_updates task on targets in scap (which is what master → master is doing)
[18:37:29] <bd808>	 what we are trying to do now in the mira process is the reverse (recreate the cdbs from the json)
[18:38:29] <bd808>	 I don't think I thought this through fully when I set it up to happen
[18:39:19] <bd808>	 we need to recreate the cdbs on mira as though l10nupdate had created them in the first place.
[18:39:40] <bd808>	 that's going to take code changes in scap (new wrapper script)
[18:40:03] <bd808>	 because we need to sudo as l10nupdate when recreating the cdbs
[18:40:26] <ottomata>	 hi again!  bd808, asking you cause I SEE YOU.  
[18:40:37] <ottomata>	 the new eventlogging repo I created is not being replicated to github
[18:40:40] <ottomata>	 i see
[18:40:44] <thcipriani>	 sudo -u l10nupdate is something mwdeploy can do in sudoers files.
[18:40:47] <ottomata>	 Missing repository created; retry replication to git@github.com:wikimedia/eventlogging
[18:40:54] <ottomata>	 (along with the same message for several other repos)
[18:40:55] <ottomata>	 in
[18:41:03] <ottomata>	  /var/lib/gerrit2/review_site/logs/error_log
[18:41:20] <greg-g>	 twentyafterfour: I see what thcipriani sees re "no newline" by the way: https://phabricator.wikimedia.org/F2973024
[18:41:26] <bd808>	 thcipriani: yeah, we just need a scap entry point that can run the right command as the l10nupdate user I think
[18:41:49] <bd808>	 ottomata: above my pay grade, but ostriches might be able to help
[18:42:05] <ottomata>	 :
[18:42:06] <ottomata>	 :)
[18:42:55] <bd808>	 thcipriani: am I making sense when I babble about this?
[18:44:00] <thcipriani>	 bd808: heh, yeah, this is the part of scap that I generally try to not touch so I'm a little behind.
[18:44:10] <bd808>	 I think what we are missing is an entry point script that we can use to run tasks.update_l10n_cdb as the l10nupdate user
[18:44:33] <bd808>	 and then use that instead of directly calling tasks.update_l10n_cdb when we sync to the master
[18:44:46] <thcipriani>	 right. I can put this on my todo list. Would probably help me get more familiar with this section of code.
[18:45:35] <bd808>	 thcipriani: cool. yell at me if you give up, but it shouldn't be too tricky
[18:45:49] <thcipriani>	 bd808: thanks, will do :)
[18:54:11] <ostriches>	 ottomata: New repo on gerrit?
[18:54:48] <ottomata>	 ostriches: ja, couple days ago I made https://gerrit.wikimedia.org/r/#/admin/projects/eventlogging
[18:54:52] <ottomata>	 everything is cool w gerrit
[18:54:54] <ostriches>	 Hmm, "Missing repository created" should mean it exists now
[18:54:58] <ottomata>	 but it isn't replicating to github
[18:54:59] <ottomata>	 oh
[18:55:01] <ostriches>	 It'll replicate next time something changes.
[18:55:04] <ottomata>	 https://github.com/wikimedia/eventlogging
[18:55:10] <ottomata>	 hm, lots has been changing
[18:55:18] <ottomata>	 there was some trickiness with it though too
[18:55:28] <ostriches>	 I hate github.
[18:55:32] <ottomata>	 i created the repo, and accidentally committed the wrong first commit
[18:55:36] <ottomata>	 so i deleted it from gerrit and recreated it
[18:56:09] <ottomata>	 i first pushed to it, not through gerrit
[18:56:11] <ottomata>	 so maybe that's the problem?
[18:56:14] <ottomata>	 hmm, i bet that is it
[18:56:17] <ottomata>	 for most of them
[18:56:22] <ottomata>	 there are other repos there that have that problem
[18:56:32] <ottomata>	 since i was filter-branching to create this new repo from a dir of another
[18:56:43] <ottomata>	 i had to push (ddin't want to review hundreds of commits in gerrit)
[18:57:24] <ostriches>	 Maybe? I dunno, gerrit replication breaks over the tiniest thing.
[18:57:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Graphoid: Graphoid does not deploy sync to deployment-sca01 - https://phabricator.wikimedia.org/T118929#1814427 (10akosiaris) 5Open>3Resolved a:3akosiaris For some reason there were empty objects under .git directory. Deleting them fixed that and trebuchet works again.
[19:05:00] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Deployment-Systems, 5Patch-For-Review, 7WorkType-Maintenance: beta-scap-eqiad mira / deployment-bastion permissions problem - https://phabricator.wikimedia.org/T117016#1814439 (10bd808) >>! In T117016#1813816, @hashar wrote: > https://gerrit.wikimedia.org/r/253040 got merge...
[19:29:28] <shinken-wm>	 PROBLEM - Host integration-labsvagrant is DOWN: CRITICAL - Host Unreachable (10.68.16.4)
[20:06:41] <twentyafterfour>	 uhm ... wmf.7 contains reference to undefined constant CURLOPT_SAFE_UPLOAD
[20:07:21] <twentyafterfour>	 https://phabricator.wikimedia.org/T118988
[20:13:45] <paladox>	 jzerebecki: Could you review https://gerrit.wikimedia.org/r/#/c/253376/ and https://gerrit.wikimedia.org/r/#/c/253165/ the first link is for moving jshint to check: since i moved jshint to npm. Source patch was merged. Second link is for migrating WikiEditor to extension-gate. And since codeeditor depends on wikieditor but is not added as dependacy in jenkins yet so this makes it easier.
[20:41:42] <hashar>	 andrewbogott: looks like I created some tech debt a year or so ago :-/   
[20:41:57] <hashar>	 the zuul-merger ssh key is only on gallium and is not in puppet.
[20:42:01] <hashar>	 (was https://gerrit.wikimedia.org/r/#/c/253925/ )
[20:49:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Graphoid: Graphoid does not deploy sync to deployment-sca01 - https://phabricator.wikimedia.org/T118929#1814906 (10hashar) Great finding @akosiaris!  It might be due to one of the labvirt node that had a full disk a few days ago. In such a case, files can still be created (ther...
[21:21:05] <wmf-insecte>	 Project browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce build #74: 04FAILURE in 5 min 4 sec: https://integration.wikimedia.org/ci/job/browsertests-QuickSurveys-en.m.wikipedia.beta.wmflabs.org-linux-chrome-sauce/74/
[21:24:31] <grrrit-wm>	 (03PS1) 10Ottomata: Removing python code from EventLogging MW extension, don't need to tox anymore [integration/config] - 10https://gerrit.wikimedia.org/r/254032 (https://phabricator.wikimedia.org/T118863) 
[21:24:42] <grrrit-wm>	 (03PS2) 10Ottomata: Removing python code from EventLogging MW extension, don't need to tox anymore [integration/config] - 10https://gerrit.wikimedia.org/r/254032 (https://phabricator.wikimedia.org/T118863) 
[21:24:56] <ottomata>	 hashar: , whenever you get a chance, danke
[21:25:39] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Removing python code from EventLogging MW extension, don't need to tox anymore [integration/config] - 10https://gerrit.wikimedia.org/r/254032 (https://phabricator.wikimedia.org/T118863) (owner: 10Ottomata)
[21:25:48] <hashar>	 ottomata: will deploy once merged :-)
[21:28:31] <grrrit-wm>	 (03Merged) 10jenkins-bot: Removing python code from EventLogging MW extension, don't need to tox anymore [integration/config] - 10https://gerrit.wikimedia.org/r/254032 (https://phabricator.wikimedia.org/T118863) (owner: 10Ottomata)
[21:29:18] <hashar>	 ottomata: done!
[21:29:25] <hashar>	 ottomata: one day we will get it migrated to scap :-}
[21:29:37] <hashar>	 I am off, have fun everyone
[21:30:05] <ottomata>	 danke laters!
[21:43:06] <wikibugs>	 10Deployment-Systems, 6Release-Engineering-Team, 5Patch-For-Review: Move the train deployment from Thursday to Wednesday for some Wikipedia sites - https://phabricator.wikimedia.org/T115002#1815170 (10greg) Alright, given the next two weeks we won't be having the train anyways (see: https://wikitech.wikimedi...
[23:13:44] <twentyafterfour>	 this is super cool: https://corner.squareup.com/2015/11/fastclone.html
[23:22:11] <thcipriani>	 that is pretty neat.
[23:23:51] <thcipriani>	 oh, ruby. https://github.com/square/git-fastclone/blob/master/lib/git-fastclone.rb#L18
[23:24:11] <marxarelli>	 haha
[23:24:49] <marxarelli>	 iirc, the gem has a very unprofessional post-install message
[23:25:55] <marxarelli>	 oh maybe it was just the description
[23:25:58] <marxarelli>	 "A small library for doing (command) lines"
[23:29:58] <marxarelli>	 thcipriani: are you thinking we could port it to python?
[23:30:06] <marxarelli>	 fastclone that is
[23:30:26] <wikibugs>	 10Deployment-Systems, 3Scap3: Need a way to see config diffs in Scap - https://phabricator.wikimedia.org/T118206#1815478 (10mmodell) p:5Triage>3High
[23:30:41] <thcipriani>	 marxarelli: heh, I wasn't thinking that, but that is a pattern we follow :)
[23:31:01] <thcipriani>	 no, twentyafterfour posted it, I just started digging through.
[23:31:58] <twentyafterfour>	 I was thinking it might be nice for CI stuff
[23:32:09] <twentyafterfour>	 combine that with drydock and we'd have a jenkins killer
[23:33:45] <marxarelli>	 i was thinking recently that we should combine drydock and kubernetes :)
[23:34:19] <marxarelli>	 the latter is already coming to Tool Labs
[23:35:04] <twentyafterfour>	 kubernetes for allocating VM instances?
[23:35:11] <marxarelli>	 containers
[23:35:15] <twentyafterfour>	 ah
[23:35:17] <twentyafterfour>	 right on
[23:37:06] <marxarelli>	 so what is the scope of drydock exactly?
[23:37:11] <marxarelli>	 i'm a little confused about that
[23:38:24] <twentyafterfour>	 drydock is all about allocating and leasing resources
[23:38:50] <twentyafterfour>	 so instances, working trees, or any arbitrary resource (it's very abstract)
[23:45:46] <twentyafterfour>	 currently the resources are pre-configured statically and it handles leasing the jobs out to harbormaster builds
[23:46:49] * marxarelli reads all the docs
[23:48:02] <twentyafterfour>	 they just wrote a bunch of new documentation, which I read last night. it's still not 100% clear but it's pretty good.
[23:49:51] <marxarelli>	 it'd be great to sit down and suss out a long-term vision for CI and Phab in January