[00:26:44] <shinken-wm>	 PROBLEM - Puppet run on deployment-depurate01 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[00:56:43] <shinken-wm>	 RECOVERY - Puppet run on deployment-depurate01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:24:20] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-stream is CRITICAL: CRITICAL: deployment-prep.deployment-stream.diskspace._var.byte_percentfree (<10.00%)
[02:33:43] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul, 07Tracking, 07WorkType-Maintenance: Scale merge operations (tracking) - https://phabricator.wikimedia.org/T70480#2500874 (10Phabricator_maintenance)
[02:33:55] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: Gallium must be backed up (tracking) - https://phabricator.wikimedia.org/T65934#2500884 (10Phabricator_maintenance)
[02:33:59] <wikibugs>	 10Browser-Tests-Infrastructure, 10MediaWiki-extensions-WikibaseRepository, 10Wikidata, 10Wikidata-Gadgets, 07Tracking: Create tests for default gadgets (tracking) - https://phabricator.wikimedia.org/T67257#2500881 (10Phabricator_maintenance)
[02:34:30] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Technical-Debt, 07Tracking: All repositories should pass jshint test (tracking) - https://phabricator.wikimedia.org/T62619#2500892 (10Phabricator_maintenance)
[02:34:40] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Tracking: create API level tests to monitor services on beta/test2wiki and also on production (tracking) - https://phabricator.wikimedia.org/T60353#2500899 (10Phabricator_maintenance)
[02:35:15] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Tracking: automatically import some content from production into Beta-Cluster (tracking) - https://phabricator.wikimedia.org/T54382#2500917 (10Phabricator_maintenance)
[02:35:17] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: continuous integration monitoring (tracking) - https://phabricator.wikimedia.org/T51459#2500922 (10Phabricator_maintenance)
[02:36:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Tracking: live hack in beta mediawiki-config (tracking) - https://phabricator.wikimedia.org/T43132#2500954 (10Phabricator_maintenance)
[02:36:26] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul, 07Tracking: deploy Zuul (tracking) - https://phabricator.wikimedia.org/T42681#2500957 (10Phabricator_maintenance)
[02:37:24] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Tracking: use scap to deploy on apaches (tracking) - https://phabricator.wikimedia.org/T39078#2500986 (10Phabricator_maintenance)
[02:37:28] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: Implementing continuous integration workflow (tracking) - https://phabricator.wikimedia.org/T37584#2500995 (10Phabricator_maintenance)
[02:38:09] <wikibugs>	 10Beta-Cluster-Infrastructure, 07Tracking: several bugs on deployment test site for 1.19 - https://phabricator.wikimedia.org/T35578#2501008 (10Phabricator_maintenance)
[03:01:57] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[03:17:57] <shinken-wm>	 PROBLEM - SSH on deployment-logstash2 is CRITICAL: Server answer
[03:57:59] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[04:03:57] <shinken-wm>	 PROBLEM - SSH on deployment-logstash2 is CRITICAL: Server answer
[04:18:22] <wmf-insecte>	 Project selenium-MultimediaViewer » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #87: 04FAILURE in 22 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/87/
[05:18:57] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[05:24:55] <shinken-wm>	 PROBLEM - SSH on deployment-logstash2 is CRITICAL: Server answer
[05:50:26] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[05:50:56] <shinken-wm>	 PROBLEM - SSH on deployment-logstash2 is CRITICAL: Server answer
[08:07:39] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Packaging, 10Traffic: piuparts fail with WARN: Broken symlinks: /etc/systemd/system... - https://phabricator.wikimedia.org/T141454#2501246 (10hashar) syslog-ng fixed the warning by adding `rm` and `rmdir` in the package `postrm` step: http://anonscm....
[09:35:52] <hashar>	 !log Regenerated Nodepool image for Trusty.  The snapshot failed while upgrading grub-pc for some reason. Noticed with thcipriani yesterday
[09:35:56] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[09:46:43] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2501313 (10hashar) After discussions with @greg-g and others lets start it on the week of August 22nd. I will be back from vacations and all fresh to baby sit / train up people.   Id...
[09:46:56] <hashar>	 !log Nodepool: Image ci-trusty-wikimedia-1469698821 in wmflabs-eqiad is ready
[09:47:00] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[09:52:48] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: debian-glue: Add a non-voting job and migrate all jobs to it [integration/config] - 10https://gerrit.wikimedia.org/r/301566 
[09:52:50] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: debian-glue: Vote for all contenttranslation repos [integration/config] - 10https://gerrit.wikimedia.org/r/301567 
[10:20:56] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[10:26:56] <shinken-wm>	 PROBLEM - SSH on deployment-logstash2 is CRITICAL: Server answer
[10:36:59] <shinken-wm>	 RECOVERY - SSH on deployment-logstash2 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[10:39:05] <grrrit-wm>	 (03PS2) 10Hashar: debian-glue: Add a non-voting job and migrate all jobs to it [integration/config] - 10https://gerrit.wikimedia.org/r/301566 (owner: 10Alexandros Kosiaris)
[10:44:03] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Thank you!!!" [integration/config] - 10https://gerrit.wikimedia.org/r/301566 (owner: 10Alexandros Kosiaris)
[10:45:14] <grrrit-wm>	 (03Merged) 10jenkins-bot: debian-glue: Add a non-voting job and migrate all jobs to it [integration/config] - 10https://gerrit.wikimedia.org/r/301566 (owner: 10Alexandros Kosiaris)
[10:48:39] <grrrit-wm>	 (03PS2) 10Hashar: debian-glue: Vote for all contenttranslation repos [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[10:53:07] <grrrit-wm>	 (03CR) 10Hashar: "There is a potential chicken and egg problem there. On a new apertium major release , all the repo but apertium will probably fail buildi" [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[11:01:56] <paladox>	 hashar i have been testing the Zuul version: 2.1.0-391-gbc58ea3-wmf1jessie1 for jessie and is working.
[11:01:59] <paladox>	 sorry for ping
[11:02:00] <paladox>	 and hi
[11:02:07] <paladox>	 i saw -operations
[11:02:50] <paladox>	 we could use 2.1.0-391-gbc58ea3-wmf1jessie1 for the zuul merger
[11:12:41] <grrrit-wm>	 (03PS1) 10Hashar: Add operations/debs/python-thumbor-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/301577 
[11:13:43] <grrrit-wm>	 (03CR) 10Hashar: "Will cause jenkins to attempt to build the .deb package eg on https://gerrit.wikimedia.org/r/#/c/301573/ :)" [integration/config] - 10https://gerrit.wikimedia.org/r/301577 (owner: 10Hashar)
[11:13:53] <hashar>	 paladox: I will rebuild it 
[11:13:59] <hashar>	 ;-)
[11:13:59] <paladox>	 Ok thanks
[11:14:02] <paladox>	 :)
[11:14:07] <hashar>	 have to double checks the python dependencies
[11:14:11] <hashar>	 and the set of pathes
[11:14:13] <hashar>	 patches
[11:14:14] <paladox>	 Yeh
[11:14:19] <hashar>	 after snack though :}
[11:14:29] <paladox>	 I may have build it wrong but anyways thanks and ok :)
[11:14:42] <hashar>	 bb
[11:14:47] <paladox>	 :)
[11:25:37] <grrrit-wm>	 (03CR) 10Gilles: [C: 031] Add operations/debs/python-thumbor-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/301577 (owner: 10Hashar)
[11:27:24] <paladox>	 Hi it seems this https://integration.wikimedia.org/ci/job/debian-glue-non-voting/4/console has been running along time
[11:27:26] <paladox>	 30mins+
[11:27:32] <paladox>	 legoktm jzerebecki ^^
[11:27:41] <paladox>	 Never mind
[11:27:50] <paladox>	 it failed it self
[11:35:04] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Add operations/debs/python-thumbor-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/301577 (owner: 10Hashar)
[11:36:17] <grrrit-wm>	 (03Merged) 10jenkins-bot: Add operations/debs/python-thumbor-wikimedia [integration/config] - 10https://gerrit.wikimedia.org/r/301577 (owner: 10Hashar)
[11:39:23] <paladox>	 hashar that was quick?
[11:39:29] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 031] "yeah, well aware. It is an unsolveable in an automated way chicken and egg problem though currently, so I think I 'll be ok with solving i" [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[11:42:36] <wikibugs>	 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Sprint-77-Segmentation-fault, and 4 others: Spike [2hrs] Wikidata description browser tests do not run anywhere - https://phabricator.wikimedia.org/T137756#2501448 (10Jhernandez) I've merged. @Jdlrobson can you help sign off and verify...
[11:53:23] <grrrit-wm>	 (03PS3) 10Hashar: debian-glue: Vote for all contenttranslation repos [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[11:53:50] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Per discussion with Alexandros :)" [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[11:54:27] <grrrit-wm>	 (03Merged) 10jenkins-bot: debian-glue: Vote for all contenttranslation repos [integration/config] - 10https://gerrit.wikimedia.org/r/301567 (owner: 10Alexandros Kosiaris)
[12:03:07] <grrrit-wm>	 (03PS1) 10Hashar: Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 
[12:13:02] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2501550 (10hashar)
[12:14:17] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2479965 (10hashar) Since I have filled this task, the zuul server on gallium (Precise) has been upgraded to further c...
[12:14:18] <grrrit-wm>	 (03CR) 10Paladox: "Need to add DEB_BUILD_OPTIONS=nocheck in rules" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 (owner: 10Hashar)
[12:14:52] <hashar>	 paladox: I pass no check when building the package locally
[12:15:02] <paladox>	 Oh
[12:15:06] <hashar>	 eventually I would like to have the .deb package to run the test suite
[12:15:12] <paladox>	 yeh
[12:15:15] <hashar>	 it is going to be broken until figured out :D
[12:15:19] <paladox>	 yep
[12:15:35] <paladox>	 But if we add  DEB_BUILD_OPTIONS=nocheck it works
[12:15:36] <paladox>	 :)
[12:16:06] <grrrit-wm>	 (03CR) 10Hashar: "Build on https://people.wikimedia.org/~hashar/debs/zuul_2.1.0-391-gbc58ea3-jessie/ seems to work." [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 (owner: 10Hashar)
[12:17:34] <paladox>	 I installed ^^ 
[12:17:36] <paladox>	 and works
[12:17:39] <paladox>	 :)
[12:18:41] <hashar>	 !log installed 2.1.0-391-gbc58ea3-wmf1jessie1 on zuul-dev-jessie.integration.eqiad.wmflabs
[12:18:45] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[12:18:49] <hashar>	 !log installed 2.1.0-391-gbc58ea3-wmf1jessie1 on zuul-dev-jessie.integration.eqiad.wmflabs T140894
[12:18:50] <stashbot>	 T140894: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894
[12:18:52] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[12:22:30] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 (owner: 10Hashar)
[12:53:15] <addshore>	 :>
[12:53:19] <addshore>	 cheers for the recheck hashar ! :D
[12:53:30] <addshore>	 I did think that shouldn't have needed a rebase!
[12:54:04] <addshore>	 oh, even though it still says it does...
[12:56:12] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 (owner: 10Hashar)
[12:56:17] <grrrit-wm>	 (03CR) 10Hashar: [V: 032] Merge branch 'debian/precise-wikimedia' into debian/jessie-wikimedia [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/301584 (owner: 10Hashar)
[13:00:34] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2501693 (10hashar) 05Open>03Resolved a:03elukey Luca has done the upgrade and it went flawlessly.  I have doubl...
[13:01:23] <hashar>	 addshore: I have really no idea what is going on :(
[13:01:29] <hashar>	 addshore: maybe the change really requires a rebase
[13:01:56] <addshore>	 apparently it did, I think I mixed something up with the new gerrit UI and checked out the wrong change or soemthing
[13:03:52] <paladox>	 hashar should we do trusty for zuul too?
[13:15:06] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501716 (10hashar)
[13:27:46] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501747 (10hashar)
[13:30:23] <wikibugs>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 10DBA, 10Datasets-General-or-Unknown, and 3 others: Automatize the check and fix of object, schema and data drifts between production masters and slaves - https://phabricator.wikimedia.org/T104459#2501749 (10jcrespo) @ArielGlenn, before you devel...
[13:32:48] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501752 (10hashar)
[13:57:39] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501794 (10hashar)
[13:58:30] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501716 (10hashar)
[14:18:15] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2501833 (10hashar) I have added a new Grafana panel showing the state of that specific check on [[ https://grafana.wikimedia...
[14:49:38] <Nikerabbit>	 is something stuck https://integration.wikimedia.org/zuul/ ?
[14:50:20] <Nikerabbit>	 https://integration.wikimedia.org/ci/job/mwext-testextension-php55/17514/console I mean
[14:51:42] <mutante>	 /w/win 17
[14:58:30] <shinken-wm>	 PROBLEM - Puppet run on deployment-sca02 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[14:59:16] <shinken-wm>	 PROBLEM - Puppet run on deployment-sca01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[15:12:20] <grrrit-wm>	 (03PS1) 10Alexandros Kosiaris: jenkins-debian-glue: SKIP_COWBUILDER_UPDATE set to true [integration/config] - 10https://gerrit.wikimedia.org/r/301615 
[15:12:25] <thcipriani>	 hashar: looking at https://gerrit.wikimedia.org/r/#/c/300738/ the "jenkins-deploy kvm membership" exec—why was that moved out of labs.pp? Is it not needed there anymore?
[15:14:05] <grrrit-wm>	 (03PS1) 10Hashar: debian-gue: stop updating the cow image [integration/config] - 10https://gerrit.wikimedia.org/r/301616 
[15:15:04] <hashar>	 thcipriani: that is for the android emulator testing yeah
[15:15:09] <hashar>	 thcipriani: oh that is what I forgot to say yesterday
[15:15:22] <hashar>	 so to polish up the puppet I have created a new role   role::ci::slave::android 
[15:15:25] <grrrit-wm>	 (03Abandoned) 10Alexandros Kosiaris: jenkins-debian-glue: SKIP_COWBUILDER_UPDATE set to true [integration/config] - 10https://gerrit.wikimedia.org/r/301615 (owner: 10Alexandros Kosiaris)
[15:15:33] <hashar>	 and move the emu + kvm membership to that new role class
[15:15:44] <grrrit-wm>	 (03CR) 10Alexandros Kosiaris: [C: 031] debian-gue: stop updating the cow image [integration/config] - 10https://gerrit.wikimedia.org/r/301616 (owner: 10Hashar)
[15:16:16] <grrrit-wm>	 (03Restored) 10Hashar: jenkins-debian-glue: SKIP_COWBUILDER_UPDATE set to true [integration/config] - 10https://gerrit.wikimedia.org/r/301615 (owner: 10Alexandros Kosiaris)
[15:16:27] <grrrit-wm>	 (03Abandoned) 10Hashar: debian-gue: stop updating the cow image [integration/config] - 10https://gerrit.wikimedia.org/r/301616 (owner: 10Hashar)
[15:16:44] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] jenkins-debian-glue: SKIP_COWBUILDER_UPDATE set to true [integration/config] - 10https://gerrit.wikimedia.org/r/301615 (owner: 10Alexandros Kosiaris)
[15:17:42] <grrrit-wm>	 (03Merged) 10jenkins-bot: jenkins-debian-glue: SKIP_COWBUILDER_UPDATE set to true [integration/config] - 10https://gerrit.wikimedia.org/r/301615 (owner: 10Alexandros Kosiaris)
[15:17:46] <thcipriani>	 hashar: but is jenkins-deploy kvm membership no longer needed for trusty?
[15:17:52] <hashar>	 yeah
[15:17:54] <hashar>	 well
[15:17:56] <hashar>	 it is still around
[15:18:09] <hashar>	 moving the stanza in puppet hasn't removed jenkins-deploy from kvm
[15:18:12] <hashar>	 it is still present
[15:18:25] <hashar>	 though if we had to rebuild the trusty slave out of puppet,  that will be missing
[15:18:59] <hashar>	 and https://wikitech.wikimedia.org/wiki/Nova_Resource:Integration-slave-jessie-android.integration.eqiad.wmflabs
[15:19:05] <hashar>	 Puppet Class:  role::ci::slave::android
[15:19:11] <thcipriani>	 yup.
[15:19:39] <hashar>	 with my crap puppet being https://gerrit.wikimedia.org/r/#/c/300738/  cherry picked on puppetmaster
[15:20:31] <thcipriani>	 puppet thing needs to merge, for sure. OK, I'm going to restore the block removed from labs.pp in case we need to build a new trusty slave. Maybe do a bit of massage so it's not needed in 2 places. Sound good?
[15:20:51] <paladox>	 ostriches, hi i found another new feature in gerrit 2.12.
[15:21:23] <paladox>	 When the commit msg is full, there is a little arrow that points down you click that and it shows your commit msg without diagnal bars
[15:21:30] <paladox>	 ie it shows the full commit msg
[15:21:45] <hashar>	 neat
[15:25:38] <ostriches>	 Yeah I don't like that. I was looking at some javascript to leave it expanded by default :)
[15:25:54] <ostriches>	 (otherwise you're like "why can't I see the full message?" "Oh, this little tiny arrow...."
[15:26:02] <paladox>	 Oh
[15:26:02] <ostriches>	 (people mentioned it on the "please test gerrit" thread)
[15:26:03] <paladox>	 yeh
[15:26:15] <paladox>	 I improved it
[15:26:16] <paladox>	 a bit
[15:26:19] <ostriches>	 I tried doing it just with CSS but JS seems to override it
[15:26:20] <ostriches>	 :)
[15:26:26] <paladox>	 with pre wrap the message
[15:26:33] <paladox>	 so the horrizontal bars doint show
[15:26:50] <paladox>	 But im working on phabricator links
[15:27:11] <paladox>	 ostriches this works https://gerrit.wikimedia.org/r/#/c/256663/
[15:27:16] <paladox>	 except from when you do
[15:27:18] <paladox>	 for example
[15:27:43] <paladox>	 just this T34 in the comment message no links just plain text like that it dosent work in comment box
[15:27:43] <stashbot>	 T34: Configure Phabricator for our needs - https://phabricator.wikimedia.org/T34
[15:27:50] <paladox>	 LOL
[15:28:00] <paladox>	 Im trying to fix that so it works
[15:28:13] <ostriches>	 Yeah not having horiz. scrollbars is good
[15:28:19] <paladox>	 Yep
[15:28:19] <ostriches>	 But expand the full height would be nice :)
[15:28:35] <paladox>	 Oh maybe that could be possible
[15:28:40] <paladox>	 Do you mean by default
[15:28:42] <paladox>	 or a button
[15:29:10] <paladox>	 ostriches ^^
[15:29:11] <ostriches>	 I wanna expand it and remove the open/close button entirely
[15:29:16] <ostriches>	 Which is why I tried CSS
[15:29:22] <paladox>	 Oh
[15:29:59] <ostriches>	 I tried tweaking the overflow property but you end up with weird floating text.
[15:30:03] <ostriches>	 Need a clear: somewhere, probably
[15:32:09] <paladox>	 MaybeOh
[15:34:33] <paladox>	 ostriches maybe we could find the js it does
[15:34:39] <paladox>	 and see what css it applir
[15:34:41] <paladox>	 applied
[15:34:44] <paladox>	 applies
[15:35:56] <grrrit-wm>	 (03PS1) 10Hashar: operations/debs/pybal add back tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/301622 
[15:36:51] <ostriches>	 paladox: The JS swaps com-google-gerrit-client-change-CommitBox_BinderImpl_GenCss_style-collapsed for com-google-gerrit-client-change-CommitBox_BinderImpl_GenCss_style-expanded
[15:37:01] <paladox>	 Ah
[15:37:07] <paladox>	 Thanks
[15:37:13] <paladox>	 Im looking on the github mirror
[15:37:19] <ostriches>	 Which is why I was thinking a JS snippet (somewhere?) could do the same automatically.
[15:37:20] <ostriches>	 :)
[15:38:04] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] operations/debs/pybal add back tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/301622 (owner: 10Hashar)
[15:38:18] <paladox>	 Oh
[15:38:20] <Glaisher>	 ostriches: will the gerrit upgrade make most of the keyboard shortcuts work properly again?
[15:38:31] <ostriches>	 To 2.12.3? Yep
[15:38:39] <ostriches>	 Hoping to have that in the next couple of days at most.
[15:38:40] <paladox>	 ostriches found it
[15:38:41] <paladox>	 https://github.com/gerrit-review/gerrit/blob/b40933b3680f6c8e82de541f79a5bf9c0523753a/polygerrit-ui/app/elements/change/gr-message/gr-message.js#L43
[15:38:46] <paladox>	 ostriches :)
[15:39:00] <Glaisher>	 Cool. I have been having a hard time because Ctrl+C and Ctrl+T doesn't work
[15:39:03] <wmf-insecte>	 Project selenium-MobileFrontend » chrome,beta,Linux,contintLabsSlave && UbuntuTrusty build #93: 04FAILURE in 17 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=chrome,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/93/
[15:39:09] <paladox>	 ostriches i really hope phabricator support on the web editing
[15:39:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: operations/debs/pybal add back tox-jessie [integration/config] - 10https://gerrit.wikimedia.org/r/301622 (owner: 10Hashar)
[15:39:30] <paladox>	 make it easy for windows users to edit files that are for unix only
[15:39:31] <paladox>	 like
[15:39:33] <paladox>	 for example
[15:39:36] <paladox>	 Example.txt
[15:39:37] <paladox>	 and 
[15:39:40] <paladox>	 example.txt
[15:39:47] <paladox>	 they will merge together on windows
[15:40:19] <paladox>	 Glaisher i have a test install you can try in a min.
[15:40:34] <ostriches>	 Well if people do that they're already making life hard for their Windows friends :)
[15:40:38] <paladox>	 Im testing something so it will be on and off since gerrit needs restarts to test things
[15:40:43] <Glaisher>	 paladox: link?
[15:40:58] <paladox>	 Glaisher http://gerrit-test.wmflabs.org/
[15:41:10] <paladox>	 ostriches not really i have BASH now.
[15:41:21] <paladox>	 So windows 10 users can do it already 
[15:41:27] <paladox>	 in 4 days
[15:41:34] <Glaisher>	 paladox: 503 for me
[15:41:37] <paladox>	 Yeh
[15:41:40] <paladox>	 I need to start
[15:41:41] <paladox>	 it
[15:41:41] <ostriches>	 Well, either way, having two files only differing in case is just Confusing :)
[15:41:48] <ostriches>	 I mean, what's the difference between the two?
[15:42:03] <ostriches>	 Is "important.txt" less important than "IMPORTANT.TXT" ?
[15:42:04] <ostriches>	 lolol
[15:42:10] <paladox>	 Well jenkins only tests by repo name
[15:42:12] <ostriches>	 Must be!
[15:42:13] <ostriches>	 :)
[15:42:38] <paladox>	 So for example if a repo name is testing and your main php file is Testing.php 
[15:42:41] <paladox>	 it will fail
[15:42:49] <paladox>	 so you sybmlink it to prevent failures
[15:43:10] <paladox>	 Glaisher ive started it, will take few mins to start not sure why though
[15:43:33] <paladox>	 ostriches but the good thing is people are asking microsoft to support that
[15:43:36] <paladox>	 and there is 
[15:43:39] <Glaisher>	 paladox: ok, will try it again in a moment
[15:43:42] <paladox>	 Ok
[15:43:49] <paladox>	 I will tell you when it is started
[15:43:50] <paladox>	 :)
[15:44:00] <Glaisher>	 :)
[15:44:03] <paladox>	 Microsoft supports unix formats
[15:44:08] <paladox>	 but only in basg
[15:44:09] <paladox>	 bash
[15:44:38] <ostriches>	 I imagine most windows users aren't using bash :)
[15:44:40] <paladox>	 ostriches but 400 million people will have bash ready to install in 4 days when windows 10 redstone 1 is released
[15:44:43] <ostriches>	 Even if they're on Win10 :p
[15:44:46] <paladox>	 ostriches actually they are
[15:45:05] <paladox>	 ostriches they are realeasing bash next week in the windows 10 update due to popularity
[15:45:10] <ostriches>	 No no no. My point is that most people aren't using a command line at all, much less bash ;-)
[15:45:29] <paladox>	 ostriches oh you can start the ubuntu desktop
[15:45:33] <paladox>	 natively
[15:45:36] <paladox>	 against windows 10
[15:45:37] <paladox>	 lol
[15:46:09] <ostriches>	 Anyway, neither here nor there.
[15:46:26] <paladox>	 ostriches see https://github.com/Microsoft/BashOnWindows/issues/637 please
[15:46:29] <ostriches>	 I prefer OSX which is like pretend *nix :p
[15:46:33] <paladox>	 Yeh i like
[15:46:35] <paladox>	 osx
[15:46:40] <paladox>	 but so expensive for memory
[15:46:44] <wmf-insecte>	 Project selenium-MobileFrontend » firefox,beta,Linux,contintLabsSlave && UbuntuTrusty build #93: 04FAILURE in 24 min: https://integration.wikimedia.org/ci/job/selenium-MobileFrontend/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=contintLabsSlave%20&&%20UbuntuTrusty/93/
[15:47:07] <paladox>	 ostriches I got a 1tb with 6gb ram for £299 from very.co.uk
[15:47:17] <ostriches>	 paladox: I crammed 16g into my last two macbooks when I bought them :p
[15:47:21] <ostriches>	 Still haven't had to upgrade!
[15:47:23] <paladox>	 Oh lol
[15:47:32] <paladox>	 I got an ipad pro 12inch
[15:47:43] <paladox>	 Which is bigger then the apple mac air
[15:48:02] <ostriches>	 16g is nice. I can throw like 8g at my VM and that still leaves *plenty* for the host OS.
[15:48:15] <paladox>	 Yeh
[15:48:43] <paladox>	 I spends most of the times buying iphones
[15:48:44] <paladox>	 and ipads
[15:49:05] <paladox>	 i own two iphones, iphone 5s and 6 plus (Just sold the iphone 5s).
[15:49:26] <paladox>	 I own 3 ipads, ipad 2nd gen, ipad 4th gen and an ipad pro 12inch
[15:50:09] <paladox>	 ostriches but what is so funny is i brough the 2nd gen apple tv, i waited a few years and got tired of waiting for them to bring out the next big update for apple tv so i brough an apple tv 3rd gen
[15:50:19] <paladox>	 but within the year apple tv 4th gen came out
[15:50:30] <paladox>	 it was like why now when i waited years for it to come out
[15:51:11] <paladox>	 but lucky i got a £25 itunes gift card for free with apple tv 3rd gen
[15:51:45] <paladox>	 But as my iphone is american and i got it repaired in the uk they replaced it for free and didnt have to pay the excess charge
[15:51:46] <paladox>	 lol
[15:53:12] <paladox>	 ostriches but i know people are trying to get osx installed using wsl
[15:53:13] <paladox>	 lol
[15:55:47] <ostriches>	 lol but why?
[15:55:50] <ostriches>	 Just buy a mac then
[15:56:04] <paladox>	 Im not sure
[15:56:06] <paladox>	 why i doint
[15:56:12] <paladox>	 but i really do like 1tb
[15:56:20] <paladox>	 only used 200gb 
[15:57:05] <paladox>	 ostriches but you can buy warrenty up to 6 years after you purchese the iphone from apple
[15:57:20] <paladox>	 so in america it is 30 days, here it is 6 years due to the law
[15:57:45] <ostriches>	 I don't have an iphone or ipad. Not a huge fan of iOS tbh.
[15:57:48] <paladox>	 Oh
[15:57:55] <paladox>	 I really like iOS 10
[15:57:56] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1013 is CRITICAL: CRITICAL: 40.00% of data above the critical threshold [0.0]
[15:58:07] <paladox>	 it is in beta.
[15:58:11] <ostriches>	 I've been with android since the HTC G1/Dream
[15:58:14] <paladox>	 But i get widgets
[15:58:16] <paladox>	 oh
[15:58:29] <paladox>	 HTC, do you get android 6
[15:58:31] <paladox>	 ?
[15:58:55] <ostriches>	 I don't have an HTC now. Rocking an LG V10 right now, and yeah I'm on Android 6.
[15:59:02] <paladox>	 Oh
[15:59:06] <paladox>	 I have a lg tv
[15:59:09] <ostriches>	 LG puts *damn* good cameras in their phones so I've been using them for the last few phones I've had
[15:59:10] <paladox>	 and samsung tv
[15:59:14] <paladox>	 Oh
[15:59:16] <paladox>	 LOL
[15:59:18] <paladox>	 My phone
[15:59:30] <paladox>	 now looks through my photos with my permission of course
[15:59:33] <paladox>	 and makes videos
[15:59:40] <paladox>	 plus face detection
[15:59:41] <paladox>	 lol
[16:00:01] <paladox>	 I really do like the answers siri gives, so funny
[16:00:27] <paladox>	 I got tomtom on my phone.
[16:00:29] <paladox>	 lol
[16:00:46] <paladox>	 i do like big screen phones so went with iphone 6 plus
[16:01:20] <ostriches>	 "The LG V10 also features a 5.7-inch screen with a 1440 x 2560 resolution. That works out to a 513ppi pixel density. The secondary screen measures 2.1-inches and has a 160 x 1040 resolution, also resulting in a 513ppi pixel density."
[16:01:29] <ostriches>	 The second screen was a big selling point for me.
[16:01:34] <paladox>	 Oh
[16:01:39] <paladox>	 Mine is a 5.5 inch
[16:01:48] <paladox>	 with a retina hd screen
[16:01:48] <ostriches>	 You can do things like change songs, see if you've got notifications, check the time/date without having to turn on the main screen
[16:01:50] <paladox>	 1080p
[16:01:55] <ostriches>	 Saves sooooooo much battery over time
[16:01:56] <paladox>	 oh
[16:02:23] <paladox>	 mine last me some good times
[16:02:32] <paladox>	 I still carry around a backup power
[16:02:43] <ostriches>	 http://cdn01.androidauthority.net/wp-content/uploads/2015/09/LG-V10-Hands-On-24.jpg
[16:02:48] <paladox>	 Since long journeys i do like to stay online
[16:02:48] <paladox>	 o
[16:02:50] <paladox>	 oh
[16:02:54] <ostriches>	 That little screen at the top with the launch icons
[16:02:59] <ostriches>	 That can turn on without the main screen
[16:03:20] <paladox>	 Oh
[16:03:36] <paladox>	 Actually the iphone 6s now includes when you lift it it shows the screen
[16:03:52] <paladox>	 since apple included a powerful touch id it opened so fast you coulden see your main screen lol
[16:04:04] <paladox>	 I still remeber that imessenger bug
[16:04:28] <ostriches>	 Yeah, mine's based on the accelerometer too. Both stay dark when unused, but if it's laying upright or you pick it up the small screen lights up to show date/time and you can scroll through a couple of small menus.
[16:04:31] <paladox>	 I sent my brothers imessages and it was encoded text that you copied from a website, a blog
[16:04:38] <ostriches>	 Tapping the screen or power button then brings up the main screen
[16:04:42] <paladox>	 oh
[16:05:02] <paladox>	 But this bug resulted in the device being restarted
[16:05:02] <ostriches>	 LG did some research and found that people spend a ton of battery wasted just to see "Do I have a message at all" and "What time is it?"
[16:05:11] <paladox>	 oh lol
[16:05:13] <ostriches>	 So you can get both without wasting battery lighting up the main display :)
[16:05:18] <paladox>	 oh
[16:05:20] <paladox>	 ah
[16:05:21] <paladox>	 :)
[16:05:34] <paladox>	 I have my phone set on do not disturb
[16:05:36] <paladox>	 all day
[16:05:48] <ostriches>	 On a one-off check, it's not bad, but you'd be surprised how much it actually saves over a day!
[16:05:55] <paladox>	 yeh
[16:06:14] <paladox>	 i never watch videos on my phone, i always bring my ipad around
[16:06:23] <paladox>	 i can find my iphone on all those .
[16:06:37] <paladox>	 no one will be stealing anything without a lound noise
[16:06:38] <paladox>	 lol
[16:06:41] <Reedy>	 ostriches: 
[16:06:41] <Reedy>	 remote: Compressing objects: 100% (206760/206760)
[16:06:41] <Reedy>	 error: RPC failed; result=56, HTTP code = 200MiB | 573.00 KiB/s    
[16:06:41] <Reedy>	 fatal: The remote end hung up unexpectedly
[16:06:41] <Reedy>	 fatal: early EOFs:  91% (87406/96050), 16.69 MiB | 661.00 KiB/s   
[16:06:42] <Reedy>	 fatal: index-pack failed
[16:06:44] <Reedy>	 Clone of 'https://gerrit.wikimedia.org/r/p/VisualEditor/VisualEditor.git' into submodule path 'lib/ve' failed
[16:06:48] <ostriches>	 ruh roh
[16:06:50] <ostriches>	 Lemme look
[16:06:53] <Reedy>	 thanks
[16:07:05] <ostriches>	 Reedy: Also I wanna stuff gerrit logs into logstash.
[16:07:10] <Reedy>	 orly?
[16:07:16] <ostriches>	 Yeah
[16:07:17] <Reedy>	 just config? or a lot more work than that?
[16:07:37] <ostriches>	 Should just be some config with log4j I hope
[16:07:43] <paladox>	 Glaisher its started
[16:07:47] <ostriches>	 Maybe some mild massaging on the ingest side
[16:08:59] <paladox>	 Glaisher it runs gerrit 2.12.3
[16:09:07] <paladox>	 You can also test using the admin account
[16:09:11] <Glaisher>	 paladox: It works :)
[16:09:11] <paladox>	 it dosent have a password
[16:09:19] <paladox>	 Glaisher ok thanks :)
[16:09:23] <paladox>	 ostriches ^^
[16:09:25] <ostriches>	 Reedy: Not yours, but lol wtf?
[16:09:31] <ostriches>	 [2016-07-28 16:00:14,722] [Diff-305] ERROR com.google.gerrit.server.patch.Text : Detected charset 'HZ-GB-2312' not supported: java.nio.charset.UnsupportedCharsetException: HZ-GB-2312
[16:09:40] <ostriches>	 (unless that is you....)
[16:09:50] <paladox>	 What is that?
[16:09:52] <Reedy>	 It's Chinese
[16:10:07] <Reedy>	 Simplified Chinese
[16:10:08] <Reedy>	 https://docs.oracle.com/cd/E19683-01/806-6642/new-tbl-72/index.html
[16:10:20] <ostriches>	 [2016-07-28 15:55:56,676] [HTTP-11286] WARN  /r : Internal error during upload-pack from /srv/gerrit/git/VisualEditor/VisualEditor.git
[16:10:20] <ostriches>	 java.io.IOException: java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms
[16:10:26] <ostriches>	 Hmmm, lots of timeouts.
[16:10:31] <ostriches>	 Especially SSH
[16:10:33] <Reedy>	 is it a really big repo?
[16:10:40] <Reedy>	 well, not as big as core presumably
[16:11:56] <ostriches>	 Uno momento....
[16:12:13] <paladox>	 Maybe that is a bug?
[16:13:43] <paladox>	 Maybe
[16:13:44] <paladox>	 https://groups.google.com/forum/#!topic/repo-discuss/t1lWeA8ebkc
[16:14:08] <paladox>	 Reedy did you abort ssh ?
[16:14:27] <Reedy>	 No
[16:14:32] <paladox>	 Ok
[16:14:41] <ostriches>	 Reedy: Pretty poorly packed.
[16:14:47] <ostriches>	 Went 107M -> 45M
[16:14:50] * Reedy blames James_F|Away
[16:14:51] <ostriches>	 But still.
[16:14:54] <ostriches>	 Shouldn't be it.
[16:14:59] <Reedy>	 Probably a lot of churn
[16:15:13] <paladox>	 Doint we limit objects if it is that big
[16:15:16] <paladox>	 it would have failed
[16:15:27] <Reedy>	 MW core is a lot bigger
[16:15:32] <paladox>	 yep
[16:16:51] <ostriches>	 We limit the size of an individual object upload.
[16:16:59] <paladox>	 Oh
[16:17:01] <ostriches>	 So you can't upload like a 100MB tar file and screw everyone
[16:17:07] <ostriches>	 But existing packs can easily be that big.
[16:17:18] <ostriches>	 MW core hovers around 500M - 1G, depending on pack.
[16:17:49] <paladox>	 Yep
[16:17:51] <paladox>	 idleTimeout = 864000
[16:17:58] <Reedy>	 cloning it to tool labs is stupidly slow
[16:18:34] <Reedy>	 Receiving objects:   8% (8332/96050), 2.36 MiB | 26.00 KiB/s     
[16:19:54] <ostriches>	 I can do it over https in 7 seconds.
[16:19:55] <ostriches>	 To home
[16:19:56] <ostriches>	 Hmmm
[16:20:35] <ostriches>	 10s over ssh
[16:21:04] <ostriches>	 (6.85 avg mbps vs 3.67 avg mbps)
[16:21:11] <Reedy>	 dunno if it's disk issues etc too on tools
[16:23:08] <ostriches>	 Could try iperf to see if it's network or local IO
[16:24:01] <Reedy>	 it's taking ages to resolve deltas too
[16:24:10] <Reedy>	 so I presume it's local io issue
[16:24:56] <ostriches>	 How's it to your local machine?
[16:25:29] <paladox>	 Reedy this looks like the same problem 
[16:25:30] <paladox>	 https://bugs.chromium.org/p/gerrit/issues/detail?id=2295
[16:25:32] <paladox>	 ostriches ^^
[16:26:14] <Reedy>	 ostriches: silly quick to a vm I use
[16:26:16] <paladox>	 In my case, the reason is that my repository size (200M) is larger than my git server's memory (128M).
[16:26:40] <Reedy>	 So that's not gerrit
[16:26:44] <ostriches>	 Our gerrit server is at 20g heap, speaking of, we should go ahead and bump that to 28
[16:26:49] <ostriches>	 Reedy: Yeah unlikely.
[16:26:57] <ostriches>	 So you end up hitting a timeout of SSH
[16:27:01] <ostriches>	 Gerrit disconnections
[16:27:04] <ostriches>	 And git goes wtf?
[16:27:04] <Reedy>	 heh
[16:27:10] <Reedy>	 That... sucks
[16:27:19] <paladox>	 ostriches maybe that is it, yeh we should bump it
[16:27:49] <ostriches>	 We could also enable ssh compression
[16:28:03] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-jessie-1001 is CRITICAL: CRITICAL: 10.00% of data above the critical threshold [0.0]
[16:28:09] <ostriches>	 In the general case, we want to disable transparent compression, since the majority of our data transfer is highly compressed Git pack files and we cannot make them any smaller than they already are.
[16:28:09] <ostriches>	 However, if there are CPU in abundance and the server is reachable through slow networks, gits with huge amount of refs can benefit from SSH-compression since git does not compress the ref announcement during handshake.
[16:28:31] <paladox>	 ostriches what is packedGitWindowSize 
[16:29:04] <paladox>	 and windowMemory 
[16:29:10] <ostriches>	 Default.
[16:29:17] <ostriches>	 Doesn't matter much, since it's not *our end* that fails.
[16:29:20] <paladox>	 oh
[16:29:22] <ostriches>	 technically
[16:29:23] <paladox>	 ok
[16:29:26] <ostriches>	 But yeah, using defaults there.
[16:29:27] <paladox>	 yeh
[16:30:32] <paladox>	 Lol mw had this problem before
[16:30:34] <paladox>	 https://phabricator.wikimedia.org/T46129
[16:30:38] <paladox>	 ostriches Reedy ^^
[16:32:06] <ostriches>	 Nah, not the same problem.
[16:33:26] <ostriches>	 Eh, similar.
[16:33:34] <ostriches>	 That was more regarding shallow clones.
[16:33:51] <paladox>	 Oh
[16:33:59] <paladox>	 COuld it be git version related
[16:34:09] <ostriches>	 Sometimes things like that can happen.
[16:34:12] <paladox>	 What version is lead of git is it running
[16:34:12] <paladox>	 ?
[16:34:19] <ostriches>	 I always recommend people run the *latest* version of git they can.
[16:34:26] <paladox>	 Reedy also what version of git do you run
[16:34:28] <ostriches>	 Anything below 1.8 is a mess w.r.t. performance.
[16:34:28] <paladox>	 please
[16:34:29] <paladox>	 ?
[16:34:32] <paladox>	 yep
[16:34:34] <Reedy>	 On what host?
[16:34:41] <paladox>	 Where you had the problem
[16:34:42] <paladox>	 ?
[16:34:48] <Reedy>	 tools-bastion-03 is on 14.04
[16:34:50] <Reedy>	 so lololdgit
[16:34:58] <paladox>	 Oh that may be why
[16:34:59] <ostriches>	 Yeah I have a bug about wanting to backport a newer git.
[16:34:59] <paladox>	 lol
[16:35:08] <paladox>	 they are using 1.*
[16:35:26] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.14 deployment blockers - https://phabricator.wikimedia.org/T139217#2502251 (10greg) a:03mmodell
[16:36:11] <paladox>	 ostriches but i belive sudo add-apt-repository ppa:git-core/ppa
[16:36:12] <ostriches>	 Reedy: Which version? 1.7.x?
[16:36:15] <Reedy>	 1.9.1
[16:36:16] <paladox>	 will install the new version
[16:36:18] <ostriches>	 Eh, ok.
[16:36:23] <paladox>	 http://askubuntu.com/questions/579589/upgrade-git-version-on-ubuntu-14-04
[16:36:24] <Reedy>	 paladox: No, we're not adding a ppa
[16:36:28] <paladox>	 Ok
[16:36:47] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2502252 (10demon) Would be nice for tool users too, who are routinely doing git operations and are stuck on an ancient 1.9.x as well.
[16:36:49] <paladox>	 Or just backport it from the newer ubuntu releases and package it
[16:37:00] <Reedy>	 yeah, exactly
[16:37:03] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2502254 (10greg) a:03mmodell
[16:37:07] <Reedy>	 Which is what the ops guys will likely do
[16:38:03] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-jessie-1001 is OK: OK: Less than 1.00% above the threshold [0.0]
[16:38:11] <paladox>	 https://launchpad.net/~git-core/+archive/ubuntu/ppa
[16:38:31] <paladox>	 Reedy you can just download the dpkg from 16.04 onto 14.04
[16:38:32] <paladox>	 for git
[16:38:38] <paladox>	 from ubuntu offical website
[16:38:48] <Reedy>	 Do all the dependancies tie up?
[16:39:22] <paladox>	 http://packages.ubuntu.com/xenial/git
[16:39:26] <paladox>	 Reedy not sure
[16:39:28] <ostriches>	 Jessie only has 2.1.4 lol
[16:39:32] <paladox>	 you can give it ago
[16:39:45] <Reedy>	 No benefit for me to do so
[16:39:59] <paladox>	 http://packages.ubuntu.com/xenial/amd64/git/download
[16:40:01] <Reedy>	 I run recent git on the hosts I can control/care about
[16:40:08] <paladox>	 oh ok
[16:41:06] <ostriches>	 Reedy: Deps on jessie are: git-man, libc6, libcurl3-gnutls, liberror-perl, libexpat1, libpcre3, perl-modules, zlib1g
[16:41:10] <ostriches>	 Pretty run of the mill stuff.
[16:41:11] <Reedy>	 versions?
[16:41:12] <Reedy>	 :P
[16:41:21] <ostriches>	 https://packages.debian.org/jessie/git
[16:41:25] <ostriches>	 #toolazy
[16:41:33] <paladox>	 LOL
[16:41:34] <Reedy>	 error: The following untracked working tree files would be overwritten by checkout:
[16:41:34] <Reedy>	 	images/code.png
[16:41:34] <Reedy>	 Please move or remove them before you can switch branches.
[16:41:34] <Reedy>	 Aborting
[16:41:35] <Reedy>	 stfu git
[16:41:41] <paladox>	 oh
[16:41:43] <Reedy>	 IN WHICH TREE
[16:41:49] <Reedy>	 SO HELP ME GOD
[16:41:52] <ostriches>	 All the trees.
[16:42:14] <paladox>	 git reset --hard origin/<bracnh>
[16:42:31] <Reedy>	 it's in one of the over 800 submodules
[16:42:33] <Reedy>	 That won't help
[16:42:39] <Reedy>	 unless run in a git submodule foreach
[16:42:46] <ostriches>	 lol git submodule foreach
[16:42:47] <ostriches>	 so slow
[16:42:51] <paladox>	 LOL
[16:43:10] <ostriches>	 It literally is just like a `for F in */; do ...; done;`
[16:43:14] <ostriches>	 Sooooo slow
[16:43:20] <ostriches>	 It needs --jobs like fetch has
[16:43:21] <paladox>	 LOL
[16:43:36] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Packaging: debian-glue have the pbuilderrc override cowbuilder parameters - https://phabricator.wikimedia.org/T141538#2502289 (10hashar)
[16:45:29] <paladox>	 ostriches doing that would not be slow for me on 52mbps-80mbps connection
[16:45:30] <paladox>	 lol
[16:45:44] * paladox does like being on bt i get faster speeds lol
[16:45:48] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2502307 (10greg) Ori is out for family reasons right now, but since he helped craft this alert I'm adding him here for his t...
[16:45:56] <wikibugs>	 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Sprint-77-Segmentation-fault, and 5 others: Spike [2hrs] Wikidata description browser tests do not run anywhere - https://phabricator.wikimedia.org/T137756#2502309 (10Jdlrobson) 05Open>03Resolved a:03Jdlrobson These are running a...
[16:48:04] <paladox>	 Does anyone know how to get phabricator links working in gerrit
[16:48:08] <paladox>	 ive done http://gerrit-test.wmflabs.org/gerrit/#/c/16/
[16:48:11] <paladox>	 ostriches ^^
[16:48:27] <paladox>	 I applied https://gerrit.wikimedia.org/r/#/c/256663/
[16:48:45] <paladox>	 But if it is a plain comment like T1 it dosent link to phabricator
[16:48:46] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[16:48:50] <paladox>	 LOL
[16:49:05] <ostriches>	 They are working?
[16:49:11] <ostriches>	 You just used an anchor that doesn't exist.
[16:49:19] <ostriches>	 (they don't go #1 #2 #3 #4)
[16:49:37] <ostriches>	 Oh hmm
[16:49:58] <ostriches>	 Oh I see yeah
[16:50:03] <paladox>	 Yep
[16:50:14] <paladox>	 Once we figure out that part we can merge it
[16:50:20] <paladox>	 and links will work again
[16:50:23] <paladox>	 and look better
[16:50:24] <paladox>	 :)
[16:52:02] <ostriches>	 Could capture them in one big group actually.
[16:52:26] <paladox>	 Oh
[16:52:28] <paladox>	 how?
[16:52:32] <ostriches>	 So instead of:
[16:52:34] <ostriches>	 "\\bT(\\d+)(#\\d+)?\\b(?![#\"<])"
[16:52:41] <paladox>	 Yep
[16:52:43] <ostriches>	 Oh wait.
[16:52:44] <ostriches>	 One sec.
[16:52:46] <paladox>	 Ok
[16:52:47] <ostriches>	 You added the lookahead.
[16:52:52] <paladox>	 Oh
[16:52:56] <ostriches>	 Let's not do that in the same patch....lookaheads are weird.
[16:53:06] <paladox>	 Yep
[16:53:46] <ostriches>	 Soooo, here we go. I suggested this:
[16:53:48] <ostriches>	 \\bT(\\d+)(#\\d+)?\\b
[16:53:52] <ostriches>	 We should do....
[16:54:00] <paladox>	 Ok, but that will also break the link again
[16:54:11] <paladox>	 That will fix T1
[16:54:11] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[16:54:13] <ostriches>	 \\b(T\\d+(#\\d+)?)\\b
[16:54:21] <paladox>	 Oh
[16:54:23] <paladox>	 i see now
[16:54:25] <ostriches>	 Then you just need $1
[16:54:26] <paladox>	 Lets try that
[16:54:32] <paladox>	 Yep
[16:54:34] <ostriches>	 Drop the T and just do /$1 as the dest
[16:54:40] <paladox>	 Ok
[16:54:45] <ostriches>	 Make sense?
[16:54:47] <paladox>	 Yep
[16:54:52] <paladox>	 Im applying it now
[16:55:12] <ostriches>	 Ok yeah let's do that on 301580, then we'll revisit 256663 (lookahead) later.
[16:55:16] <ostriches>	 That second one is harder.
[16:55:22] <paladox>	 ok
[16:55:29] <paladox>	 so
[16:55:30] <paladox>	 this
[16:55:37] <paladox>	 \\b(T\\d+(#\\d+)?)\\b wont fix the ugly links ?
[16:55:42] <paladox>	 ie
[16:55:52] <paladox>	 phabricator.wikimedia.org/T1
[16:56:14] <ostriches>	 Hmmm, I can think of a fairly dumb (and easy) fix for that....
[16:56:26] <paladox>	 Oh
[16:56:41] * paladox i will wait for your fix lol
[16:57:09] <ostriches>	 Err, nah.
[16:57:11] <ostriches>	 Let's just do this now
[16:57:13] <paladox>	 Oh
[16:57:14] <ostriches>	 I don't have it right
[16:57:15] <paladox>	 Ok
[16:57:15] <ostriches>	 :)
[16:57:25] <ostriches>	 K, gonna walk my puppy, I'll have a look when I'm back
[16:57:31] <paladox>	 Ok
[16:57:32] <paladox>	 :)
[16:57:36] <paladox>	 Isent that a dog
[16:57:56] <ostriches>	 lol yeah, but he's always gonna be a puppy to me <3
[16:58:11] <paladox>	 Oh ah, arnt puppy's small dogs or big ones
[16:58:14] <paladox>	 ?
[17:04:39] <andre__>	 Small but they get bigger over time, I've been told. :)
[17:04:44] <paladox>	 Oh
[17:04:50] <paladox>	 thanks
[17:04:51] <paladox>	 :)
[17:05:36] <ostriches>	 Puppeh!!!! https://anyonecanedit.org/plato_pumpkins.jpg
[17:05:47] <ostriches>	 (that was last fall)
[17:05:52] <paladox>	 oh
[17:06:01] <paladox>	 ostriches i also may have found another fix
[17:06:01] <paladox>	 for
[17:06:06] <paladox>	 refs/meta/config links
[17:06:12] <paladox>	 in gerrit redirecting to diffusion
[17:06:44] <paladox>	 We can just strip refs/meta/config on phabricator side and leave it with just refs as the branch then add the commit at the end of url
[17:06:47] <paladox>	 seems to work
[17:06:48] <paladox>	 lol
[17:07:04] <paladox>	 LOL
[17:07:18] <paladox>	 ostriches is that your puppeh
[17:07:34] <ostriches>	 mah pupppppyyyyy <3 <3
[17:07:43] <paladox>	 Oh :) 
[17:11:44] <paladox>	 :)
[17:21:05] <paladox>	 ostriches done, ive submitted the patch now, ive tested it 
[17:21:17] <paladox>	 we will need to figure out how to fix the broken
[17:21:21] <paladox>	 link though
[17:21:24] <paladox>	 in the commit box
[17:21:38] <paladox>	 Ie
[17:21:39] <paladox>	 T76459" target="_blank">https://phabricator.wikimedia.org/T76459
[17:21:40] <stashbot>	 T76459: Allow links to phabricator comments from gerrit messages. - https://phabricator.wikimedia.org/T76459
[17:21:54] <paladox>	 :)
[17:26:02] <paladox>	 http://www.neowin.net/news/youve-got-one-day-to-upgrade-to-windows-10-for-free---do-it-even-if-you-dont-want-it
[17:26:04] <paladox>	 lol
[17:26:15] <ostriches>	 Minor inline comment on the regex/link, otherwise good.
[17:43:52] <paladox>	 ostriches ok
[17:43:54] <paladox>	 done 
[17:43:55] <paladox>	 thanks
[17:43:56] <paladox>	 :)
[17:44:10] <paladox>	 ostriches that was quick walking the puppy :)
[17:46:34] <paladox>	 ?
[17:48:17] <mutante>	 ostriches: puppet dependency cycle that the compiler did not see .. boo
[17:48:40] <mutante>	 it's the Letsencrypt part.. looking
[17:48:51] <mutante>	 (not a user facing issue)
[17:48:52] <ostriches>	 Ugh, sorry.
[17:49:11] <mutante>	 well, i am boo-ing at puppet not you
[17:49:23] <mutante>	 how it does not catch that.. hrmm
[17:49:36] <mutante>	 we both checked everything
[17:53:07] <paladox>	 ostriches have you walked the puppy already ?
[17:54:02] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2502702 (10greg)
[17:54:04] <paladox>	 https://gerrit.wikimedia.org/r/#/c/301580/ is now ready :)
[17:54:05] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Identify inaugural SWAT members for the European SWAT window - https://phabricator.wikimedia.org/T139544#2502699 (10greg) 05Open>03Resolved a:03greg Alright, I think we've identified everyone and @hashar's plan at T137970#2501313 makes sens...
[17:54:31] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2385565 (10greg) From the subtask of who will do it:  **Experienced deployers (can help train)** //These are the names that will be initially in the list of SWATers// * @hashar  * @D...
[17:58:12] <ostriches>	 All walked, back to work now
[17:58:52] <paladox>	 Ok
[17:58:53] <paladox>	 :)
[17:58:59] <ostriches>	 mutante: What's the error?
[17:59:22] <mutante>	 ostriches: https://gerrit.wikimedia.org/r/#/c/301655/
[17:59:37] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2502733 (10greg)
[17:59:40] <ostriches>	 Ahhh, I see it
[17:59:55] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.15 deployment blockers - https://phabricator.wikimedia.org/T140971#2482849 (10greg)
[18:00:07] <ostriches>	 mutante: Ah, I see what I did.
[18:00:14] <ostriches>	 There was no need for that.
[18:00:18] <ostriches>	 Lame that compiler didn't catch it
[18:00:38] <mutante>	 ostriches: yep, it's interesting that it doesnt get that
[18:01:00] <mutante>	 let me merge that
[18:12:47] <wikibugs>	 06Release-Engineering-Team, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2502772 (10greg) Added week of August 22nd to the Deploy calendar: https://wikitech.wikimedia.org/wiki/Deployments#Week_of_August_22nd  Added corresponding events/changed the morning...
[18:12:55] <paladox>	 ostriches hi, would we be able to work on the broken phabricator links please?
[18:13:05] <paladox>	 this
[18:13:06] <paladox>	 T76459" target="_blank">https://phabricator.wikimedia.org/T76459
[18:13:06] <stashbot>	 T76459: Allow links to phabricator comments from gerrit messages. - https://phabricator.wikimedia.org/T76459
[18:13:40] <mutante>	 paladox: there's a meeting right now
[18:13:52] <paladox>	 Oh
[18:13:52] <mutante>	 but your fix is being merged
[18:13:55] <paladox>	 Ok
[18:13:59] <paladox>	 this is something else
[18:14:13] <paladox>	 That fixes doing T1#1 
[18:14:13] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[18:14:25] <paladox>	 im on about fixing phabricator http links
[18:27:40] <shinken-wm>	 PROBLEM - SSH on deployment-changeprop is CRITICAL: CRITICAL - Socket timeout after 10 seconds
[18:38:05] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Add a European mid-day SWAT window - https://phabricator.wikimedia.org/T137970#2502856 (10greg) a:03greg
[18:40:22] <paladox>	 ostriches we could expand comments by using js to overide https://github.com/gerrit-review/gerrit/blob/adfeb643a259ab709cb31a498e90a9ada3de057a/gerrit-gwtui/src/main/java/com/google/gerrit/client/change/CommitBox.java#L83
[18:41:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream, 07WorkType-Maintenance: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2079136 (10greg) Is this still UBN! ?
[18:49:14] <grrrit-wm>	 (03CR) 10Chad: "We should resurrect this." [integration/config] - 10https://gerrit.wikimedia.org/r/286207 (https://phabricator.wikimedia.org/T129357) (owner: 10Hashar)
[18:56:11] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1012 is CRITICAL: CRITICAL: 44.44% of data above the critical threshold [0.0]
[19:02:52] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2502892 (10hashar) The alarm is not delayed by half an hour. It is delay but not as much as I thought.  Looking at the green...
[19:05:19] <bd808>	 thcipriani: could you give this scap3 checks.yaml a quick look? -- https://gerrit.wikimedia.org/r/#/c/301662
[19:05:45] * bd808 -> lunch
[19:05:52] <thcipriani>	 bd808: yup
[19:06:46] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Zuul, 07Upstream, 07WorkType-Maintenance: Zuul deadlocks if unknown repo has activity in Gerrit - https://phabricator.wikimedia.org/T128569#2502893 (10hashar) p:05Unbreak!>03High I got Zuul upgraded on July 27th at 14:00 UTC: `Zuul version: 2.1.0-391-gbc58ea3-...
[19:31:15] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1012 is OK: OK: Less than 1.00% above the threshold [0.0]
[19:31:20] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1006 is CRITICAL: CRITICAL: 33.33% of data above the critical threshold [0.0]
[19:45:42] <paladox>	 ostriches hi, thanks for reverting my change
[19:45:48] <paladox>	 i think i may know
[19:45:49] <paladox>	 why
[19:45:58] <paladox>	 may be because gerribot needs to also be updated
[19:46:03] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2502733 (10hashar) I should be able to handle this one. With the European swat starting that will be my deploy week :]
[19:46:11] <paladox>	 but do you know why?
[19:47:36] <mutante>	 paladox: see his comment on the revert
[19:47:53] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: "MediaWiki exceptions and fatals per minute" alarm is too slow (half an hour delay!) - https://phabricator.wikimedia.org/T141520#2503030 (10hashar) p:05Triage>03Normal
[19:47:57] <paladox>	 Ok
[19:53:37] <paladox>	 thanks
[19:53:49] <paladox>	 i did a different fix that should not break it this time
[19:53:57] <paladox>	 im just starting gerrit-test with the fix now
[19:54:00] <paladox>	 mutante ^^
[19:54:01] <paladox>	 :)
[19:54:09] <paladox>	 once started i will log it in -labs
[19:54:36] <wmf-insecte>	 Project beta-code-update-eqiad build #114782: 04FAILURE in 1 min 34 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/114782/
[19:54:38] <mutante>	 ok. i restarted grrrit-wm again just now.  the next service restart has to wait at least a little bit
[19:54:48] <paladox>	 ok
[19:54:50] <paladox>	 thanks
[19:55:00] <mutante>	 thanks for logging!
[19:55:33] <paladox>	 your welcome
[19:56:00] <paladox>	 mutante you logged the gerrit-test site down on the wrong project
[19:56:06] <paladox>	 it is git not gerrit
[20:01:54] <mutante>	 paladox: why would gerrit not be in gerrit?
[20:02:19] <paladox>	 Since i have gerrit-test created under project git
[20:02:22] <paladox>	 mutante ^^
[20:02:35] <paladox>	 i havent been added as a user to the gerrit project
[20:02:52] <mutante>	 ok
[20:02:56] <mutante>	 and that one wasnt me
[20:03:04] <mutante>	 as opposed as to the other restarts
[20:04:29] <paladox>	 Oh
[20:04:36] <wmf-insecte>	 Yippee, build fixed!
[20:04:36] <wmf-insecte>	 Project beta-code-update-eqiad build #114783: 09FIXED in 1 min 35 sec: https://integration.wikimedia.org/ci/job/beta-code-update-eqiad/114783/
[20:08:18] <paladox>	 twentyafterfour hi
[20:08:53] <paladox>	 Im wondering weather you would be able to update the phabricator extension with gerrit in it
[20:09:01] <paladox>	 to support stripping out
[20:09:11] <paladox>	 refs/meta/config the the url to only refs
[20:09:36] <paladox>	 please like we do with refs/heads/ but with that one we leave the branch, but with refs/meta/config we want refs/
[20:10:00] <paladox>	 since we can then rely on the commitid to get to view the files
[20:10:03] <paladox>	 please
[20:11:20] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1006 is OK: OK: Less than 1.00% above the threshold [0.0]
[20:14:29] <paladox>	 ostriches i have another fix for linking to comments
[20:14:37] <paladox>	 I read your comment
[20:14:54] <paladox>	 and i have amended to add the T to link and have the T outside the expression
[20:15:16] <bd808>	 thcipriani: do you remember what magic is needed to accept a host key so that scap3 will stop telling me "Host key verification failed."? I tried accepting the key as the mwdeploy user but that doesn't seem to fix it
[20:15:34] <bd808>	 This is in a random Labs project
[20:15:42] <thcipriani>	 ah, you have to accept as your user
[20:15:50] <thcipriani>	 or put it in /etc/ssh/...something...
[20:15:52] <bd808>	 oh. easy enough
[20:16:23] <bd808>	 cool that gets me closer
[20:16:43] <bd808>	 now I need to get keyholder armed
[20:16:56] <thcipriani>	 oh boy.
[20:17:08] <bd808>	 yeah, happy fun times
[20:17:49] <thcipriani>	 heh, so fwiw, scap::target *should* make this somewhat easy. As long as the keys are in puppet's secrets
[20:18:34] <bd808>	 that needs to be applied on each host that scap3 will talk to correct?
[20:19:08] <thcipriani>	 yup
[20:19:27] <thcipriani>	 yeah, so that'll get the public keys in place, anyway
[20:20:17] <bd808>	 this is all getting close to working :)
[20:20:19] <thcipriani>	 scap::server will get all the keyholder::agents setup using hiera info
[20:21:46] <ostriches>	 paladox: That's not enough. It still needs the $1 to just be the 1234 in T1234#567.
[20:21:47] <stashbot>	 T1234: Restrict Bugzilla access to read-only - https://phabricator.wikimedia.org/T1234
[20:21:56] <ostriches>	 Moving the T just turns it into 1234#567
[20:22:01] <ostriches>	 (better, but still not right)
[20:22:23] <paladox>	 ostriches i mean
[20:22:30] <paladox>	 i also added the T to link
[20:22:37] <paladox>	 as it was before
[20:22:49] <ostriches>	 That's still not enough.
[20:22:49] <paladox>	 Im testing it out
[20:22:52] <ostriches>	 Like I just said.
[20:23:29] <paladox>	 Im testing
[20:23:30] <paladox>	     match =  "\\bT(\\d+)(#\\d+2)?\\b"
[20:23:30] <paladox>	     link = https://phabricator.wikimedia.org/T$1$2
[20:23:32] <paladox>	 ostriches ^^
[20:23:48] <ostriches>	 That's what we tried first, right? I thought that didn't work on just a plain T123
[20:23:49] <stashbot>	 T123: Turn on "diffusion.allow-http-auth" - https://phabricator.wikimedia.org/T123
[20:23:53] <paladox>	 No
[20:23:55] <ostriches>	 stashbot go away
[20:24:01] <paladox>	 ostriches we did
[20:24:29] <paladox>	     match =  "\\b(T\\d+(#\\d+))?\\b"
[20:24:29] <paladox>	     link = https://phabricator.wikimedia.org/$1
[20:24:32] <paladox>	 ostriches ^^
[20:24:50] <paladox>	 Totaly different to how im going to do it now
[20:24:58] <ostriches>	 Yes, but before we applied that, you tried with $1$2 locally and it didn't work right?
[20:25:07] <paladox>	 It worked
[20:25:08] <paladox>	 locally
[20:25:35] <paladox>	 I was talking about phab links breaking not that the patch caused that but you know when you enter 
[20:25:42] <paladox>	 phabricator.wikimedia.org/T1 
[20:25:45] <paladox>	 it makes it like
[20:26:04] <paladox>	 T1" target="_blank">https://phabricator.wikimedia.org/T1
[20:26:09] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[20:26:09] <paladox>	 ostriches ^^
[20:26:43] <paladox>	 Yay works
[20:26:44] <paladox>	 http://gerrit-test.wmflabs.org/gerrit/#/c/16/5
[20:26:46] <paladox>	 ostriches ^^
[20:27:17] <paladox>	 No 500 errors either
[20:27:19] <paladox>	 yay
[20:27:19] <ostriches>	 Meh, I'm not applying that any doing another restart today.
[20:27:24] <ostriches>	 I'm kinda over gerrit now
[20:27:26] <ostriches>	 #badmood
[20:27:31] <paladox>	 oh
[20:27:38] <paladox>	 Twitter #
[20:27:44] <ostriches>	 Also: the 500s were showing because of its-phabricator stacktraces, which I don't think you have installed ;-)
[20:27:54] <paladox>	 Oh i installed it
[20:28:08] <ostriches>	 Oh you do. Did it post to Phab?
[20:28:16] <paladox>	 http://gerrit-test.wmflabs.org/gerrit/#/admin/projects/integration/zuul
[20:28:19] <paladox>	 No
[20:28:46] <paladox>	 ostriches yeh i knew it was because of the gerritbot, dosent the bot need to be updated to support T1#1
[20:28:46] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[20:29:40] <ostriches>	 Do it again stashbot, I dare you.
[20:29:48] <paladox>	 LOL
[20:29:50] <paladox>	 T1
[20:29:50] <stashbot>	 T1: Get puppet runs into logstash - https://phabricator.wikimedia.org/T1
[20:30:20] <paladox>	 It did it again ^^
[20:30:21] <paladox>	 LOL
[20:31:19] <paladox>	  You have been kicked from #wikimedia-releng by ostriches (not funny)
[20:32:23] <paladox>	 ostriches but you said this is the revert msg
[20:32:26] <paladox>	 I forgot that $1 has to be a reference to a task number and only a
[20:32:26] <paladox>	 task number. Including "T" or the fragment in $1 breaks things.
[20:32:32] <paladox>	 Which i fixed in the test now
[20:35:13] <paladox>	 ostriches https://gerrit.wikimedia.org/r/#/c/301673/
[20:38:50] <bd808>	 ostriches: do you want me to pull stashbot permanently? That will keep your !logs from ending up in https://tools.wmflabs.org/sal/releng
[20:39:04] <hashar_>	 no please
[20:39:06] <hashar_>	 :D
[20:39:17] <ostriches>	 No, I just want it to please stop mentioning dumb links just because I say T1234.
[20:39:21] <ostriches>	 If I want a link I'll paste a link
[20:39:34] <hashar_>	 maybe the Txxx can be disabled on a per channel basis?
[20:39:39] <ostriches>	 ^ Ideal
[20:39:50] <hashar_>	 cat .stashbot
[20:39:54] <hashar_>	 pfff
[20:39:56] <bd808>	 patches welcome -- https://github.com/bd808/tools-stashbot
[20:39:59] <hashar_>	 IRC is so outdated
[20:40:26] * ostriches forks
[20:41:08] <bd808>	 it would go around here -- https://github.com/bd808/tools-stashbot/blob/master/stashbot/bot.py#L119-L120
[20:41:09] * hashar_ officially fork to Gerrit and setup replication to wikimedia/stashbot  *evil*
[20:41:23] <ostriches>	 forking gerrit? you must really hate yourself :p
[20:41:25] <mutante>	 this dialog will soon end in a suggestion to move bots to a -feed or -bots channel.  it's the normal cycle
[20:41:44] <hashar_>	 ahah
[20:41:48] <hashar_>	 mutante: that is a core value
[20:42:28] <paladox>	 ostriches does https://gerrit.wikimedia.org/r/#/c/301673/2/modules/gerrit/templates/gerrit.config.erb that look better
[20:42:41] <paladox>	 i followed what you said here https://gerrit.wikimedia.org/r/#/c/301665/2
[20:42:42] <paladox>	 ?
[20:43:15] <ostriches>	 I said I'm not looking at gerrit anymore today.
[20:43:23] <paladox>	 Oh yes sorry
[20:45:09] <mutante>	 be back later too
[21:14:02] <paladox>	 ostriches hi, do you know how we setup the its-phabricator plugin for gerritbot
[21:14:13] <ostriches>	 The configuration is all public.
[21:14:21] <paladox>	 Oh
[21:14:29] <paladox>	 I did the arc install-certificate
[21:14:33] <ostriches>	 The only private piece is the certificate, but you'll have to generate/use your own anyway :)
[21:14:38] <ostriches>	 It goes in secure.config
[21:14:40] <paladox>	 Yeh i did that
[21:15:38] <paladox>	 I have it saved in
[21:15:47] <paladox>	 ~/.arcrc
[21:15:49] <paladox>	 root account
[21:16:26] <paladox>	 ostriches ^^
[21:17:15] <paladox>	 Maybe its because i did this
[21:17:24] <ostriches>	 It doesn't need to be in .arcrc
[21:17:27] <ostriches>	 It goes in secure.config.
[21:17:34] <bd808>	 thcipriani: I think I actually got it to work! -- https://wikitech.wikimedia.org/wiki/User:BryanDavis/Scap3_in_a_Labs_project#Syncing_with_the_cluster
[21:17:42] <paladox>	 [plugin "its-phabricator"]
[21:17:42] <paladox>	 	enabled = true
[21:17:43] <paladox>	 	branch = ^*
[21:17:43] <thcipriani>	 \o/
[21:17:49] <paladox>	 Oh
[21:17:59] <paladox>	 I saved it also in that secure file
[21:18:00] <paladox>	 too
[21:18:18] <paladox>	 ostriches ^^ But maybe its because of ^^
[21:21:21] <thcipriani>	 bd808: sweet, looks like it! scap deploy-log -v may give you a bit deeper info, the info level messages still leave a little to be desired. Also, since we try to not blow up things that have already been deployed, if it finds a /srv/deployment/striker/deploy-cache/revs/[sha1] directory, it won't try to "re" check it out (which is what the --force message means there).
[21:23:38] <bd808>	 thcipriani: a little bug here -- https://phabricator.wikimedia.org/P3596 -- "TypeError: not enough arguments for format string" from a failed check
[21:24:24] <bd808>	 my promote check is totally expected to fail right now
[21:24:36] * bd808 works on that next
[21:25:21] <thcipriani>	 bd808: ah, blerg. I think I've seen something like that before. IIRC the error message shows up in deploy-log. I thought I fixed that message though...maybe it happens somewhere else too. :\
[21:31:47] <shinken-wm>	 PROBLEM - Puppet run on deployment-aqs01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:33:13] <shinken-wm>	 RECOVERY - SSH on deployment-changeprop is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u3 (protocol 2.0)
[21:33:23] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1002 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:33:31] <thcipriani>	 hmm, guess I fixed a different problem that presents in a similar way: https://phabricator.wikimedia.org/D83
[21:33:33] * thcipriani digs
[21:34:19] <paladox>	 ostriches i get 500 error when setting the its-phabricator plugin to true
[21:34:31] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Packaging: debian-glue have the pbuilderrc override cowbuilder parameters - https://phabricator.wikimedia.org/T141538#2503478 (10hashar) So I believe jenkins-debian-glue load the configfile last on purpose so people can override the assumption it is making such as the...
[21:34:38] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1012 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:34:48] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-sentry2 is CRITICAL: CRITICAL: deployment-prep.deployment-sentry2.diskspace._var.byte_percentfree (<100.00%)
[21:35:33] <bd808>	 tgr: ^ disk full on deployment-sentry2 
[21:35:44] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-precise-1011 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [0.0]
[21:36:13] <tgr>	 bd808: mind if I drop that box?
[21:36:32] <tgr>	 IIRC that's the old one you set up when there was no puppet role for it
[21:36:34] <bd808>	 I'm not using it for anything
[21:36:38] <bd808>	 ah right
[21:36:43] <bd808>	 sure kill it dead
[21:40:05] <tgr>	 "Free space < 100%" is probably not the best way to phrase that warning, FWIW
[21:40:50] <bd808>	 icinga/shinken checks are mostly gibberish
[21:41:30] <ostriches>	 Bahahahahahahaha.
[21:41:38] <ostriches>	 if config.enable_identica:
[21:41:46] <ostriches>	 😂
[21:41:50] <ostriches>	 Oh man, I laughd
[21:41:57] <shinken-wm>	 PROBLEM - Host deployment-sentry2 is DOWN: CRITICAL - Host Unreachable (10.68.17.204)
[21:42:16] <paladox>	 lol
[21:42:41] <paladox>	 ostriches strange that it works for wikimedia but doing it on my test install 
[21:42:44] <paladox>	 it gets 500 error
[21:42:45] <paladox>	 lol
[21:42:48] <shinken-wm>	 PROBLEM - Puppet run on deployment-changeprop is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[21:43:01] <tgr>	 do I need to change some icinga config or will it eventually notice that the box is gone?
[21:44:10] * greg-g pours one out for identi.ca, for the 112th time
[21:44:55] <ostriches>	 bd808: You've got 2 pull requests :)
[21:45:04] <bd808>	 ostriches: sweet
[21:46:28] <hashar_>	 !log xintegration: change sudo policy for jenkins-deploy to help on T141538 : env_keep+=WORKSPACE
[21:46:31] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[21:48:42] <paladox>	 ostriches i used the plugin you used on wikimedia
[21:48:44] <paladox>	 and seems to
[21:48:48] <paladox>	 resolve the 500 error
[21:48:57] <paladox>	 but still dosent seem to connect to phab-01
[21:50:35] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.16 deployment blockers - https://phabricator.wikimedia.org/T141551#2503539 (10greg) a:03hashar And noted in https://etherpad.wikimedia.org/p/RelEngWeekly ;)
[21:56:19] <paladox>	 ostriches is the certificate, what you generate by doing arc install-certificate
[21:56:24] <paladox>	 going to /api/
[21:56:33] <paladox>	 ?
[22:06:50] <MaxSem>	 greg-g, I have a ton of labs-only changes - wonder if I can deploy it outside of SWAT
[22:07:02] <greg-g>	 MaxSem: sure
[22:07:09] <MaxSem>	 thx
[22:17:21] <wikibugs>	 10Continuous-Integration-Config, 10Incident-20160126-WikimediaDomainRedirection, 06Operations, 07Regression, 05Wikimedia-Incident: operations-apache-config-lint replacement doesn't check syntax - https://phabricator.wikimedia.org/T114801#2503652 (10greg)
[22:17:52] <wikibugs>	 10Continuous-Integration-Config, 10Incident-20150312-whitespace, 05Wikimedia-Incident: add a check for whitespace before leading <?php - https://phabricator.wikimedia.org/T92531#2503669 (10greg)
[22:18:01] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Staging, 10Incident-20160126-WikimediaDomainRedirection, 05Wikimedia-Incident: Rework beta apache config - https://phabricator.wikimedia.org/T1256#2503674 (10greg)
[22:18:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Incident-20160126-WikimediaDomainRedirection, 05Wikimedia-Incident: Write and implement tests for Wikimedia's Apache configuration (redirects.conf, etc.) - https://phabricator.wikimedia.org/T45266#2503673 (10greg)
[22:28:47] <grrrit-wm>	 (03PS1) 10Hashar: debian-glue: override Wikimedia BUILDRESULT [integration/config] - 10https://gerrit.wikimedia.org/r/301714 (https://phabricator.wikimedia.org/T141538) 
[22:31:40] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Packaging: debian-glue have the pbuilderrc override cowbuilder parameters - https://phabricator.wikimedia.org/T141538#2503747 (10hashar) I have tried /mnt/home/jenkins-deploy/.pbuilderrc , it does not override the env variable.  The dirty fix is  https://gerrit.wikime...
[22:55:20] <grrrit-wm>	 (03PS1) 10Thcipriani: Add job apps-android-wikipedia-periodic-test-new [integration/config] - 10https://gerrit.wikimedia.org/r/301720 (https://phabricator.wikimedia.org/T139137) 
[22:56:16] <paladox>	 Im getting this error
[22:56:17] <paladox>	 Caused by: com.googlesource.gerrit.plugins.its.phabricator.conduit.ConduitErrorException: Method 'conduit.connect' gave: ERR-INVALID-CERTIFICATE, Your authentication certificate for this server is invalid.
[22:56:17] <paladox>	         at com.googlesource.gerrit.plugins.its.phabricator.conduit.ConduitConnection.call(ConduitConnection.java:126)
[22:56:17] <paladox>	         at com.googlesource.gerrit.plugins.its.phabricator.conduit.Conduit.conduitConnect(Conduit.java:152)
[22:56:18] <paladox>	         at com.googlesource.gerrit.plugins.its.phabricator.conduit.Conduit.fillInSession(Conduit.java:95)
[22:56:21] <paladox>	         at com.googlesource.gerrit.plugins.its.phabricator.conduit.Conduit.maniphestUpdate(Conduit.java:177)
[22:56:24] <paladox>	         at com.googlesource.gerrit.plugins.its.phabricator.PhabricatorItsFacade.addComment(PhabricatorItsFacade.java:59)
[22:56:27] <paladox>	 ostriches ^^
[22:56:55] <paladox>	 twentyafterfour ^^
[23:01:04] <twentyafterfour>	 paladox: where do you see that error?
[23:01:14] <paladox>	 In the its-phabricator plugin
[23:01:22] <paladox>	 Im trying it out in gerrit-test
[23:01:27] <paladox>	 and trying to connect to phab-01
[23:01:50] <paladox>	 twentyafterfour ^^
[23:02:09] <twentyafterfour>	 oh, well you need to set up .arcconfig I guess? I have no idea I've never touched its-phabricator
[23:02:14] <paladox>	 oh
[23:02:24] <paladox>	 What do i put in .arcconfig
[23:02:56] <paladox>	 twentyafterfour ^^
[23:13:59] <twentyafterfour>	 paladox: I'm trying to see how its-phabricator accesses conduit but I don't see any docs
[23:14:14] <paladox>	 Ok
[23:14:31] <paladox>	 twentyafterfour https://gerrit.googlesource.com/plugins/its-phabricator/+/master/src/main/resources/Documentation/config-connectivity.md
[23:15:39] <paladox>	 [its-phabricator]
[23:15:39] <paladox>	     url = https://phabricator.wikimedia.org/
[23:15:39] <paladox>	     username = gerritbot
[23:15:39] <paladox>	     commentOnChangeAbandoned = false
[23:15:39] <paladox>	     commentOnChangeMerged = false
[23:15:39] <paladox>	     commentOnChangeRestored = false
[23:15:41] <paladox>	     commentOnChangeCreated = false
[23:15:43] <paladox>	     commentOnCommentAdded = false
[23:15:45] <paladox>	     commentOnPatchSetCreated = false
[23:15:47] <paladox>	     commentOnRefUpdatedGitWeb = false
[23:16:37] <twentyafterfour>	 https://gerrit.googlesource.com/plugins/its-phabricator/+/master/src/main/resources/Documentation/config-connectivity.md
[23:16:50] <twentyafterfour>	   certificate=CERTIFICATE_FOR_ABOVE_USERNAME
[23:16:51] <paladox>	 Yep
[23:16:55] <paladox>	 I followed that
[23:16:56] <paladox>	 i did
[23:17:12] <twentyafterfour>	 you need a certificate from phabricator for the gerritbot user
[23:17:14] <paladox>	 arc install-certificator https://phab-01.wmflabs.org/
[23:17:23] <paladox>	 arc install-certificate https://phab-01.wmflabs.org/
[23:17:44] <paladox>	 Oh im using my own account
[23:18:04] <twentyafterfour>	 there is no gerritbot user on phab-01
[23:18:12] <paladox>	 Nope
[23:18:15] <paladox>	 i am using paladox
[23:18:18] <paladox>	 as the account
[23:18:22] <paladox>	 since i am testing
[23:18:27] <twentyafterfour>	 so you need to specify the username = paladox in the its-phabricator config
[23:18:32] <paladox>	 Yep
[23:18:34] <paladox>	 i have
[23:18:37] <paladox>	 gerrit.conf
[23:18:54] <paladox>	 gerrit.config
[23:19:18] <twentyafterfour>	 and you have certificate = in there too?
[23:19:21] <paladox>	 Yep
[23:19:41] <paladox>	 I saved the certificate in secure.config
[23:19:52] <twentyafterfour>	 then I don't know what's up
[23:19:56] <paladox>	 Oh
[23:20:08] <wikibugs>	 06Release-Engineering-Team, 06Operations, 15User-greg, 05Wikimedia-Incident: Institute quarterly(?) review of incident reports and follow-up - https://phabricator.wikimedia.org/T141287#2503945 (10greg)
[23:33:33] <paladox>	 I will try tomarror again.
[23:33:39] <paladox>	 since it is 00:33am here.
[23:34:30] <twentyafterfour>	 ok
[23:37:14] <TimStarling>	 I would like to have a comment from releng on https://phabricator.wikimedia.org/T141586
[23:40:32] * greg-g starts typing
[23:46:42] <greg-g>	 replied
[23:50:52] <TimStarling>	 thanks