[00:04:03] <Krenair>	 how are there suddenly 5 hosts with broken puppet in deployment-prep?
[00:05:00] <mutante>	 is it suddenly or that the checks have been turned on again
[00:28:38] <tom29739>	 mutante: I recall someone turning on checks earlier today
[00:29:32] <mutante>	 yea, shinken checks re-enabled is what i was thinking of
[00:29:53] <mutante>	 Krenair: ^ 
[00:32:26] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<11.11%)
[00:45:51] <grrrit-wm>	 (03CR) 10Jforrester: "Any movement on this? It'd be a lovely way to reduce the differentiation for old-hands vs. newbies." [integration/config] - 10https://gerrit.wikimedia.org/r/291301 (owner: 10Legoktm)
[01:07:14] <wikibugs>	 10Browser-Tests-Infrastructure, 07Easy: Use rspec-expectations "expect" syntax instead of "should" syntax - https://phabricator.wikimedia.org/T68369#2478316 (10Danny_B)
[01:25:37] <shinken-wm>	 RECOVERY - SSH on integration-aptly01 is OK: SSH OK - OpenSSH_6.7p1 Debian-5+deb8u2 (protocol 2.0)
[01:30:48] <shinken-wm>	 PROBLEM - Puppet staleness on integration-aptly01 is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [43200.0]
[01:31:02] <legoktm>	 ...
[01:31:06] <legoktm>	 ^ i rebooted it
[01:31:10] <shinken-wm>	 PROBLEM - Puppet run on integration-aptly01 is CRITICAL: CRITICAL: 88.89% of data above the critical threshold [0.0]
[01:33:21] <legoktm>	 puppet run looks fine
[01:35:48] <shinken-wm>	 RECOVERY - Puppet staleness on integration-aptly01 is OK: OK: Less than 1.00% above the threshold [3600.0]
[01:38:57] <wikibugs>	 06Release-Engineering-Team, 06Operations, 10Wikimedia-Apache-configuration, 07HHVM: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#2478415 (10Danny_B)
[01:39:20] <wikibugs>	 06Release-Engineering-Team, 06Operations, 10Wikimedia-Apache-configuration, 07HHVM: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#760100 (10Danny_B)
[01:41:09] <shinken-wm>	 RECOVERY - Puppet run on integration-aptly01 is OK: OK: Less than 1.00% above the threshold [0.0]
[01:49:41] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: MediaWiki extensions with failing phpunit tests (tracking) - https://phabricator.wikimedia.org/T67874#2478486 (10Danny_B)
[01:51:50] <grrrit-wm>	 (03PS1) 10Legoktm: GeoCrumbs depends on CustomData [integration/config] - 10https://gerrit.wikimedia.org/r/299930 
[01:52:08] <grrrit-wm>	 (03CR) 10Legoktm: [C: 032] GeoCrumbs depends on CustomData [integration/config] - 10https://gerrit.wikimedia.org/r/299930 (owner: 10Legoktm)
[01:52:47] <grrrit-wm>	 (03Merged) 10jenkins-bot: GeoCrumbs depends on CustomData [integration/config] - 10https://gerrit.wikimedia.org/r/299930 (owner: 10Legoktm)
[01:53:05] <legoktm>	 !log deploying https://gerrit.wikimedia.org/r/299930
[01:53:11] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[02:10:48] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Tracking: Jenkins: Automated testing for various Fundraising-tech components (tracking) - https://phabricator.wikimedia.org/T50269#2478520 (10Danny_B)
[04:06:35] <wmf-insecte>	 Yippee, build fixed!
[04:06:36] <wmf-insecte>	 Project selenium-MultimediaViewer » safari,beta,OS X 10.9,contintLabsSlave && UbuntuTrusty build #79: 09FIXED in 10 min: https://integration.wikimedia.org/ci/job/selenium-MultimediaViewer/BROWSER=safari,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=OS%20X%2010.9,label=contintLabsSlave%20&&%20UbuntuTrusty/79/
[04:50:54] <grrrit-wm>	 (03PS1) 10Ejegg: Quit testing DonationInterface against REL1_25 [integration/config] - 10https://gerrit.wikimedia.org/r/299940 
[04:55:20] <grrrit-wm>	 (03PS2) 10Ejegg: Run DonationInterface LintYaml job via composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight)
[04:55:42] <grrrit-wm>	 (03CR) 10Ejegg: "PS2: PS1 was rebased" [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight)
[04:57:29] <grrrit-wm>	 (03CR) 10Ejegg: [C: 031] "More peace of mind!" [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight)
[05:14:50] <grrrit-wm>	 (03CR) 10Awight: [C: 04-1] "Sadly, I think we still need some sort of php53 lint job to keep us from adopting short array syntax ;-)" [integration/config] - 10https://gerrit.wikimedia.org/r/299940 (owner: 10Ejegg)
[05:44:59] <wmf-insecte>	 Project mediawiki-core-code-coverage build #2146: 04STILL FAILING in 2 hr 44 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2146/
[06:06:18] <grrrit-wm>	 (03CR) 10Madhuvishy: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/290597 (https://phabricator.wikimedia.org/T132182) (owner: 10Madhuvishy)
[06:06:30] <grrrit-wm>	 (03PS4) 10Madhuvishy: Add maven release job template and analytics-refinery-release project [integration/config] - 10https://gerrit.wikimedia.org/r/290597 (https://phabricator.wikimedia.org/T132182) 
[06:07:09] <MaxSem>	 https://gerrit.wikimedia.org/r/#/c/299948/ fixes coverage ^^^
[06:07:13] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] Add maven release job template and analytics-refinery-release project [integration/config] - 10https://gerrit.wikimedia.org/r/290597 (https://phabricator.wikimedia.org/T132182) (owner: 10Madhuvishy)
[06:11:12] <madhuvishy>	 hmmm our jjb clone isn't updated yet - can we update it with the latest upstream commits?
[07:38:16] <grrrit-wm>	 (03CR) 10Paladox: "Not all tests have been migrated yet, they are stalled due to labs not having the capacity at the moment." [integration/config] - 10https://gerrit.wikimedia.org/r/291301 (owner: 10Legoktm)
[08:05:33] <paladox>	 hashar hi, would it be something like https://github.com/openstack-infra/zuul/commit/f95e7232acacab4560390e66af3747e4b5984112
[08:06:18] <hashar>	 paladox: "it" ?
[08:06:36] <paladox>	 The thing we talked with ostriches yesturday
[08:06:43] <paladox>	 with git erroring out
[08:06:48] <paladox>	 due to it not finding head
[08:06:49] <hashar>	 unrelated 
[08:06:52] <paladox>	 Oh
[08:07:17] <paladox>	 He filled a bug report https://storyboard.openstack.org/#!/story/2000678
[08:07:18] <paladox>	 :)
[08:07:24] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-jobrunner01 is CRITICAL: CRITICAL: deployment-prep.deployment-jobrunner01.diskspace.root.byte_percentfree (<40.00%)
[08:07:39] <hashar>	 look at our zuul/layout.yaml file , there is a "publish" pipeline which reacts to the Gerrit event "ref-updated"
[08:07:47] <paladox>	 Oh
[08:08:00] <hashar>	 that is whenever a branch head / tag  is created/deleted/modified
[08:08:09] <hashar>	 and we only keep changes to the tags
[08:08:29] <paladox>	 Oh
[08:08:31] <hashar>	 the ignore-deletes  option that is being proposed, is to have Zuul to always ignore deletions
[08:08:50] <paladox>	 Seems this https://github.com/openstack-infra/zuul/commit/bc58ea34125f11eb353abc3e5b96ac1efad06141 is mostly related
[08:08:59] <hashar>	 else, you still have to trigger the job and have the job to abort early when it detects it is a deletion
[08:09:03] <paladox>	 Maybe even the fix for us.
[08:09:08] <paladox>	 Oh
[08:09:23] <paladox>	 Wrong link
[08:09:34] <paladox>	 This one
[08:09:35] <paladox>	 https://github.com/openstack-infra/zuul/commit/6c54b531a36644d53ae054d3fdf846fb2e538677
[08:10:02] <paladox>	 It swaps out origin.update()  for origin.fetch(tags=True) 
[08:10:15] <paladox>	 Which will get us current branch information including tags.
[08:11:02] <hashar>	 paladox:  http://docs.openstack.org/infra/zuul/launchers.html#reference-updated-parameters  is a good read.  It shows the ZUUL_* parameters that are sent for ref update/creation/deletion.  Namely a deleted ref has  ZUUL_NEWREV=00000000000000000000000000000000
[08:11:07] <paladox>	 Oh
[08:11:16] <paladox>	 Thanks
[08:11:46] <hashar>	 so if you want to skip branch deletion you do something like  [[ "$ZUUL_NEWREV" == "00000000000000000000000000000000" ]] && exit 0
[08:11:52] <paladox>	 Oh
[08:11:59] <hashar>	 which is usually what you wanna do
[08:12:04] <paladox>	 Yep
[08:12:09] <hashar>	 it make little sense to build a job when a ref is deleted
[08:12:20] <paladox>	 Oh yep
[08:12:25] <hashar>	 so that commit you linked to let one to always skip it :D
[08:12:47] <paladox>	 Oh
[08:12:59] <paladox>	 So that would fix it for us or would that cause more problems then good
[08:13:00] <paladox>	 ?
[08:14:35] * paladox my broadband has been blocking phabricator today, using personnel
[08:14:36] <hashar>	 the issue Chad has reported is different
[08:14:36] <paladox>	 point chaud :). 
[08:14:39] <paladox>	 Oh
[08:14:52] <hashar>	 that is in the zuul-cloner  apparently not properly handling the branch deletion
[08:14:58] <paladox>	 Oh
[08:15:10] <paladox>	 But woulden git fetch get the current branch information
[08:15:38] <hashar>	 no clue :-}
[08:15:42] <paladox>	 Oh
[08:15:46] <hashar>	 have to reproduce first
[08:15:49] <paladox>	 Ok
[08:16:00] <hashar>	 well really, need an easy way to reproduce
[08:16:06] <paladox>	 Oh
[08:16:33] <paladox>	 Also thanks for updating the jessie package for zuul
[08:16:34] <paladox>	 :)
[08:16:36] <paladox>	 hashar ^^
[08:17:51] <hashar>	 !log deployment-fluorine : deleting a puppet lock file /var/lib/puppet/state/agent_catalog_run.lock   (created at 2016-07-18 19:58:46 UTC)
[08:17:54] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[08:18:19] <hashar>	 paladox: will get the Jessie package setup on a labs instance and have it pointing to gerrit-new
[08:18:24] <hashar>	 and see what happens :D
[08:18:28] <paladox>	 hashar yay
[08:18:29] <paladox>	 :)
[08:19:21] <paladox>	 It seems on jenkins wikimedia on ios when zoomed out it blocks the jenkins logo
[08:19:28] <paladox>	 with a white block over it
[08:19:31] <paladox>	 which is strange
[08:19:38] <paladox>	 but may be a bug in ios 10.
[08:20:41] <paladox>	 hashar Also i can now load integration/config layout.yaml changes on my iphone since they included the fix i submitted early this year and was fixed :)
[08:20:50] <paladox>	 I didnt submit the fix
[08:20:59] <paladox>	 i submited the bug report and someone else fixed it.
[08:21:26] <hashar>	 \O/
[08:22:03] <paladox>	 Yep
[08:22:30] <paladox>	 But it worked on my ipad pro before due to it having extra ram but now i can do it on my phone instantly.
[08:22:41] <paladox>	 gerrit 2.12 includes improvements to loading diffs too.
[08:24:16] * tom29739 boos iPhones
[08:27:45] <wikibugs>	 06Release-Engineering-Team, 06Operations, 10Wikimedia-Apache-configuration, 07HHVM: Make it possible to quickly and programmatically pool and depool application servers - https://phabricator.wikimedia.org/T73212#2479025 (10hashar)
[08:29:54] <shinken-wm>	 RECOVERY - Puppet staleness on deployment-fluorine is OK: OK: Less than 1.00% above the threshold [3600.0]
[08:32:06] <wikibugs>	 06Release-Engineering-Team, 10LDAP-Access-Requests, 06Operations, 10Ops-Access-Requests, and 2 others: Determine a core set or a checklist of permissions for deployment purpose - https://phabricator.wikimedia.org/T140270#2479027 (10hashar) related: @Neil_P._Quinn_WMF has overhauled the wikitech page [[ htt...
[08:33:37] <wikibugs>	 06Release-Engineering-Team (Deployment-Blockers), 05Release: MW-1.28.0-wmf.11 deployment blockers - https://phabricator.wikimedia.org/T139212#2479033 (10mobrovac)
[09:57:19] * paladox wow topping up for unlimited data from three they give me an extra 150mb for free. Not sure why i need an extra 150mb since my limit is 1tb.
[09:57:21] <paladox>	 hashar ^^
[10:08:33] <paladox>	 hashar Is it that we have to point zuul at lead to pickup the changeds
[10:08:35] <paladox>	 changes
[10:08:42] <paladox>	 and add the ssh key to jenkins.
[10:15:02] <zeljkof>	 gehel: can you help me debug my cirrus-browser-bot.eqiad.wmflabs ssh problems now?
[10:15:57] <hashar>	 !log beta: removing puppet cherry pick of https://gerrit.wikimedia.org/r/#/c/258979/ "mediawiki: add conftool-specifc credentials and scripts"  abandonned/superseeded and caused a conflict
[10:16:00] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[10:16:14] <gehel>	 zeljkof: sure! Can you pastebin the log of your SSH connect? With "-v -v" to add some more info?
[10:17:06] <zeljkof>	 gehel: sure, just a minute
[10:19:29] <zeljkof>	 gehel: https://phabricator.wikimedia.org/P3516
[10:19:40] <zeljkof>	 is that ok? ^
[10:20:19] <paladox>	 hashar what do i put as the server
[10:20:27] <paladox>	 in zuul to test with gerrit-new
[10:20:32] <paladox>	 please
[10:20:36] <hashar>	 dont
[10:20:43] <paladox>	 Oh ok
[10:21:10] <hashar>	 you already have a Gerrit so there is imho no need to connect to the future prod instance gerrit-new :}
[10:21:18] <paladox>	 Oh
[10:21:59] <paladox>	 yep
[10:22:45] <paladox>	 hashar i thought we are testing the zuul jessie package.
[10:23:22] <gehel>	 zeljkof: yep, looks good, having a look...
[10:24:23] <paladox>	 :)
[10:24:56] <gehel>	 zeljkof: could you also send me your ssh config?
[10:25:20] <zeljkof>	 gehel: sure, the problem is probably there
[10:25:40] <gehel>	 zeljkof: do you have issue connecting to other labs hosts?
[10:26:15] <hashar>	 !log beta: rebased puppetmaster git repo. "Parsoid: Move to service::node"  has weird conflict https://gerrit.wikimedia.org/r/#/c/298436/
[10:26:18] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[10:26:45] <zeljkof>	 gehel: I don't ssh to labs at all, I did it a few times a while back, and probably have outdated ssh config
[10:27:34] <shinken-wm>	 PROBLEM - Puppet run on deployment-mediawiki02 is CRITICAL: CRITICAL: 66.67% of data above the critical threshold [0.0]
[10:27:44] <zeljkof>	 gehel: a stupid question, but there is nothing secret in ssh config file, right? I can make it public?
[10:28:19] <hashar>	 !log beta dropped salt-key on deployment-salt02 for the three instances: deployment-upload.deployment-prep.eqiad.wmflabs , deployment-logstash3.deployment-prep.eqiad.wmflabs and deployment-ores-web.deployment-prep.eqiad.wmflabs
[10:28:21] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[10:28:25] <gehel>	 zeljkof: there should not be anything secret, but have a look and use your best judgement
[10:29:12] <zeljkof>	 gehel: it's all greek to me :) looking and will send you the link in a minute if nothing looks strange
[10:29:37] <gehel>	 zeljkof: yeah, I don't spead Greek either... :P
[10:31:39] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: On beta nutcracker::verbosity default to 5 while prod uses 4, fills disk with a lot of log spam - https://phabricator.wikimedia.org/T136078#2479380 (10hashar) 05stalled>03Resolved a:03hashar Fixed via https://gerrit.wikimedia.org/r/299146 ``` -DAEMON_OPT...
[10:33:53] <zeljkof>	 gehel: here it is https://github.com/zeljkofilipin/dotfiles/blob/master/.ssh/config
[10:34:17] <zeljkof>	 it probably needs a lot of cleanup, I remember hashar helped me set it up a year or two ago
[10:34:40] <zeljkof>	 probably a lot of things changed since (and I did not use it much since I have set it up)
[10:35:35] <wikibugs>	 10Beta-Cluster-Infrastructure, 13Patch-For-Review: On beta nutcracker::verbosity default to 5 while prod uses 4, fills disk with a lot of log spam - https://phabricator.wikimedia.org/T136078#2479400 (10hashar) All beta cluster nutcracker demons now have verbose=4 ``` # salt --out=text  -v '*' cmd.run 'grep ver...
[10:37:34] <shinken-wm>	 RECOVERY - Puppet run on deployment-mediawiki02 is OK: OK: Less than 1.00% above the threshold [0.0]
[10:37:50] <paladox>	 hashar hi, you know this bit apscheduler 3.1.0 fail with old setuptools, it is resolved in apscheduler 3.2.0 which has been released
[10:38:06] <paladox>	 so i presume we can push it up to 3.2.0 now for precise and jessie
[10:39:10] <gehel>	 zeljkof: I'm far from an SSH expert, so bear with me...
[10:39:42] <gehel>	 zeljkof: That being said, I think that in your ssh config IdentityFile should point to your private key, not to your public key
[10:40:05] <zeljkof>	 gehel: ouch, let me check that
[10:40:12] <hashar>	 paladox: maybe later :-}
[10:40:21] <paladox>	 hashar ok :) thanks
[10:40:24] <paladox>	 :) :)
[10:40:27] <hashar>	 paladox: I dont want to update / upgrade too many modules at once. But will keep that in mind
[10:40:34] <paladox>	 Ok
[10:40:47] <paladox>	 :)
[10:48:32] <zeljkof>	 gehel: looks like you are correct, how did I manage to connect to anything then o.O
[10:48:37] <zeljkof>	 will fix and try again
[10:50:04] <gehel>	 zeljkof: that's strange...
[10:50:49] <gehel>	 zeljkof: so it works? Ping me if it doesn't and I'll try to find something else ... lunch time for now
[10:51:21] <zeljkof>	 gehel: researching still, thanks a lot, will try and let you know
[10:51:35] <gehel>	 zeljkof: good luck!
[11:00:55] <zeljkof>	 gehel: hm, did not help :( I have updated the terminal output, take a look at the diff https://phabricator.wikimedia.org/P3516
[11:01:29] <zeljkof>	 it now points to id_rsa, not id_rsa.pub, but it still asks me for password
[11:01:36] <paladox>	 zeljkof maybe this will help you i set it up like this for me
[11:01:37] <paladox>	 Host git-redirects-01
[11:01:37] <paladox>	 ProxyCommand ssh -a -W %h:%p -A paladox@bastion.wmflabs.org
[11:01:37] <paladox>	 User paladox
[11:02:08] <paladox>	 I use git for windows so that should work for you.
[11:02:23] <zeljkof>	 paladox: thanks, but I am not sure what I should do :)
[11:02:24] <paladox>	 I set it in config file and doint have anything else set in there.
[11:02:39] <zeljkof>	 oh, that is your entire .ssh/config file?
[11:02:48] <gehel>	 zeljkof: that would have been too easy...
[11:02:49] <paladox>	 Well not entire but yes
[11:03:15] <paladox>	 You replace paladox with your username and should be able to ssh into labs.
[11:03:36] <paladox>	 You also replace git-redirects-01 with the instance you want to ssh into
[11:04:01] <zeljkof>	 gehel: I  have my ssh config backed up, so I can change it at will, is there a minimal config file that you think should work?
[11:04:36] <gehel>	 zeljkof: ssh -v -v still gives the same output? I'll have another look after lunch
[11:04:37] <zeljkof>	 should I have a one ssh key for gerrit/github, one for labs, one for production?
[11:04:40] <paladox>	 zeljkof you can use 
[11:04:41] <paladox>	 Host git-redirects-01
[11:04:41] <paladox>	 ProxyCommand ssh -a -W %h:%p -A paladox@bastion.wmflabs.org
[11:04:41] <paladox>	 User paladox
[11:04:50] <paladox>	 Replace git-redirects-01 with instance name
[11:05:01] <paladox>	 replace paladox with your username
[11:05:03] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016), 15User-greg: Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2479471 (10Aklapper) >>! In T137727#2476444, @greg wrote: > @mmodell / @Aklapper: Were we going to do this on the main Wikime...
[11:05:10] <paladox>	 and then run 
[11:05:29] <paladox>	 ssh -i /c/Users/****/.ssh/id_rsa.pub git-redirect-01
[11:05:37] <zeljkof>	 gehel: yes, the same output, the only change (visible in the phab paste diff) is that now ssh config points to private, not public file
[11:05:55] <paladox>	 again replacing git-redirect-01 with your instance name that you put in the config file.
[11:06:06] <paladox>	 zeljkof ^^
[11:06:10] <zeljkof>	 paladox: thanks, will read the docs and try to create a small config file, it should be easier to debug
[11:06:17] <paladox>	 Ok
[11:06:53] <paladox>	 That /c/Users/ is the windows path and will need to be changed to what ever os you use.
[11:06:58] <paladox>	 zeljkof ^^
[11:06:59] <paladox>	 :)
[11:10:08] <paladox>	 That's why phabricator was not working for me today http://www.bbc.co.uk/news/technology-36844712
[11:10:13] <paladox>	 hashar ^^
[11:23:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Wikipedia-Android-App-Backlog, 07Spike: [Dev] [Spike] Stand up a secure production build server - https://phabricator.wikimedia.org/T136662#2343451 (10hashar) We would need to have Jenkins configuration to be puppetized / in a configuration management system which i...
[11:28:18] <zeljkof>	 gehel, paladox, so if I use the recommended .ssh/config from https://wikitech.wikimedia.org/wiki/Help:Access#Accessing_instances_with_ProxyCommand_ssh_option_.28recommended.29
[11:29:58] <zeljkof>	 I am getting this https://phabricator.wikimedia.org/P3516
[11:30:24] <paladox>	 yep
[11:30:27] <paladox>	 that is correct
[11:30:37] <paladox>	 sometimes you hit the password dialog sometimes you doint
[11:30:45] <paladox>	 does it allow you to login.
[11:30:49] <paladox>	 zeljkof ^^
[11:30:51] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Monitoring: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#2479567 (10hashar) 05Open>03stalled @Paladox the problem is determining a metric to detect that processing of jobs is stall.  One can query zuul-server RPC: ``` $ zuul...
[11:31:03] <wikibugs>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 06Release-Engineering-Team: Identify metric (or metrics) that gives a useful indication of user-perceived (Wikimedia developer) service of CI - https://phabricator.wikimedia.org/T139771#2442096 (10hashar)
[11:31:04] <zeljkof>	 paladox: but there should not no password, as far as I know cc gehel 
[11:31:05] <wikibugs>	 10Continuous-Integration-Infrastructure, 10Monitoring: Alert when Zuul/Gearman queue is stalled - https://phabricator.wikimedia.org/T70113#2479571 (10hashar)
[11:31:21] <zeljkof>	 anyway, I will wait for gehel to come back
[11:31:22] <paladox>	 zeljkof actually yes sometimes there is :)
[11:31:30] <zeljkof>	 ok, time for lunch
[11:31:33] <paladox>	 Ok
[11:31:34] * zeljkof is out of lunch
[11:31:35] <paladox>	 :)
[11:32:30] <paladox>	 hashar :)
[11:33:07] <wikibugs>	 10Continuous-Integration-Infrastructure, 05Continuous-Integration-Scaling, 06Release-Engineering-Team: Identify metric (or metrics) that gives a useful indication of user-perceived (Wikimedia developer) service of CI - https://phabricator.wikimedia.org/T139771#2479577 (10hashar) From  T70113#2479567:  Maybe...
[12:11:38] <grrrit-wm>	 (03CR) 10Paladox: ":)" [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/299869 (owner: 10Hashar)
[12:12:18] <Amir1>	 hey, this is a beta change that would be amazing if someone +2 it so I can test it: https://gerrit.wikimedia.org/r/#/c/299976/
[12:29:32] <gehel>	 zeljkof: I'm back, let me know when you are
[12:29:43] <zeljkof>	 gehel: just got back too :)
[12:30:27] <mobrovac>	 hashar: thnx for removing https://gerrit.wikimedia.org/r/#/c/298436/ from beta, was about to do it when i saw your comment
[12:30:36] <gehel>	 zeljkof: next thing I'd like to check is that you can connect to the bastion directly (ssh primary.bastion.wmflabs.org)
[12:31:41] <zeljkof>	 gehel: checking
[12:34:29] <wikibugs>	 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2479747 (10Dereckson)
[12:34:38] <hashar>	 mobrovac: not sure in which stance are the target instances though :(  I havent run puppet on them
[12:34:58] <wikibugs>	 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2479759 (10Dereckson)
[12:35:06] <mobrovac>	 hashar: should be good, that patch has been split into multiple ones, most of which have been merged
[12:35:15] <mobrovac>	 hashar: the ones for the role applied on the target, at least
[12:35:21] <mobrovac>	 but i'll check on it a bit later
[12:35:23] <hashar>	 mobrovac: feel free to cherry pick whatever you need :}
[12:35:28] <mobrovac>	 we're in the middle of a transition now
[12:35:31] <wikibugs>	 10Deployment-Systems, 03Scap3, 06Operations: Warning: rename(): Permission denied in /srv/mediawiki/wmf-config/CommonSettings.php on line 189 - https://phabricator.wikimedia.org/T136258#2479761 (10Dereckson) 05Open>03Resolved a:03Dereckson Error disappears from the logs.  Another issue has been spotted...
[12:35:35] <hashar>	 I did the manual rebase to bring in a change to nutcracker and be able to verify it
[12:35:42] <mobrovac>	 kk
[12:35:57] <hashar>	 mobrovac: will cxserver be migrated as well ? ;)
[12:36:14] <wikibugs>	 10Deployment-Systems, 03Scap3, 06Operations, 15User-bd808: Warning: rename(): Permission denied in /srv/mediawiki/wmf-config/CommonSettings.php on line 189 - https://phabricator.wikimedia.org/T136258#2479780 (10Dereckson) a:05Dereckson>03bd808
[12:36:15] <mobrovac>	 hashar: yeah, we'd need to kill its update job too
[12:36:27] <hashar>	 mobrovac: awesome
[12:37:55] <zeljkof>	 gehel: the same thing, asking me for password :(
[12:38:05] <zeljkof>	 so, I have something configured wrong for sure
[12:38:23] <zeljkof>	 not sure how to check which key I have configured for labs
[12:38:40] <gehel>	 zeljkof: ssh -vv ?
[12:40:01] <gehel>	 zeljkof: sorry, could you send me the output of your connection to bastion with -v -v ?
[12:40:17] <gehel>	 zeljkof: lemme check where those SSH keys are configured...
[12:40:51] <zeljkof>	 gehel: will send in a minute
[12:41:07] <gehel>	 zeljkof: your public key should be uploaded to https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack
[12:43:07] <zeljkof>	 gehel: paste https://phabricator.wikimedia.org/P3517
[12:44:50] <gehel>	 zeljkof: ok, your key does not seem to match... can you check that the key you have uploaded in https://wikitech.wikimedia.org/wiki/Special:Preferences#mw-prefsection-openstack matches the key you are using?
[12:45:42] <zeljkof>	 gehel: my current .ssh/config https://github.com/zeljkofilipin/dotfiles/tree/master/.ssh
[12:45:47] <zeljkof>	 just in case
[12:45:52] <zeljkof>	 checking keys
[12:46:01] <zeljkof>	 so I have 3 keys uploaded to wikitech
[12:47:57] <hashar>	 !log integration: enabled unattended upgrade on all instances by adding contint::packages::apt to https://wikitech.wikimedia.org/wiki/Hiera:Integration
[12:48:00] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[12:48:11] <gehel>	 zeljkof: another dumb idea... could you try to explicitely give your username when connecting to bastion? (ssh <user@primary.bastion...)
[12:48:19] <zeljkof>	 gehel: ok, that's it, looks like the key uploaded to wikitech is old, will replace it
[12:48:27] <hashar>	 Host: *
[12:48:32] <hashar>	     User: zfillippin
[12:48:52] <gehel>	 zeljkof: ok, let's hope that's the only issue!
[12:48:57] <hashar>	 bah there are no :   
[12:49:51] <zeljkof>	 hashar: oh, I should add "User zfilipin" under "Host: *"?
[12:50:05] <zeljkof>	 hashar: this is what I have now https://github.com/zeljkofilipin/dotfiles/tree/master/.ssh
[12:50:15] <hashar>	 ssh default to use your local machine user name
[12:50:27] <hashar>	 and your labs account probably uses something different
[12:50:47] <zeljkof>	 hashar: should I have a separate ssh key for labs, or should I reuse the one I use for gerrit/github?
[12:52:08] <hashar>	 I got one for production access, another for github and another for labs :}
[12:52:32] <zeljkof>	 hashar: ok, so then I should probably do the same, thanks
[12:53:23] <shinken-wm>	 PROBLEM - Puppet run on integration-saltmaster is CRITICAL: CRITICAL: 11.11% of data above the critical threshold [0.0]
[12:53:36] <zeljkof>	 hashar: related question, do you reuse the same key across machines? meaning, I have two machines, should I create two separate labs keys, or can I reuse the one I create for labs?
[12:54:06] <zeljkof>	 two labs keys, one per machine; or one labs key, shared on two machines?
[12:57:03] <zeljkof>	 gehel: logged in! "zfilipin@cirrus-browser-bot:~$" cc paladox hashar 
[12:57:20] <zeljkof>	 thanks everybody, the problem was in the old key being uploaded to wikitech
[13:00:59] <hashar>	 ;-D
[13:03:26] <shinken-wm>	 RECOVERY - Puppet run on integration-saltmaster is OK: OK: Less than 1.00% above the threshold [0.0]
[13:04:07] <grrrit-wm>	 (03CR) 10Hashar: [C: 032 V: 032] "Based on my testing that looks all fine." [integration/zuul] (debian/jessie-wikimedia) - 10https://gerrit.wikimedia.org/r/299869 (owner: 10Hashar)
[13:10:12] <gehel>	 zeljkof: everything looks easy once we know what the issue is! Let me know if there is any other issue...
[13:11:04] <zeljkof>	 gehel: thanks a lot for your help!
[13:11:12] <gehel>	 zeljkof: my pleasure!
[13:18:53] <shinken-wm>	 PROBLEM - Puppet run on integration-slave-trusty-1017 is CRITICAL: CRITICAL: 22.22% of data above the critical threshold [0.0]
[13:33:53] <shinken-wm>	 RECOVERY - Puppet run on integration-slave-trusty-1017 is OK: OK: Less than 1.00% above the threshold [0.0]
[13:35:38] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2479965 (10hashar)
[13:39:13] <grrrit-wm>	 (03Abandoned) 10Hashar: Update jQuery to 1.11.3 and jshint to 2.9.2 [integration/raita] - 10https://gerrit.wikimedia.org/r/296883 (owner: 10Paladox)
[13:43:25] <paladox>	 zeljkof :)
[13:43:56] <paladox>	 hashar :)
[13:44:20] <paladox>	 Yay you merged the commits in zuul for jessie
[13:44:21] <paladox>	 :)
[13:53:59] <grrrit-wm>	 (03PS2) 10Hashar: Update beta-scap-eqiad job to use subcommands [integration/config] - 10https://gerrit.wikimedia.org/r/287951 (owner: 10Thcipriani)
[13:55:00] <hashar>	 !log beta: switching job beta-scap-eqiad to use 'scap sync' per https://gerrit.wikimedia.org/r/#/c/287951/  (poke thcipriani )
[13:55:02] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[13:57:30] <paladox>	 hashar I can test your patch that adds recheck in gate and submit
[13:57:31] <paladox>	 :)
[13:57:53] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] "Job updated and passing:" [integration/config] - 10https://gerrit.wikimedia.org/r/287951 (owner: 10Thcipriani)
[13:59:45] * paladox applying some of https://phabricator.wikimedia.org/rCICF9eefe8de9f2f342e158af14f0a2f47604e17aa25 :)
[13:59:49] <grrrit-wm>	 (03Merged) 10jenkins-bot: Update beta-scap-eqiad job to use subcommands [integration/config] - 10https://gerrit.wikimedia.org/r/287951 (owner: 10Thcipriani)
[14:10:30] <paladox>	 hashar i applied the patch
[14:10:36] <paladox>	 but dosent work
[14:10:45] <paladox>	 Still tests in test pipeline even if you do +2
[14:12:04] <grrrit-wm>	 (03CR) 10Hashar: "That works fine on the master branch. On the deployment branch the script tests/LintYaml.php is not present though :-/" [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight)
[14:12:10] <grrrit-wm>	 (03CR) 10Hashar: [C: 04-1] Run DonationInterface LintYaml job via composer-test [integration/config] - 10https://gerrit.wikimedia.org/r/299185 (owner: 10Awight)
[14:13:36] <paladox>	 hashar yay it works now.
[14:13:55] <paladox>	 but maybe what to do recheck gate otherwise it will test both pipelines
[14:15:26] <paladox>	 hashar yay it works
[14:15:31] <paladox>	 it merged the patch
[14:15:39] <andrewbogott>	 Is CI/nodepool still behaving fine since I added the new labvirt hosts to the pool?
[14:15:46] <paladox>	 hashar ^^
[14:24:11] <grrrit-wm>	 (03PS2) 10Hashar: Skip tests on the DonationInterface deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[14:25:10] <grrrit-wm>	 (03CR) 10Hashar: "The Zuul layout diff Jenkins job was not showing any change. I have added a test to make sure the mwext-* jobs are not triggered on deplo" [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[14:25:32] <paladox>	 hashar :)
[14:30:33] <grrrit-wm>	 (03PS18) 10Paladox: 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[14:30:35] <paladox>	 hashar ^^
[14:30:41] <paladox>	 Works and tested
[14:30:48] <paladox>	 Tested on http://gerrit-test.wmflabs.org/gerrit/#/c/14/
[14:30:59] <paladox>	 and
[14:31:00] <paladox>	 http://gerrit-test.wmflabs.org/gerrit/#/c/13/
[14:32:53] <grrrit-wm>	 (03PS19) 10Paladox: 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[14:33:43] <paladox>	 hashar :)
[14:37:50] <grrrit-wm>	 (03PS20) 10Paladox: 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[14:39:38] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] "I have tested this and it works." [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[14:40:28] <paladox>	 hashar :)
[14:41:43] <paladox>	 hashar or do you want it so that users can recheck gate weather they plus 1 or 2 but without merging it if it is not +2.
[14:50:09] <wikibugs>	 10Deployment-Systems, 10scap, 10Analytics, 10Analytics-Cluster, and 3 others: Deploy analytics-refinery with scap3 - https://phabricator.wikimedia.org/T129151#2480203 (10elukey) >>! In T129151#2476622, @thcipriani wrote: >>>! In T129151#2474657, @elukey wrote: >> Summary: >>  >> 1) External scap repository...
[14:50:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2480205 (10Danny_B)
[14:55:52] <grrrit-wm>	 (03PS21) 10Paladox: 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[14:56:27] <grrrit-wm>	 (03CR) 10Paladox: [C: 031] 'recheck gate' on CR+2 now triggers gate-and-submit [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[15:02:55] <wikibugs>	 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations, 13Patch-For-Review: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2480268 (10elukey)
[15:03:14] <grrrit-wm>	 (03PS1) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) 
[15:03:29] <wikibugs>	 10Deployment-Systems, 10MediaWiki-extensions-WikimediaMaintenance, 06Operations, 13Patch-For-Review: WikimediaMaintenance refreshMessageBlobs: wmf-config/wikitech.php requires non existing /etc/mediawiki/WikitechPrivateSettings.php - https://phabricator.wikimedia.org/T140889#2479747 (10elukey) p:05Triage...
[15:04:23] <zeljkof>	 hashar: I want to have two projects named VisualEditor (ruby browser tests and node language screenshots) but then JJB complains :(
[15:04:35] <zeljkof>	 https://gerrit.wikimedia.org/r/#/c/300035/
[15:05:38] <zeljkof>	 ERROR:jenkins_jobs.parser:Duplicate entry found in '(...)selenium.yaml: 'VisualEditor' already defined
[15:06:05] <grrrit-wm>	 (03CR) 10Zfilipin: ":(" [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) (owner: 10Zfilipin)
[15:07:26] <grrrit-wm>	 (03CR) 10jenkins-bot: [V: 04-1] WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) (owner: 10Zfilipin)
[15:09:35] <grrrit-wm>	 (03PS3) 10Hashar: Skip tests on the DonationInterface deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[15:09:43] <grrrit-wm>	 (03CR) 10Hashar: [C: 032] Skip tests on the DonationInterface deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[15:10:57] <grrrit-wm>	 (03Merged) 10jenkins-bot: Skip tests on the DonationInterface deployment branch [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[15:11:58] <paladox>	 zeljkof maybe name it VisualEditor-lang
[15:12:34] <zeljkof>	 paladox: the trick is that the name of the project, "VisualEditor", is used in the job
[15:12:52] <zeljkof>	 for defining repository url, for example
[15:13:09] <zeljkof>	 in _both_ projects, so one of them will have to be tweaked somehow
[15:13:29] <paladox>	 Oh
[15:16:30] <gehel>	 hashar: I'm on clinic duty, having a look at T140894. Looks like you need a review of that package and ops support to actually deploy (you don't seem to be root on scandium). Is that correct?
[15:16:30] <stashbot>	 T140894: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894
[15:16:49] <paladox>	 zeljkof hi here is a template called that
[15:17:01] <paladox>	 See https://github.com/wikimedia/integration-config/blob/df9877644f674166422ceb6b6e7bf391d13fef92/jjb/misc.yaml#L188
[15:17:03] <paladox>	 please
[15:17:17] <paladox>	 zeljkof ^^
[15:17:50] <zeljkof>	 paladox: thanks, that is lowercase visualeditor, I think I need camel case VisualEditor
[15:17:56] <zeljkof>	 nevermind, I will figure it out
[15:18:07] <paladox>	 Ok
[15:18:36] <paladox>	 zeljkof but it seems to that weather it is lowercase or capital it is interfearing.
[15:20:04] <grrrit-wm>	 (03CR) 10Paladox: "This is ready now :)." [integration/config] - 10https://gerrit.wikimedia.org/r/227223 (https://phabricator.wikimedia.org/T105474) (owner: 10Hashar)
[15:25:58] <ostriches>	 heh, I just noticed gerrit 2.12.2 still serves itself as html 4.01 :)
[15:26:17] <grrrit-wm>	 (03CR) 10Hashar: "Deployed" [integration/config] - 10https://gerrit.wikimedia.org/r/299618 (owner: 10Awight)
[15:26:17] <paladox>	 Ha, so it dosent use html 5
[15:26:22] <paladox>	 ostriches ^^
[15:26:48] * ostriches blames <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
[15:27:07] <paladox>	 Is there anyway we can change that?
[15:27:13] <paladox>	 ostriches ^^
[15:28:39] <paladox>	 :)
[15:29:13] <zeljkof>	 paladox: could be
[15:29:21] <zeljkof>	 I think I have an idea
[15:29:29] <paladox>	 Oh :)
[15:30:42] <ostriches>	 paladox: Not that I know of. It's no biggie anyway, just me idly joking about it.
[15:30:52] <paladox>	 Oh
[15:31:18] <paladox>	 ostriches maybe gerrit 2.12 will fix the branch problem.
[15:31:21] <paladox>	 for zuul
[15:34:17] <grrrit-wm>	 (03PS2) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) 
[15:38:32] <grrrit-wm>	 (03PS3) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) 
[15:44:31] <grrrit-wm>	 (03PS4) 10Zfilipin: WIP Run language screenshots script for VisualEditor in Jenkins [integration/config] - 10https://gerrit.wikimedia.org/r/300035 (https://phabricator.wikimedia.org/T139613) 
[15:52:34] <paladox>	 .query hashar
[15:57:31] <grrrit-wm>	 (03CR) 10Jforrester: "> Not all tests have been migrated yet, they are stalled due to labs not having the capacity at the moment." [integration/config] - 10https://gerrit.wikimedia.org/r/291301 (owner: 10Legoktm)
[15:57:32] <wmf-insecte>	 Project language-screenshots-VisualEditor » firefox,beta,Linux,ci-jessie-wikimedia build #1: 04FAILURE in 9 min 33 sec: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=ci-jessie-wikimedia/1/
[15:58:19] <grrrit-wm>	 (03CR) 10Paladox: "Oh yes, we can move those tests into the check pipelines, the ones that work." [integration/config] - 10https://gerrit.wikimedia.org/r/291301 (owner: 10Legoktm)
[15:58:57] <grrrit-wm>	 (03CR) 10Paladox: "recheck" [integration/config] - 10https://gerrit.wikimedia.org/r/291301 (owner: 10Legoktm)
[16:02:56] <hashar>	 paladox: busy busy and I will be off soonish
[16:03:03] <paladox>	 Ok
[16:03:09] <paladox>	 hashar i tested your patch
[16:03:14] <paladox>	 hashar it works
[16:05:11] <paladox>	 :)
[16:07:30] <paladox>	 should  be ready to be merged now
[16:07:36] <paladox>	 :)
[16:10:13] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Zuul: Write / update tutorial for Zuul Debian packaging - https://phabricator.wikimedia.org/T140912#2480636 (10hashar)
[16:10:15] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches): Make long-lived branches on gerrit - https://phabricator.wikimedia.org/T140913#2480649 (10thcipriani)
[16:11:21] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Zuul: Write / update tutorial for Zuul Debian packaging - https://phabricator.wikimedia.org/T140912#2480636 (10Paladox) Oh, I thought you run this command  DEB_BUILD_OPTIONS=nocheck GIT_PBUILDER_AUTOCONF=no DIST=precise WIKIMEDIA=yes git-...
[16:11:56] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Release-Engineering-Team, 10Zuul: Write / update tutorial for Zuul Debian packaging - https://phabricator.wikimedia.org/T140912#2480671 (10Paladox) https://www.mediawiki.org/wiki/Continuous_integration/Zuul#new_package
[16:12:32] <hashar>	 paladox: ah thank you ! yOu have found the doc, I was looking for it :}
[16:12:47] <hashar>	 paladox: will overhaul that one and move it to a standalone sub page
[16:12:56] <hashar>	 including how to setup the build environement
[16:12:59] <wikibugs>	 10scap: Tell user who currently has the lock file - https://phabricator.wikimedia.org/T140914#2480676 (10Reedy)
[16:13:02] <paladox>	 hashar oh, your welcome, i followed it it took some time but i got it to work
[16:13:14] <paladox>	 hashar :)
[16:13:43] <hashar>	 great!
[16:13:50] <paladox>	 Yep
[16:13:56] <hashar>	 I am off, gotta commute back home
[16:13:57] <paladox>	 :)
[16:14:01] <paladox>	 ok
[16:16:31] <wikibugs>	 10scap: Tell user who currently has the lock file - https://phabricator.wikimedia.org/T140914#2480676 (10greg) ```lang=irc 02:26 < ostriches> da heck? 02:26 < ostriches> 02:26:31 sync-file failed: <LockFailedError> Failed to lock /var/lock/scap: [Errno 13] Permission denied: '/var/lock/scap' 02:26 < ostriches> m...
[16:19:11] <Reedy>	 greg-g: I thought we'd have a bug for that at least
[16:19:41] <greg-g>	 :)
[16:20:10] <Reedy>	 based on chads comment.. if user == mwdeploy, we can say "is localisation update running?" or something
[16:20:10] <wmf-insecte>	 Yippee, build fixed!
[16:20:11] <wmf-insecte>	 Project language-screenshots-VisualEditor » firefox,beta,Linux,ci-jessie-wikimedia build #2: 09FIXED in 6 min 6 sec: https://integration.wikimedia.org/ci/job/language-screenshots-VisualEditor/BROWSER=firefox,MEDIAWIKI_ENVIRONMENT=beta,PLATFORM=Linux,label=ci-jessie-wikimedia/2/
[16:32:41] <wikibugs>	 03Scap3, 10scap: Tell user who currently has the lock file - https://phabricator.wikimedia.org/T140914#2480893 (10greg) p:05Triage>03Low Setting as low as this doesn't happen too often (right?).
[16:33:26] <Reedy>	 now everyone comes out of the woodwork
[16:33:32] <Reedy>	 IT HAPPENS ALL THE FRICKEN TIME
[16:34:21] <greg-g>	 shush
[16:34:42] <Reedy>	 heh
[16:34:56] <wikibugs>	 10Continuous-Integration-Config, 06Release-Engineering-Team, 07Regression: mediawiki-core coverage missing ("Error: Empty directory!") - https://phabricator.wikimedia.org/T140917#2480904 (10Krinkle)
[16:35:06] <Reedy>	 It's something that is likely to only "get worse". But I'd suggest low is alright
[16:35:14] <Reedy>	 possibly an easy ish task too
[16:36:32] <Reedy>	 lol
[16:36:33] <Reedy>	 https://github.com/wikimedia/scap/blob/master/scap/utils.py#L342
[16:36:37] <Reedy>	 try catch in the finally
[16:36:38] * Reedy grins
[16:37:43] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): create merge-wmf-branch the successor to make-wmf-branch - https://phabricator.wikimedia.org/T140918#2480927 (10mmodell)
[16:38:53] <ostriches>	 We should always handle that better.
[16:39:08] <wikibugs>	 03Scap3, 10scap, 07Easy: Tell user who currently has the lock file - https://phabricator.wikimedia.org/T140914#2480959 (10greg)
[16:39:14] <greg-g>	 Reedy: https://phabricator.wikimedia.org/T140914#2480959 :P
[16:39:19] <ostriches>	 Funny thing is, you'd hope it get caught earlier.
[16:39:37] <ostriches>	 But it can happen when you have two similarly-priv'd users competing for the lock.
[16:39:40] <ostriches>	 They can race.
[16:40:10] <ostriches>	 One opens, second locks, closes lock, first one (or second or both) can close their fd. Second one to unlink loses.
[16:40:21] <ostriches>	 It's pretty funny tbh
[16:40:35] <ostriches>	 We can do a few things here. Caught the IO race in the finally block.
[16:40:56] <ostriches>	 Second, we should probably ensure our initial permissions when creating the file are restrictive so nobody else can try stealing it
[16:41:02] <ostriches>	 (ie: same group with a bad umask?)
[16:41:34] <ostriches>	 (nobody except root could steal our lock then, but root can't run scap anyway without hacking it)
[16:44:27] <ostriches>	 Patch incoming for both.
[16:45:44] * paladox http://www.bbc.co.uk/bbcthree/item/94a8a9dc-9b22-4206-8ec3-ef0277f38fab
[16:46:30] * paladox http://www.theverge.com/2016/7/20/12234636/pokemon-go-dating-service
[16:47:09] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921#2480982 (10thcipriani)
[17:01:58] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Plan basic logistic of 2016 RelEng team offsite - https://phabricator.wikimedia.org/T134830#2481027 (10greg)
[17:02:23] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Create agenda outline for 2016 RelEng team offsite - https://phabricator.wikimedia.org/T138437#2481029 (10greg)
[17:02:25] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Plan basic logistic of 2016 RelEng team offsite - https://phabricator.wikimedia.org/T134830#2278940 (10greg)
[17:09:02] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Plan basic logistics of 2016 RelEng team offsite - https://phabricator.wikimedia.org/T134830#2481045 (10greg)
[17:12:05] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 06Performance-Team, 07perfnotice: Don't trash cache for front-end resources - https://phabricator.wikimedia.org/T102578#2481048 (10mmodell)
[17:18:01] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Pre 2016 offsite "must do"s - https://phabricator.wikimedia.org/T140923#2481061 (10greg)
[17:18:26] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Pre 2016 offsite "must do"s - https://phabricator.wikimedia.org/T140923#2481075 (10greg)
[17:18:28] <wikibugs>	 06Release-Engineering-Team, 15User-greg, 15User-zeljkofilipin: Plan basic logistics of 2016 RelEng team offsite - https://phabricator.wikimedia.org/T134830#2481076 (10greg)
[17:19:15] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Create agenda outline for 2016 RelEng team offsite - https://phabricator.wikimedia.org/T138437#2481078 (10greg)
[17:19:17] <wikibugs>	 06Release-Engineering-Team, 15User-greg: Pre 2016 offsite "must do"s - https://phabricator.wikimedia.org/T140923#2481061 (10greg)
[17:20:39] <greg-g>	 (sorry 'bout that spam)
[17:21:38] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Labs, 13Patch-For-Review, 07Puppet: /etc/puppet/puppet.conf keeps getting double content - first for labs-wide puppetmaster, then for the correct puppetmaster - https://phabricator.wikimedia.org/T132689#2481082 (10mmodell) a:05mmodell>03None I'm not sure if my patch fix...
[17:22:24] <wikibugs>	 05Gitblit-Deprecate, 06Release-Engineering-Team, 10Diffusion, 07WorkType-NewFunctionality: Use Diffusion as canonical location for browsing code repos (not gitblit) - https://phabricator.wikimedia.org/T752#2481085 (10mmodell)
[17:22:27] <wikibugs>	 05Gitblit-Deprecate, 10Diffusion: Replicate open patchsets to diffusion - https://phabricator.wikimedia.org/T89940#2481084 (10mmodell) 05Open>03Resolved
[17:25:19] <wikibugs>	 05Gerrit-Migration, 10Differential, 07Documentation: Document use of Owners in Phabricator and advertise it - https://phabricator.wikimedia.org/T128372#2481095 (10mmodell)
[17:35:01] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921#2480982 (10mmodell)
[17:35:06] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 06Performance-Team, 07perfnotice: Don't trash cache for front-end resources - https://phabricator.wikimedia.org/T102578#1376635 (10mmodell)
[17:39:03] <grrrit-wm>	 (03PS1) 10Jforrester: GeoCrumbs no longer depends on CustomData in master [integration/config] - 10https://gerrit.wikimedia.org/r/300068 
[17:39:35] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches), 07WorkType-NewFunctionality: Use subrepos instead of git submodules for  deployed MediaWiki extensions - https://phabricator.wikimedia.org/T98834#2481126 (10mmodell)
[17:45:04] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches): Static asset time on disk - https://phabricator.wikimedia.org/T140921#2481159 (10Krinkle) > This may be unnecessary given that we've moved towards using `static.php` where the max-age for assets is 24 hours: >  > ``` > curl -sI 'https://en.wikipedia.org/w/extensi...
[17:49:15] <wikibugs>	 10Deployment-Systems, 06Release-Engineering-Team (Long-Lived-Branches): Remove "php-" from wiki version numbers - https://phabricator.wikimedia.org/T63733#2481181 (10mmodell)
[17:51:01] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016), 15User-greg: Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2481193 (10mmodell) @aklapper: That sounds good to me, let's put it on phab first and then get the ball rolling on a WM Blog...
[17:52:17] <wikibugs>	 06Release-Engineering-Team, 06Developer-Relations (Jul-Sep-2016), 15User-greg: Write blog post highlighting recent Phabricator improvements - https://phabricator.wikimedia.org/T137727#2481194 (10mmodell) Are we happy with the current version?  The formatting looks good in phabricator. I don't anticipate depl...
[17:53:05] <wmf-insecte>	 Project mediawiki-core-code-coverage build #2147: 04STILL FAILING in 2 hr 53 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2147/
[18:06:20] <wikibugs>	 06Release-Engineering-Team, 06ArchCom, 06Developer-Relations, 10Phabricator: Consider alternative processes for Unbreak Now bugs, especially those which cross-cut components - https://phabricator.wikimedia.org/T140207#2481230 (10JAufrecht) As per TPG practice, since a TPGer is following this directly, remo...
[18:37:44] <wikibugs>	 10scap, 10Parsoid, 06Services, 03Scap3 (Scap3-Adoption-Phase1), 15User-mobrovac: Deploy Parsoid with scap3 - https://phabricator.wikimedia.org/T120103#2481310 (10ssastry)
[18:37:52] <wikibugs>	 10Beta-Cluster-Infrastructure, 06Operations, 13Patch-For-Review: Unify ::production / ::beta roles for *oid - https://phabricator.wikimedia.org/T86633#2481313 (10ssastry)
[18:42:23] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2481328 (10demon)
[18:43:23] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2481341 (10demon)
[18:56:22] <paladox>	 hashar hi, i followed that guide on making dpkg for zuul, took a bit but worked in the end.
[18:56:24] <paladox>	 :)
[19:10:32] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2481546 (10hashar) So puppet.log has:Debug: /User[mnoushad]: Autorequiring Group[wikidev]  ``` <LONG PAUSE> Info: /Stage[main]/...
[19:12:05] <icinga-wm>	 PROBLEM - puppet last run on gallium is CRITICAL: CRITICAL: puppet fail
[19:30:54] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#2481685 (10ori)
[19:31:55] <icinga-wm>	 RECOVERY - puppet last run on gallium is OK: OK: Puppet is currently enabled, last run 18 seconds ago with 0 failures
[19:35:29] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#2481712 (10bd808) There are some graphite metrics for authn things: https://grafana.wikimedia.org/dashboard/db/authentication-metrics  One thing we noticed about lo...
[19:45:34] <wikibugs>	 10Continuous-Integration-Infrastructure, 07Jenkins, 13Patch-For-Review: Jenkins files under /var/lib/jenkins/config-history/config need to be garbage collected - https://phabricator.wikimedia.org/T126552#2481743 (10hashar) Take two with tmpreaper which looks easier / sane :]
[19:58:32] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#2481777 (10Tgr) >>! In T140942#2481712, @bd808 wrote: > One thing we noticed about login failures especially during the SessionManager deployment is that this can s...
[20:00:42] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#2481780 (10Tgr) FWIW I think the obvious thing to alert on in this case would have been the thousands of exceptions per day in production.
[20:10:34] <wikibugs>	 06Release-Engineering-Team, 10Monitoring, 06Operations: Monitoring and alerts for "business" metrics - https://phabricator.wikimedia.org/T140942#2481801 (10Tgr) It seems like T117470 would have helped as well (compare [[https://logstash.wikimedia.org/app/kibana#/dashboard/Fatal-Monitor?_g=(refreshInterval:(d...
[20:40:50] <wikibugs>	 07Browser-Tests, 10MobileFrontend, 06Reading-Web-Backlog, 03Reading-Web-Sprint-78-T, 07Technical-Debt: Wikidata description browser tests do not run anywhere - https://phabricator.wikimedia.org/T137756#2481894 (10Jdlrobson) Ping @jhernandez we should either fix this or delete the test asap.
[20:45:27] <ostriches>	 Heh https://phabricator.wikimedia.org/D293
[20:52:14] <wikibugs>	 07Browser-Tests, 10Continuous-Integration-Config, 10MediaWiki-extensions-RelatedArticles, 06Reading-Web-Backlog: RelatedArticles browser tests should run on a commit basis - https://phabricator.wikimedia.org/T120715#2481928 (10Jdlrobson) What happened here? It doesn't seem this ever happened (cc @Jhernandez )
[21:30:17] <wikibugs>	 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: Clones from git.wikimedia.org are not redirected - https://phabricator.wikimedia.org/T139206#2422729 (10saper) I just run into this today by adding this to my `composer.local.json`:  ```` {     "require": {         "mediawiki/vector-skin": "@dev"...
[21:31:58] <wikibugs>	 05Gitblit-Deprecate, 06Release-Engineering-Team, 06Operations: Clones from git.wikimedia.org are not redirected - https://phabricator.wikimedia.org/T139206#2482107 (10Paladox) Yes please.
[22:00:10] <wikibugs>	 05Gitblit-Deprecate, 10Vector: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482191 (10saper)
[22:05:36] <wikibugs>	 05Gitblit-Deprecate, 10Vector: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482191 (10matmarex) https://www.mediawiki.org/wiki/Topic:T7yqmvlxtyvl210h
[22:06:50] <wikibugs>	 05Gitblit-Deprecate, 10Vector: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482233 (10Paladox) p:05Triage>03Unbreak! I thought this was done.  Setting this as unbreak since currently broken.
[22:33:53] <wikibugs>	 05Gitblit-Deprecate, 10Vector, 07Composer: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482302 (10greg)
[22:50:32] <wmf-insecte>	 Project mediawiki-core-code-coverage build #2148: 04STILL FAILING in 2 hr 26 min: https://integration.wikimedia.org/ci/job/mediawiki-core-code-coverage/2148/
[22:56:22] <wikibugs>	 06Release-Engineering-Team (Long-Lived-Branches), 10scap, 06Operations: Make git 2.2.0+ (preferably 2.8.x) available - https://phabricator.wikimedia.org/T140927#2481328 (10mmodell) 2.9.2 is backported to trusty at https://launchpad.net/~git-core/+archive/ubuntu/ppa  It seems fairly straightforward to backpor...
[23:00:18] <wikibugs>	 10Continuous-Integration-Infrastructure, 06Operations, 10Zuul, 07Blocked-on-Operations: Upgrade Zuul on scandium.eqiad.wmnet (Jessie zuul-merger) - https://phabricator.wikimedia.org/T140894#2482428 (10demon) p:05Triage>03High Let's do this tomorrow morning maybe?
[23:08:24] <wikibugs>	 05Gitblit-Deprecate, 10Vector, 07Composer, 15User-bd808: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482450 (10bd808) 05Open>03Resolved a:03bd808 I changed the link to https://phabricator.wikimedia.org/diffusion/SVEC/vector.git...
[23:13:43] <wikibugs>	 05Gitblit-Deprecate, 10Vector, 07Composer, 15User-bd808: Change git.wikimedia.org link for mediawiki/vector-skin on packagist.org - https://phabricator.wikimedia.org/T140954#2482480 (10bd808) Also, just to be on record about this, a broken composer install of any skin or extension via packagist is not a UB...