[00:33:27] <wikibugs>	 10Deployment-Systems, 3Scap3: scap3 should repack / pack-refs git repos under /srv/deployment - https://phabricator.wikimedia.org/T112509#1800144 (10thcipriani)
[00:56:55] <greg-g>	 dr0ptp4kt: ish, been sick most of the day
[00:58:41] <dr0ptp4kt>	 greg-g: just wanted to check on meaning of week0 and week1 and wmf1 v wmf2. basically if stuff is merged before the tuesday morning cutoff in master, it will be on stable wikipedias by thursday, right? we can chat in person if there's more complexity than meets the eye.
[00:59:41] <greg-g>	 dr0ptp4kt: this should answer all your questions: https://wikitech.wikimedia.org/wiki/Deployments/One_week
[01:00:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1800165 (10Catrope) I analyzed all possible query types for ContributionsQuery over...
[01:00:22] <dr0ptp4kt>	 greg-g: yeah, read that. mainly, i was wondering could the table just have one row instead of two?
[01:00:44] <greg-g>	 yeah, it had two because we used to have a 2 week cycle :)
[01:00:56] <dr0ptp4kt>	 greg-g: :)
[01:01:04] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1800169 (10Catrope) >>! In T78671#1770170, @dduvall wrote: > Another instance of thi...
[01:01:07] <greg-g>	 so just to be explicit I left it as two, but I can remove it now since we're probably mostly all used to the one week cycle now
[01:01:49] <dr0ptp4kt>	 greg-g: cool. just wanted to make sure i wasn't missing something! thanks! get well!
[01:02:30] <greg-g>	 dr0ptp4kt: reload
[01:02:52] <dr0ptp4kt>	 greg-g: sweet
[01:02:55] <dr0ptp4kt>	 greg-g: 'night
[01:03:11] <greg-g>	 :) g'night
[01:10:16] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1800176 (10Catrope) >>! In T78671#1800169, @Catrope wrote: >>>! In T78671#1770170, @...
[01:17:50] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: PING CRITICAL - Packet loss = 100%
[01:39:03] <wmf-insecte>	 Yippee, build fixed!
[01:39:04] <wmf-insecte>	 Project browsertests-Wikidata-SmokeTests-linux-firefox-sauce build #438: 09FIXED in 22 min: https://integration.wikimedia.org/ci/job/browsertests-Wikidata-SmokeTests-linux-firefox-sauce/438/
[03:13:49] <wmf-insecte>	 Project beta-scap-eqiad build #78154: 04FAILURE in 9 min 2 sec: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/78154/
[05:28:39] <wmf-insecte>	 Project browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce build #600: 04FAILURE in 26 min: https://integration.wikimedia.org/ci/job/browsertests-MultimediaViewer-en.wikipedia.beta.wmflabs.org-windows_7-internet_explorer-11-sauce/600/
[06:03:01] <wmf-insecte>	 Yippee, build fixed!
[06:03:02] <wmf-insecte>	 Project beta-scap-eqiad build #78168: 09FIXED in 37 min: https://integration.wikimedia.org/ci/job/beta-scap-eqiad/78168/
[06:06:24] <shinken-wm>	 PROBLEM - Free space - all mounts on deployment-bastion is CRITICAL: CRITICAL: deployment-prep.deployment-bastion.diskspace._var.byte_percentfree (<11.11%)
[06:18:31] <grrrit-wm1>	 (03PS1) 10Gilles: Configure thumbor/result-storage [integration/config] - 10https://gerrit.wikimedia.org/r/252638 
[06:36:22] <shinken-wm>	 RECOVERY - Free space - all mounts on deployment-bastion is OK: OK: All targets OK
[08:51:01] <icinga-wm_>	 PROBLEM - puppet last run on scandium is CRITICAL: CRITICAL: puppet fail
[09:06:03] <icinga-wm_>	 RECOVERY - puppet last run on scandium is OK: OK: Puppet is currently enabled, last run 1 minute ago with 0 failures
[09:33:26] <wikibugs>	 10Beta-Cluster-Infrastructure, 6Collaboration-Team-Backlog, 10Flow, 3Collaboration-Team-Current, and 2 others: Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1800517 (10jcrespo) > This behavior is strange  No it is not, you cann...
[09:52:16] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.71 ms
[09:54:46] <shinken-wm>	 PROBLEM - Puppet staleness on integration-dev is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[10:07:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Flow, 3Collaboration-Team-Current, 5Patch-For-Review, 5WMF-deploy-2015-11-17_(1.27.0-wmf.7): Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1800539 (10Luke081515)
[11:27:15] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[11:29:05] <wikibugs>	 6Release-Engineering-Team, 7Ruby, 7Tracking: Fix easy problems reported by RuboCop - https://phabricator.wikimedia.org/T91485#1800677 (10zeljkofilipin)
[11:29:40] <wikibugs>	 7Browser-Tests, 10MediaWiki-extensions-Translate, 5Patch-For-Review, 7Ruby, 5WMF-deploy-2015-11-10_(1.27.0-wmf.6): Update Translate mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T117978#1800678 (10zeljkofilipin) 5Open>3Resolved
[11:29:41] <wikibugs>	 6Release-Engineering-Team, 10Browser-Tests-Infrastructure, 7Ruby, 7Tracking: Update repositories that use mediawiki_selenium Ruby gem to version 1.x - https://phabricator.wikimedia.org/T94083#1800679 (10zeljkofilipin)
[13:26:30] <grrrit-wm1>	 (03PS1) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) 
[13:51:29] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review, 7Ruby, 5WMF-deploy-2015-11-10_(1.27.0-wmf.6): Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1800884 (10zeljkofilipin)
[13:52:27] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review, 7Ruby, 5WMF-deploy-2015-11-10_(1.27.0-wmf.6): Add Rakefile to repositories with Ruby code - https://phabricator.wikimedia.org/T117993#1793788 (10zeljkofilipin)
[13:54:10] <zeljkof>	 hashar: this is done https://phabricator.wikimedia.org/T117993
[13:54:29] <zeljkof>	 https://gerrit.wikimedia.org/r/#/q/status:open+topic:T117993,n,z
[13:54:41] <zeljkof>	 I will add rake-jessie job to the repos now
[13:55:21] <hashar>	 good good
[13:55:29] <zeljkof>	 https://phabricator.wikimedia.org/T114860
[13:55:42] <hashar>	 don't apply it to ops-puppet for now though
[13:55:57] <zeljkof>	 hashar: no, I will create a separate patch for that
[13:56:08] <zeljkof>	 since we can not merge puppet ourselves
[13:56:09] <hashar>	 I don't think operations/puppet.git depends on Nodepool hosts at all
[13:56:21] <hashar>	 and that repo is rather critical to the infrastructure
[13:56:29] <hashar>	 so I don't want ops to be blocked because nodepool ends up having a trouble
[13:56:32] <hashar>	 (I am paranoid)
[13:56:40] <zeljkof>	 hashar: ok
[13:56:49] <zeljkof>	 I will create a separate patch
[13:56:53] <zeljkof>	 we can ignore it for now
[14:19:42] <grrrit-wm>	 (03PS1) 10Zfilipin: Added rake-jessie job to test and gate-and-submit pipelines for operations/puppet repository [integration/config] - 10https://gerrit.wikimedia.org/r/252689 (https://phabricator.wikimedia.org/T114860) 
[14:22:38] <grrrit-wm>	 (03PS2) 10Zfilipin: Added rake-jessie job to test and gate-and-submit pipelines for operations/puppet repository [integration/config] - 10https://gerrit.wikimedia.org/r/252689 (https://phabricator.wikimedia.org/T114860) 
[14:36:21] <grrrit-wm>	 (03PS2) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) 
[14:37:02] <grrrit-wm>	 (03PS1) 10Zfilipin: Run Ruby jobs using Rake [integration/config] - 10https://gerrit.wikimedia.org/r/252690 (https://phabricator.wikimedia.org/T114860) 
[14:39:36] <wikibugs>	 7Browser-Tests, 10Continuous-Integration-Config, 5Patch-For-Review, 7Ruby: Delete ruby2.0lint job and only run bundle-rubocop job for repositories with Ruby code - https://phabricator.wikimedia.org/T114262#1800925 (10zeljkofilipin)
[14:39:37] <wikibugs>	 10Continuous-Integration-Config, 5Patch-For-Review, 7Ruby: Move Bundler Jenkins jobs to Nodepool instances - https://phabricator.wikimedia.org/T114860#1800926 (10zeljkofilipin)
[15:15:25] <hashar>	 zeljkof: all those rubocop whitespaces changes should probably be made in a single commit :D
[15:15:29] <hashar>	 mass approving anyway
[15:15:43] <zeljkof>	 hashar: probably
[15:15:48] <zeljkof>	 but I am never sure
[15:15:53] <zeljkof>	 and I prefer smaller commits
[15:16:02] <zeljkof>	 and thanks :)
[15:22:22] <wikibugs>	 10Gitblit-Deprecate, 10Diffusion: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#1800964 (10demon) I do! Gimme a minute to regenerate that list and I'll pastebin it here on Phab.
[15:26:13] <wikibugs>	 10Gitblit-Deprecate, 10Diffusion: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#1800977 (10demon) This is everything as of a ~week ago: {P2306}  I haven't gone and created any repos since then really so it should be most everything minus the ~30-50 repos that hav...
[15:26:40] <ostriches>	 twentyafterfour: Ask and ye shall receive :)
[15:29:13] <grrrit-wm>	 (03PS1) 10Zfilipin: Added building the gem to `rake test` (CI entry point for Ruby) [selenium] - 10https://gerrit.wikimedia.org/r/252693 (https://phabricator.wikimedia.org/T117993) 
[15:29:18] <twentyafterfour>	 ostriches: awesome!
[15:29:29] <twentyafterfour>	 I will try to put that to use before it gets more outdated ;)
[15:29:51] <ostriches>	 It's already outdated :p
[15:29:53] <twentyafterfour>	 and I suppose we need a procedure to keep it updated until gerrit gets locked into read-only
[15:30:05] <twentyafterfour>	 I said _more_ outdated ;)
[15:30:08] <ostriches>	 But it's like 95% of the repos and it's just adding more later, no changes :)
[15:30:12] <grrrit-wm>	 (03CR) 10Zfilipin: [C: 04-2] "fixing something" [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[15:30:41] <ostriches>	 twentyafterfour: Yeah, I've been noodling the repo creation process a tad trying to help make sanity while we're in the migration period.
[15:30:42] <twentyafterfour>	 it's actually difficult to find/browse repos in phabricator, which is unfortunate
[15:31:07] <ostriches>	 Yeah! Diffusion search kinda sucks bad.
[15:31:09] <twentyafterfour>	 I found a couple of duplicates, which I think were probably my fault.
[15:31:44] <twentyafterfour>	 no deterministic unique identifier other than the callsign is still an annoyance
[15:32:44] <grrrit-wm>	 (03PS1) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252695 (https://phabricator.wikimedia.org/T117993) 
[15:33:51] <grrrit-wm>	 (03Abandoned) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252695 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[15:34:20] <grrrit-wm>	 (03CR) 10Zfilipin: [C: 04-1] Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[15:35:29] <grrrit-wm>	 (03CR) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[15:38:19] <grrrit-wm>	 (03PS1) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252698 (https://phabricator.wikimedia.org/T117993) 
[15:39:06] <grrrit-wm>	 (03Abandoned) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252683 (https://phabricator.wikimedia.org/T117993) (owner: 10Zfilipin)
[15:39:29] <grrrit-wm>	 (03PS2) 10Zfilipin: Added Rakefile [ruby/api] - 10https://gerrit.wikimedia.org/r/252698 (https://phabricator.wikimedia.org/T117993) 
[15:55:19] <andrewbogott>	 hey, can I get a hand with a sync-common issue in #wikimedia-operations?
[15:55:34] <hashar>	 looking for my headphones :/
[16:00:51] <hashar>	 found them
[16:00:58] <hashar>	 andrewbogott: paste the error I guess ?
[16:01:05] <hashar>	 most of us are entering a meeting though
[16:01:30] <andrewbogott>	 hashar: brian is helping, thanks.
[16:24:25] <grrrit-wm>	 (03CR) 10Hashar: "So mw-tools-scap-tox-doc-publish got refreshed and does poll Phabricator. There is a command error which is unrelated to CI itself (scap " [integration/config] - 10https://gerrit.wikimedia.org/r/251442 (https://phabricator.wikimedia.org/T117770) (owner: 10Thcipriani)
[16:31:42] <wikibugs>	 10Continuous-Integration-Config, 10Continuous-Integration-Infrastructure, 10Gerrit, 5Patch-For-Review, 7Technical-Debt: Disable Gerrit replication to production slaves - https://phabricator.wikimedia.org/T86661#1801181 (10demon) Gerrit is no longer replicating to gallium either. Only replication targets...
[16:33:42] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.72 ms
[16:49:33] <wikibugs>	 7Browser-Tests, 5Patch-For-Review, 3Reading Web Sprint 60 - Boom Headshot!: Investigate QuickSurveys browser tests failures - https://phabricator.wikimedia.org/T113534#1801209 (10KLans_WMF)
[16:51:32] <James_F>	 greg-g: So… wmf.7 next week (17th); wmf.8 two weeks after next (1st) as we're skipping Thanksgiving week; wmf.9 thereafter (8th); no deploys for the weeks of 15th/22nd/29th/5th; wmf.10 on 12 January 2016. Sound right?
[16:52:51] <hashar>	 James_F: I asked for a deployment freeze on releng internal li st
[16:53:01] <hashar>	 and yeah a month or so of freeze sounds good
[16:54:44] <James_F>	 OK, I did https://www.mediawiki.org/w/index.php?title=MediaWiki_1.27/Roadmap&diff=1937345&oldid=1937187 – please revert/edit if wrong.
[17:02:10] <greg-g>	 James_F: 5th as in Jan 5th? I didn't think about all hands yet, and that's right, we should skip that, sadly, but dangit, that makes it 3 weeks :/
[17:02:19] <greg-g>	 cc twentyafterfour ^ just fyi
[17:02:22] <James_F>	 greg-g: Four.
[17:02:25] <greg-g>	 twentyafterfour: also if you have any other opinoins
[17:02:39] <greg-g>	 er, 15th we have deploys
[17:03:04] <greg-g>	 week of the 14th we'll do our last of the year train run
[17:03:12] <twentyafterfour>	 cool
[17:03:24] <twentyafterfour>	 I was already in the loop ;)
[17:03:41] <greg-g>	 and week of dec 7th will also be a train run
[17:04:05] <greg-g>	 I gotta run to PT real quick, be back in an hour, but I'll make it explicit on [[wikitech:deployments]] more than it already is
[17:04:36] <thedj>	 someone broke h2 in production..
[17:04:47] * thedj checks if it's en.wp only
[17:05:19] <thedj>	 quick survey
[17:05:57] <thedj>	 @media all and (min-width:768px){.ext-qs-loader-bar,.ext-quick-survey-panel{margin-left:1.4em;width:300px;clear:right;float:right}.infobox,.last-modified-bar,h2{clear:both}}
[17:06:04] <thedj>	 that's a problem.
[17:13:10] <thcipriani>	 thedj: quicksurvey was deployed as part of SWAT this morning for enwiki
[17:15:49] <James_F>	 greg-g: Yeah, but if 8th is a train run then 15th/22nd/29th/5th is four weeks.
[17:18:27] <thedj>	 thcipriani: yeah, sorry, i noticed i was in the wrong chan. moved to -ops and -mobile
[17:21:52] <wikibugs>	 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should be aware of security patches - https://phabricator.wikimedia.org/T118477#1801277 (10demon) 3NEW
[17:23:39] <wikibugs>	 6Release-Engineering-Team, 3Scap3, 7Security-General: Scap should apply security patches - https://phabricator.wikimedia.org/T118478#1801285 (10demon) 3NEW
[17:24:05] <ostriches>	 twentyafterfour: From our discussion earlier ^
[17:25:00] <ostriches>	 (Lots of overlap, probably solved together, but I think it's actually 2 different things)
[17:29:33] <grrrit-wm>	 (03PS1) 10Zfilipin: Move bundle-rubocop job from experimental to test pipeline for operations/puppet [integration/config] - 10https://gerrit.wikimedia.org/r/252716 (https://phabricator.wikimedia.org/T110019) 
[17:30:45] <grrrit-wm>	 (03PS3) 10Zfilipin: Added rake-jessie job to test and gate-and-submit pipelines for operations/puppet repository [integration/config] - 10https://gerrit.wikimedia.org/r/252689 (https://phabricator.wikimedia.org/T110019) 
[18:15:44] <ostriches>	 thcipriani, twentyafterfour: Pushed .arcconfig to scap-vagrant (since we...erm...use phab for it :p)
[18:16:06] <thcipriani>	 ostriches: good call :)
[18:34:06] <bd808>	 hey releng folks, can I get a review from somebody on https://gerrit.wikimedia.org/r/#/c/251007 ? monolog stuff
[18:34:14] * ostriches pokes
[18:34:37] <ostriches>	 Wrong link?
[18:34:55] <bd808>	 well yeah...
[18:35:03] <bd808>	 ostriches: https://gerrit.wikimedia.org/r/#/c/252359/2
[18:37:23] <ostriches>	 So this is a workaround for a broken monolog "fix" pending another fix?
[18:37:39] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[18:40:16] <bd808>	 ostriches: yeah
[18:40:50] <bd808>	 right now we have a small number (<200) of requests a day that are blowing up because they have user-agents with latin9 encoded data
[18:40:59] <bd808>	 the new monolog pukes on that
[18:41:09] <bd808>	 the old one jsut silently dropped those logs
[18:41:18] <bd808>	 this patch brings that back
[18:42:15] <ostriches>	 lgtm then
[18:44:02] <greg-g>	 James_F: the 15th is another train
[18:44:39] <James_F>	 greg-g: Oh, OK.
[18:44:45] <greg-g>	 :)
[18:45:05] <greg-g>	 I just sat back down, I'll make [[wikitech:Deployments]] more explicit real quick
[18:50:16] <greg-g>	 James_F: https://wikitech.wikimedia.org/wiki/Deployments#Upcoming
[18:52:38] <greg-g>	 btw, grip strength test went from around 60 (somethings, not sure of the unit) to 80 or 90. yay!
[18:53:03] <James_F>	 greg-g: https://www.mediawiki.org/w/index.php?title=MediaWiki_1.27/Roadmap&diff=1937372&oldid=1937345
[18:54:05] <greg-g>	 James_F: thank you
[18:55:07] <James_F>	 greg-g: And I've created the appropriate milestone projects in Phabricator.
[18:58:15] <greg-g>	 <3
[18:58:48] * James_F waits patiently for the train in order to archive wmf.5.
[18:58:53] <greg-g>	 subbu: btw, parsoidcache02 is down in beta cluster, apparently ^^^ (at 18:37)
[19:01:02] <subbu>	 hmm .. 
[19:01:23] <subbu>	 i am wondering who set up these caches. 
[19:03:28] <subbu>	 Krenair thinks hashar changed it recently, but I dont see him around.
[19:03:47] <subbu>	 I am not familiar with the varnish setup .. so, little hard for me to poke at it.
[19:06:02] <subbu>	 bblack is marked away in ops as well.  mobrovac gwicke either of you familiar with varnish setup in beta cluster (parsoidcache02 is down there).
[19:07:41] <gwicke>	 subbu: my only guess would be something limit-related
[19:07:45] <gwicke>	 did you try restarting it?
[19:07:56] <gwicke>	 labs was down recently when the rate limit change was deployed
[19:08:02] <gwicke>	 a restart fixed it
[19:08:07] <subbu>	 i am not sure i have the permissions for it.
[19:08:26] <gwicke>	 if you are in the deployment-prep project, then you should have root on that box
[19:08:38] <gwicke>	 https://phabricator.wikimedia.org/T118362
[19:08:46] <gwicke>	 this was the limit issue ^^
[19:13:38] <subbu>	 ssastry@bastion-01:~$ ssh -A ssastry@deployment-parsoidcache02
[19:13:38] <subbu>	 ssh: Could not resolve hostname deployment-parsoidcache02: Name or service not known
[19:14:05] <subbu>	 this requires someone in ops to take a look
[19:14:37] <thcipriani>	 I think we removed deployment-parsoidcache02 some time ago
[19:15:06] <greg-g>	 heh, then why was shinken recently annoyed? stupid shinken ;)
[19:15:53] <thcipriani>	 https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL#2015-10-28
[19:16:18] <James_F>	 greg-g: Is getting a bunch of HTTP 429s for image requests in Beta Cluster a known thing?
[19:16:34] <wikibugs>	 10Beta-Cluster-Infrastructure, 10Flow, 3Collaboration-Team-Current, 5Patch-For-Review, 5WMF-deploy-2015-11-17_(1.27.0-wmf.7): Beta Cluster Special:Contributions lags by a long time and notes slow Flow queries - https://phabricator.wikimedia.org/T78671#1801612 (10Catrope) I agree it should be a proper JOI...
[19:19:26] <greg-g>	 James_F: I... dont' think so?
[19:19:40] <James_F>	 greg-g: OK, will re-purpose the task.
[19:20:17] <legoktm>	 lock errors are back?
[19:20:17] <legoktm>	 19:17:42 IOError: Lock at '/mnt/jenkins-workspace/workspace/mediawiki-extensions-qunit/src/extensions/MobileApp/.git/refs/heads/wmf/1.23wmf14.lock' could not be obtained
[19:21:17] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1801621 (10Jdforrester-WMF)
[19:23:42] <grrrit-wm>	 (03PS1) 10Paladox: [Reflect] Add Jenkins tests to extension [integration/config] - 10https://gerrit.wikimedia.org/r/252755 
[19:25:28] <paladox>	 jzerebecki: Could you review https://gerrit.wikimedia.org/r/#/c/252755/ please.
[19:26:46] <jzerebecki>	 paladox: not right now
[19:26:53] <paladox>	 Ok.
[19:32:16] <shinken-wm>	 RECOVERY - Host deployment-parsoidcache02 is UP: PING OK - Packet loss = 0%, RTA = 0.66 ms
[20:22:45] <shinken-wm>	 PROBLEM - Puppet staleness on deployment-restbase01 is CRITICAL: CRITICAL: 100.00% of data above the critical threshold [43200.0]
[20:26:01] <wikibugs>	 10Gitblit-Deprecate, 10Diffusion: redirect gerrit repo paths to diffusion callsigns - https://phabricator.wikimedia.org/T110607#1801832 (10Spage) >>! In T110607#1633472, @mmodell wrote: > @spage: upstream is working on path/to/repo support in diffusion. See [[ https://secure.phabricator.com/T4245 | upstream ta...
[20:59:41] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1801946 (10Jdforrester-WMF) Errors are being thrown by `deploym...
[21:19:23] <Krinkle>	 twentyafterfour: I think I was rejecting an idea you didn't propose. Sorry.
[21:19:31] <Krinkle>	 twentyafterfour: How would you add that commit in the other branch?
[21:19:44] <Krinkle>	 Without bypassing Gerrit
[21:19:49] <Krinkle>	 or Jenkins
[21:27:08] <twentyafterfour>	 Krinkle: wouldn't I just merge it to master?
[21:27:41] <Krinkle>	 twentyafterfour: You mean by pushing a merge commit to gerrit as review?
[21:28:09] <Krinkle>	 a merge commit between the previously merged commit and the head of master
[21:28:15] <Krinkle>	 or something like that
[21:28:38] <Krinkle>	 I'm not sure how Gerrit behaves in that regard, whether it's going to complain that the commit isn't merged yet.
[21:28:51] <Krinkle>	 It'll probably require it to go in master first, btu that could work yeah
[21:28:56] <Krinkle>	 then we'd have the same SHA1 in both branche
[21:29:30] <Krinkle>	 But then youd also pull in teh rest of master
[21:29:31] <twentyafterfour>	 Krinkle: that was my goal yeah
[21:29:32] <Krinkle>	 I see what you mean
[21:29:42] <Krinkle>	 You'd have to do it the other way around
[21:29:52] <Krinkle>	 I think that's conceptually wrong though
[21:30:04] <twentyafterfour>	 since the wmf/ branch contains only commits from master, you can always safely merge it back to master, ideally
[21:32:45] <twentyafterfour>	 as you can see I'm still thinking it through, and I appreciate your perspective on it. I agree it doesn't seem conceptually perfect. I wish git had a cherry-pick that wouldn't conflict with it's self
[21:40:26] <twentyafterfour>	 Krinkle: there may be some totally different solution, but eventually gerrit (and the convenient cherry-pick button) is going away so I'm trying to come up with a better way to manage hotfixes (to streamline deployments and generally improve the workflow)
[21:40:36] <twentyafterfour>	 so this was the best thing I came up with so far
[21:44:53] <wikibugs>	 10Deployment-Systems, 3Scap3: Write setup.py for scap - https://phabricator.wikimedia.org/T118504#1802112 (10dduvall) 3NEW a:3dduvall
[21:52:38] <shinken-wm>	 PROBLEM - Host deployment-parsoidcache02 is DOWN: CRITICAL - Host Unreachable (10.68.16.145)
[22:02:32] <ostriches>	 hashar: Is anything even pointing at parsoidcache02 anymore?
[22:02:36] <ostriches>	 It keeps flapping
[22:02:55] <hashar>	 I think we migrated out of it to Jessie
[22:02:57] <legoktm>	 qunit workspace on integration-slave-trusty-1012 is busted, I'm rm-rf'ing
[22:02:59] <hashar>	 if it is trusty I guess you can nuke it
[22:03:07] <legoktm>	 https://integration.wikimedia.org/ci/job/mediawiki-extensions-qunit/20240/consoleFull
[22:03:50] <legoktm>	 !log rm -rf mediawiki-extensions-qunit workspace on trusty-1012
[22:03:56] <qa-morebots>	 Logged the message at https://wikitech.wikimedia.org/wiki/Release_Engineering/SAL, Master
[22:05:09] <ostriches>	 parsoidcache02 dne.
[22:05:17] <ostriches>	 That's just shinken being dumb hmm
[22:05:27] <ostriches>	 You deleted it on the 28th
[22:21:16] <etonkovidova>	 marxarelli: did you send me a link to the doc about Selenium upgrade/update? :) 
[22:21:31] <marxarelli>	 etonkovidova: https://doc.wikimedia.org/rubygems/mediawiki-selenium/
[22:21:56] <etonkovidova>	 marxarelli: thx a lot! Cause mooeypoo does not have it :)))
[22:22:04] <marxarelli>	 there was an email to the QA about it a while back, too. but the readme should contain everything you need
[22:22:06] <marxarelli>	 np!
[22:22:15] <marxarelli>	 QA *list*
[22:24:56] <hashar>	 ostriches: i have no idea how Shinken collects its data :/
[22:25:11] <ostriches>	 black magic.
[22:25:25] <hashar>	 yuvipanda would know
[22:25:41] <hashar>	 maybe it grabs that using a Semantic query and the host is still on wikitech
[22:34:37] <grrrit-wm>	 (03CR) 1020after4: [C: 032] add QuickSurveys extension to make-wmf-branch conf [tools/release] - 10https://gerrit.wikimedia.org/r/251140 (owner: 1020after4)
[22:35:10] <grrrit-wm>	 (03Merged) 10jenkins-bot: add QuickSurveys extension to make-wmf-branch conf [tools/release] - 10https://gerrit.wikimedia.org/r/251140 (owner: 1020after4)
[22:36:09] <twentyafterfour>	 Krinkle: you're right, gerrit does funky things with my merge: https://gerrit.wikimedia.org/r/#/c/252854/
[22:37:24] <twentyafterfour>	 it though I think it's the 'new wmf branch' commit that causes trouble.
[22:37:39] * twentyafterfour goes back to the drawing board.
[22:38:49] <Krinkle>	 twentyafterfour: Aye, I'm sorry.
[22:39:05] <twentyafterfour>	 Krinkle: well thanks for humoring me ;)
[22:39:30] <twentyafterfour>	 https://gerrit.wikimedia.org/r/#/c/252857/
[22:42:21] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802367 (10hashar) The error 429 is...
[22:44:27] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802380 (10hashar) And `//etc/varnis...
[23:02:55] <AndyRussG>	 hi! http://en.m.wikipedia.beta.wmflabs.org/ is down?
[23:03:34] <Reedy>	 AndyRussG: Not exactly
[23:03:44] <AndyRussG>	 Reedy: ?
[23:03:51] <AndyRussG>	 Just wanted to test some mobile stuff
[23:04:06] <Reedy>	 AndyRussG: it's a rate limiting issue in varnish that was rolled out to the production cluster
[23:04:25] <Reedy>	 Hmm
[23:04:26] <Reedy>	 http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page
[23:04:27] <Reedy>	 works fine
[23:04:37] <AndyRussG>	 Reedy: but the mobile site is down ^
[23:04:42] <Reedy>	 Yeah
[23:04:45] <Reedy>	 Well, no
[23:04:47] <Reedy>	 it's not down
[23:04:56] <Reedy>	 You just can't access it :(
[23:05:15] <AndyRussG>	 https://en.wikipedia.org/wiki/Main_Page?useformat=mobile seems OK, hmmm
[23:05:26] <Reedy>	 Is mobile on a different varnish?
[23:05:28] <Reedy>	 ostriches: ?
[23:05:39] <ostriches>	 Ya
[23:05:43] <Krenair>	 yes
[23:05:45] <Reedy>	 I wonder if that didn't get updated
[23:05:47] * Reedy looks
[23:06:01] <Krenair>	 it's cache-mobile instead of cache-text
[23:06:16] <Reedy>	 AndyRussG: https://phabricator.wikimedia.org/T118362 is the original issue
[23:06:19] <Reedy>	 cheers
[23:06:45] <AndyRussG>	 Reedy: K thanks!!
[23:06:51] * Reedy reopens
[23:06:58] <wikibugs>	 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1802450 (10Reedy) 5Resolved>3Open http://en.m.wikipedia.beta.wmflabs.org/ is broken
[23:08:14] <Reedy>	 session bug
[23:08:26] <Krenair>	 I restarted varnish and it didn't fix anything :/
[23:08:51] <Reedy>	 I think it saves it in some file
[23:08:56] <AndyRussG>	 Reedy: aaarg silly me that link above wasn't beta 
[23:09:37] <AndyRussG>	 Reedy: however this does work: http://en.wikipedia.beta.wmflabs.org/wiki/Main_Page?useformat=mobile
[23:09:39] <Krenair>	 it saves what?
[23:09:59] <wikibugs>	 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1802464 (10Jdforrester-WMF)
[23:10:01] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802463 (10Jdforrester-WMF)
[23:11:57] <Reedy>	 	505		-»       // no sync to disk, tmpfs, truncate data on reload/restart - simpler 
[23:11:57] <Reedy>	 506		-»       // to reason about, and our ratelimits aren't long-term enough for 
[23:11:57] <Reedy>	 507		-»       // persistence across daemon restarts to matter much. 
[23:11:57] <Reedy>	 508		-»       tbf.open("/run/vmod_tbf/tbf.db", "mode=600;dbname=tbf.bdb;trunc");
[23:12:23] <Reedy>	 hm
[23:13:14] <Reedy>	 I'll just reboot the whole thing
[23:13:29] <Krenair>	 Reedy, got it.
[23:13:37] <Reedy>	 ?
[23:13:43] <Krenair>	 I took a look through deployment-cache-text04:/root/.bash_history and found the command they used
[23:13:47] <Krenair>	 Ran it on deployment-cache-mobile04
[23:14:01] <Reedy>	 what was it?
[23:14:03] <Krenair>	 #1447207753
[23:14:03] <Krenair>	 service  varnish restart
[23:14:03] <Krenair>	 #1447207761
[23:14:03] <Krenair>	 service  varnish-frontend restart
[23:14:07] <Reedy>	 ah, yes
[23:14:10] <Reedy>	 restart both
[23:14:11] <Reedy>	 duhh
[23:14:12] <Reedy>	 AndyRussG: fixed?
[23:14:31] <AndyRussG>	 Reedy: yep!!
[23:14:33] <AndyRussG>	 :)
[23:14:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802485 (10Krenair)
[23:14:44] <wikibugs>	 10Beta-Cluster-Infrastructure, 6operations, 7Blocked-on-Operations, 5Patch-For-Review: Varnish rate limiting has broken beta - https://phabricator.wikimedia.org/T118362#1802483 (10Krenair) 5Open>3Resolved ```<Krenair> I took a look through deployment-cache-text04:/root/.bash_history and found the comma...
[23:14:58] <Reedy>	 hahah
[23:15:15] <Reedy>	 I tried to do the exact same
[23:16:27] <Krenair>	 I had tried the first one already, but I had to find the second
[23:19:23] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802493 (10Jdforrester-WMF) @Krenair...
[23:22:14] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802504 (10Krenair) 5Open>3Resolv...
[23:22:40] <James_F>	 Krenair: Thanks!
[23:23:23] <Krenair>	 np
[23:23:29] * Krenair tries to stop getting distracted
[23:38:42] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish, 7Verified: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802581 (10Ryasmeen)
[23:53:58] <wikibugs>	 10Beta-Cluster-Infrastructure, 10VisualEditor, 6operations, 7Varnish, 7Verified: [Regression pre-wmf.7] Images for musical scores, formulæ, heiroglyphics, thumbnails are returning 429s in the Beta Cluster when using VE (and other times?) - https://phabricator.wikimedia.org/T118486#1802607 (10greg) Thanks...