[09:03:58] <wikibugs>	 10Scoring-platform-team, 10ORES, 10editquality-modeling, 10Collaboration-Team-Triage (Collab-Team-This-Quarter), and 4 others: Enable ORES filters for svwiki - https://phabricator.wikimedia.org/T174560#3962379 (10Sebastian_Berlin-WMSE)
[10:04:39] <wikibugs>	 10Scoring-platform-team, 10ORES, 10Operations: Clean up redundant ORES celery_workers defaults - https://phabricator.wikimedia.org/T186734#3962586 (10fgiunchedi) p:05Triage>03Low
[12:45:48] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3963176 (10akosiaris) >>! In T171851#3959677, @Halfak wrote: > I found this in our deploy repo.  >  > {P6677} >  > Not sure what is going on as th...
[13:35:30] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Patch-For-Review: Rebuild ORES wheels on Stretch - https://phabricator.wikimedia.org/T184135#3963324 (10akosiaris) >>! In T184135#3952838, @awight wrote: > @akosiaris Our Stretch patches are available on the `stretch_conversion` branch.  That includes Python 3.5 w...
[14:33:30] <awight>	 Fawlty wifi day
[14:33:30] <awight>	 I’m staring at myself in Hangouts
[14:36:28] <awight>	 It’s strange, websites work and hangouts sees you
[14:36:38] <awight>	 halfak: fyi ^
[14:37:10] <halfak>	 Weird.  I see you joining and leaving, but no audio
[14:42:02] <wikibugs>	 (03PS1) 10Zoranzoki21: Removed deprecated position statements from resource loader module [extensions/ORES] - 10https://gerrit.wikimedia.org/r/409900 (https://phabricator.wikimedia.org/T184257)
[14:46:52] <wikibugs>	 (03CR) 10Jayprakash12345: [C: 031] Removed deprecated position statements from resource loader module [extensions/ORES] - 10https://gerrit.wikimedia.org/r/409900 (https://phabricator.wikimedia.org/T184257) (owner: 10Zoranzoki21)
[15:55:03] <awight>	 halfak: whoa, Phabricator won’t open for me:
[15:55:04] <awight>	 Too many tasks (250).
[15:55:34] <awight>	 maybe it’s just a network thing
[16:04:25] <awight>	 yep.
[16:34:25] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3964104 (10awight) @akosiaris Do you know whether that group_size change will apply to rollback as well?  We want the rollback to be as fast as po...
[16:39:48] <wikibugs>	 10Scoring-platform-team (Current), 10ORES, 10Operations, 10Patch-For-Review: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851#3964154 (10akosiaris) Yes it will. Which is why I am experimenting already with `fetch_batch_size` [1]  [1] https://github.com/wikimedia/scap/blob...
[16:44:41] <wikibugs>	 (03Draft2) 10Alexandros Kosiaris: Remove the cluster server group and related stuff Add a new scb server group, to be removed soon Remove the ores-worker server group from production Move the production configs under [wmnet] stanza Add checks for the default group Pass --no-download to vi [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/409932 (https://phabricator.wikimedia.org/T171851)
[16:48:38] <awight>	 akosiaris: Just wondering, I have some time today to help with the ores* deployment.  Are you actively messing around with that?  Anything I can do without being in your way?
[16:49:36] <akosiaris>	 awight: An answer to https://phabricator.wikimedia.org/T184135#3963324 would be greatly appreciated
[16:49:49] <awight>	 on it
[16:49:49] <akosiaris>	 even if it's just a "go ahead with master"
[16:50:21] <awight>	 I think your analysis is correct—also, I’d like to leave master untouched until we’re fully migrated.
[16:50:23] <awight>	 I’m scared.
[16:50:34] <awight>	 Lemme prepare the missing patch.
[16:51:55] <akosiaris>	 ok thanks
[16:52:22] <halfak>	 awight, looks like wheels needs a bump in the stretch_conversion branch
[16:52:25] <halfak>	 Doing that now. 
[16:53:02] <wikibugs>	 (03PS3) 10Halfak: Bumps models and requirements for conversion to Debian Stretch. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/404886 (https://phabricator.wikimedia.org/T182799)
[16:53:17] <halfak>	 Was just sitting there waiting for the bump :|
[16:53:36] <halfak>	 Make sure to do "git submodule update --init" when testing out the deployment
[16:53:44] <awight>	 WAIT!
[16:53:48] <awight>	 DNM
[16:53:53] <awight>	 That needs to be on the new branch
[16:53:59] <halfak>	 Uh?
[16:54:00] <halfak>	 It is
[16:54:13] <awight>	 no, that’s the ”topic”
[16:54:17] <awight>	 it’ll merge to master.
[16:54:18] <halfak>	 It's part of stretch_conversion
[16:54:26] <awight>	 I’m picking over now
[16:54:37] * halfak is not planning to merge it. 
[16:54:41] <awight>	 grr, we have no branch yet, one moment
[16:54:42] <awight>	 :) tty
[16:54:46] <awight>	 *ty
[17:00:19] <awight>	 This is odd.  The branch stretch_conversion already has that commit.
[17:01:17] <awight>	 huh.  it’s a different version of the same patch.
[17:03:18] <wikibugs>	 (03Abandoned) 10Awight: Bumps models and requirements for conversion to Debian Stretch. [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/404886 (https://phabricator.wikimedia.org/T182799) (owner: 10Halfak)
[17:03:33] <wikibugs>	 (03PS1) 10Awight: Update wheels for Stretch [services/ores/deploy] (stretch_conversion) - 10https://gerrit.wikimedia.org/r/409950
[17:04:00] <awight>	 halfak: if you have a second to merge? ^
[17:04:31] <wikibugs>	 (03CR) 10Halfak: [V: 032 C: 032] Update wheels for Stretch [services/ores/deploy] (stretch_conversion) - 10https://gerrit.wikimedia.org/r/409950 (owner: 10Awight)
[17:04:37] <awight>	 akosiaris: ^ :)
[17:05:02] <akosiaris>	 ah nice
[17:05:19] <awight>	 Sorry to slow down your work by bluffing earlier ;-)
[17:05:23] <akosiaris>	 no worries
[17:05:40] <akosiaris>	 let me see if this will work
[17:05:45] <akosiaris>	 doing a test deploy on ores1001 now
[17:08:43] <wikibugs>	 (03PS3) 10Alexandros Kosiaris: Remove the cluster server group and related stuff [services/ores/deploy] - 10https://gerrit.wikimedia.org/r/409932 (https://phabricator.wikimedia.org/T171851)
[17:09:35] <awight>	 K I’ll space out on other stuff but my highest priority this week is to support the migration.
[17:10:21] <akosiaris>	 btw, the Timeout, server ores100X.eqiad.wmnet not responding messages are totally misleading
[17:10:47] <akosiaris>	 what's happening after all in the background is that tin reaches max children and refuses git clone, git fetch etc
[17:11:39] <akosiaris>	 I am not sure why that totally misleading messages gets printed though
[17:11:59] <awight>	 Cool, that’s vaguely what I had remembered.
[17:21:04] <akosiaris>	 wooohooo 
[17:21:06] <akosiaris>	 success
[17:21:09] <akosiaris>	 awight: halfak ^
[17:21:30] <akosiaris>	 ok ores1001 is now having both uwsgi-ores and celery-ores-worker working fine
[17:21:54] <akosiaris>	 it's not receiving currently requests but it would be scoring stuff already
[17:23:28] <awight>	 right on!
[17:23:48] <awight>	 I can run a wee stress test, if this is a good time?
[17:24:32] <akosiaris>	 yeah sure
[17:24:42] <akosiaris>	 remember it's only ores1001 currently up and running
[17:25:03] <akosiaris>	 I am starting deploys on all the others slowing trying to figure out the optimal value for fetch_batch_size
[17:25:05] <awight>	 ah okay well, I’ll just make internal queries.
[17:27:15] <awight>	 akosiaris: Responses are healthy.
[17:27:26] <akosiaris>	 :)
[17:59:43] <awight>	 halfak: https://etherpad.wikimedia.org/p/JADE_extension_implementation
[18:01:26] <akosiaris>	 all ores100* boxes are now successfully deployed
[18:01:40] <akosiaris>	 fetch_batch_size does what you 'd expect
[18:02:15] <akosiaris>	 takes quite a bit of time to fetch the git objects but at least the promote and check stages proceed way faster
[18:02:52] <akosiaris>	 but overall a deploy takes around 13m with a concurrency value of 1
[18:03:09] <akosiaris>	 I think I can bump this to 3 relatively harmlessly
[18:06:42] <awight>	 akosiaris: Thanks for the incremental fix, I appreciate that we aren’t blocked on something that might not be corrected :D
[18:07:12] <awight>	 And the new variable means that we have parallel rollback?
[18:07:29] <akosiaris>	 it should
[18:07:37] <akosiaris>	 note the *should*
[18:07:46] <akosiaris>	 I guess some testing will verify that
[18:13:44] <akosiaris>	 scores have dropped the in graphs in https://grafana.wikimedia.org/dashboard/db/ores?refresh=1m&orgId=1
[18:14:06] <akosiaris>	 expected .. since the new ores clusters are scoring stuff
[18:16:24] <awight>	 That’s great news.  Should we be so bold as to pool temporarily?
[18:22:01] <akosiaris>	 we can
[18:22:22] <akosiaris>	 let's pool a couple of hosts and see what's happening
[18:22:26] <akosiaris>	 halfak ^
[18:23:00] <halfak>	 In meeting.  Looks exciting.  Can't really make judgments about what is reasonable right now, but I don't see any concerns with giving it a quick test. 
[18:23:18] <akosiaris>	 just pooled ores1001
[18:23:59] <awight>	 Great, I’ll watch graphs and what logs I have access to.
[18:24:57] <awight>	 akosiaris: umm… jfyi it seems that all 18 hosts have been scoring since 17:35
[18:25:29] <awight>	 I think the celery config is pointing to eqiad’s Redis?
[18:25:41] <awight>	 err eqiad == scb1* sorry
[18:25:49] <awight>	 No harm done, it seems.
[18:26:37] <akosiaris>	 no, ores200* config points to codfw celery
[18:26:40] <awight>	 not just eqiad
[18:26:41] <awight>	 yep
[18:26:45] <awight>	 'w
[18:26:58] <awight>	 Well, that means the machines have been aggressively scoring :)
[18:28:09] <akosiaris>	 I am trying to see if fetch_batch_size=3 actually changes anything now
[18:28:14] <akosiaris>	 in codfw only that is
[18:33:45] <awight>	 akosiaris: Note for later, I think we can increase the celery worker count quite a bit.  Those machines are sitting with 36GB of free memory and <5% CPU usage.
[18:33:53] <awight>	 I forget why we chose such conservative numbers
[18:34:22] <awight>	 ah—in part this is because of my puppet change last week which dialed down worker count on existing machines due to OOM.
[18:34:39] <akosiaris>	 yup
[18:34:44] <akosiaris>	 and agreed ofc
[18:35:07] <akosiaris>	  (duration: 06m 47s) for a deploy with fetch_batch_size: 3
[18:36:12] <awight>	 Cool, that’s livable.
[18:36:15] <akosiaris>	 ok doing one final around with a dummy comment change in scap/scap.cfg and then a rollback test
[18:36:19] <awight>	 TY!
[18:39:13] <wikibugs>	 (03CR) 10WMDE-Fisch: [C: 032] Removed deprecated position statements from resource loader module [extensions/ORES] - 10https://gerrit.wikimedia.org/r/409900 (https://phabricator.wikimedia.org/T184257) (owner: 10Zoranzoki21)
[18:44:12] <wikibugs>	 (03Merged) 10jenkins-bot: Removed deprecated position statements from resource loader module [extensions/ORES] - 10https://gerrit.wikimedia.org/r/409900 (https://phabricator.wikimedia.org/T184257) (owner: 10Zoranzoki21)
[18:45:10] <wikibugs>	 (03CR) 10jenkins-bot: Removed deprecated position statements from resource loader module [extensions/ORES] - 10https://gerrit.wikimedia.org/r/409900 (https://phabricator.wikimedia.org/T184257) (owner: 10Zoranzoki21)
[18:50:05] <akosiaris>	 18:48:08 Finished deploy [ores/deploy@f7e23f4]: T171851 (duration: 12m 14s) for all 18 boxes
[18:50:05] <stashbot>	 T171851: Reimage ores* hosts with Debian Stretch - https://phabricator.wikimedia.org/T171851
[18:50:28] <akosiaris>	 rolling back now
[18:50:49] * awight wrings hands
[18:58:19] <akosiaris>	 didn't work
[18:58:25] <akosiaris>	 it actually failed ...
[18:58:29] <akosiaris>	 what on earth...
[18:59:49] <awight>	 Glad this was only a test!
[19:01:06] <akosiaris>	 so... fetch on an already fetched revision failed
[19:01:22] <akosiaris>	  /o\
[19:03:12] * akosiaris goes for dinner and bbl to debug this more
[19:04:38] <awight>	 haarrr
[19:04:53] <awight>	 Maybe this has to do with the new variable...
[19:05:01] * awight reclocates too
[20:07:30] <awight>	 halfak: These are both ready for re-review, https://github.com/adamwight/mw-ext-JADE/pulls
[20:23:49] <awight>	 halfak: https://snag.gy/oRfhTj.jpg
[20:23:57] <awight>	 Validation is solid.
[20:24:06] <halfak>	 OMG 
[20:24:08] <halfak>	 Nice
[20:24:58] <halfak>	 Just got out of mega meeting. 
[20:25:03] <halfak>	 I'm gonna eat a mega lunch
[20:25:07] <halfak>	 And then go to more mega meetings
[20:25:54] <halfak>	 Sorry awight.  Will need to review tomorrow. 
[20:26:09] <awight>	 yup see you in a minute
[20:26:20] <halfak>	 Should be AFK for 35 mins now. 
[20:59:29] <halfak>	 o/
[20:59:39] <halfak>	 Got back just in time
[21:21:19] <halfak>	 yay!  Welcome back ChanServ
[21:31:08] <awight>	 halfak: Tomorrow is great, but I wanted to point you to this: https://etherpad.wikimedia.org/p/JADE_extension_implementation
[22:55:31] <halfak>	 OMG MEETINGS. 
[22:55:37] <halfak>	 Just finished up. 
[22:55:41] * halfak clicks on etherpad
[23:00:50] <halfak>	 awight, I like it.  The order makes sense. 
[23:01:05] <awight>	 Cool, thanks for taking a look.