[07:29:06] <dcausse>	 re: full-reindex ideally we should get https://gerrit.wikimedia.org/r/c/mediawiki/extensions/CirrusSearch/+/1198219 in before and enable natural sort on titles, will update cindy with a new image 
[10:08:04] <pfischer>	 dcausse: Makes sense. Your MR is still marked WIP. Is cindy the only blocker or is there sth. else?
[10:08:56] <dcausse>	 pfischer: should be cindy the only blocker, updating at the moment and will remove the WIP as soon as cindy gives a green light
[14:14:50] <ebernhardson>	 \o
[14:15:16] <dcausse>	 o/
[14:45:05] <pfischer>	 o/
[15:23:39] <dcausse>	 errand
[15:55:32] <pfischer>	 According to https://en.wikipedia.org/wiki/Special:Version 1.46 is live. Can I move tickets under “To Be Deployed” with that tag to “Done” then?
[15:58:17] <Amir1>	 Hii, It seems the CI is failing for cookbook patches because of elasticsearch cookbooks: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1202150
[15:58:35] <Amir1>	 > E   ModuleNotFoundError: No module named 'elasticsearch'
[15:58:59] <pfischer>	 inflatador: ebernhardson: Was the plugin release successful? If so, would T406205 be done as well?
[15:58:59] <stashbot>	 T406205: Investigate and cleanup broken weighted_tags in cirrus indices - https://phabricator.wikimedia.org/T406205
[15:59:28] <gehel>	 ryankemper: are you around for the cookbook issue above? I'm about to jump into an interview. Or dcausse / pfischer : could you help unblock?
[15:59:46] <Amir1>	 Thanks!
[16:05:25] <ebernhardson>	 pfischer: the plugin fix should be released, it also requires a reindex to clean up the old data
[16:05:43] <ebernhardson>	 and then we can verify it in the following weekly hadoop dump
[16:06:27] <pfischer>	 ebernhardson: alright. thanks!
[16:07:46] <pfischer>	 ebernhardson: are you familiar with the cookbooks? dcausse is out at the moment, and I just ran a quick search to see if I can find any dependency declaration that was changed recently.
[16:08:18] <ebernhardson>	 pfischer: somewhat, although i can't run them. I have the code checked out and have writte na patch or two. Is there a ticket?
[16:09:20] <pfischer>	 ebernhardson: not that I am aware of, just what Amir1 posted: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1202150 (failing pipeline: https://integration.wikimedia.org/ci/job/tox/8111/console)
[16:10:14] <Amir1>	 Luca mentioned this T390860
[16:10:15] <stashbot>	 T390860: Elasticsearch dependency upgrade in spicerack - https://phabricator.wikimedia.org/T390860
[16:10:23] <Amir1>	 Maybe it's different
[16:10:43] <ebernhardson>	 hmm, pip deps are in setup.cfg, no significant changes recently. Curiously elasticsearch isn't even in git log
[16:12:52] <ebernhardson>	 yea not sure, the first patch to import from elasticsearch module is https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/902502 but it didn't declare any new dependencies
[16:15:14] <ebernhardson>	 can look a bit closer, but probably amounts to git bisect and re-running tests
[16:15:37] <ebernhardson>	 but thats very odd since the bot runs them every patch...
[16:16:53] <ebernhardson>	 also curious is that in the CI log py310-unit passes, but py309-unit fails. 
[16:19:36] <ebernhardson>	 something funny is going on there. I can locally run py310-unit, get a pass, but then open the interpreter tox used, import ban.py, and get an import fail
[16:21:13] <ebernhardson>	 found it, the tests explicitly disable in py 3.10+
[16:21:17] <ebernhardson>	 pfischer: ^
[16:22:09] <ebernhardson>	 so then the question is, why only 3.10+? and why did it used to pass in 3.9 but not now: https://gerrit.wikimedia.org/r/plugins/gitiles/operations/cookbooks/+/refs/heads/master/tests/unit/test_import.py#53
[16:22:23] * ebernhardson wishes more comments said "why" and not "what" :P
[16:23:01] <ebernhardson>	 maybe poke vol.ans? https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1143098
[16:25:31] <pfischer>	 ebernhardson: thanks, that’s indeed strange. The workaround is from May. Did the rename (change of naming convention for picking up tests) have an effect?
[16:26:58] <pfischer>	 Or is the pylint instruction no longer picked up? I have no clue… :-(
[16:27:28] <ebernhardson>	 i haven't looked in great detail, but i would be surprised if it does, this is gated on the package being in sre.elasticsearch, but this is sre.mysql
[16:28:48] <ebernhardson>	 the implied difference is that there was a system provided elasticsearch module available when running 3.9?  Maybe the docker image changed, looking
[16:29:10] <ebernhardson>	 at least, i'm assuming system provided because i didn't find a dependency in the repo that would install it
[16:30:24] <ebernhardson>	 hmm, no. a recent pass and this test both ran on docker-registry.wikimedia.org/releng/tox-v3:0.3
[16:31:13] <ebernhardson>	 buy py39 installed elasticsearch==7.14.2 in the passing one, our failing test did not...so back to the beginning, where is that dep coming from
[16:31:52] <gehel>	 we *should* not have a dependency to elastic anymore, but I have checked that this is actually the case.
[16:32:07] <ebernhardson>	 gehel: the ban notebook imports elasticsearch
[16:32:29] <ebernhardson>	 i guess i didn't look at how it was used though
[16:42:09] <ebernhardson>	 pfsicher: the problem is a dependency change in spicerack. Passing test used spicerack 11.10.0, fail uses 12.0.0.  Changelog for 12 includes removing elasticsearch
[16:42:58] <ebernhardson>	 or more directly, the problem is importing modules that are transitively dependend on, but it does make it easier in some cases
[17:10:14] <ebernhardson>	 ryankemper: I guess the question is for you or brian if the es node ban module can be disabled: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1202150
[17:10:54] <dcausse>	 shouldn't be too hard to drop the client from there, it's just 3 api calls...
[17:12:32] <dcausse>	 but yes not sure that the ban cookbook is something used that often and perhaps dropping it temporarily is acceptable
[17:13:04] <ebernhardson>	 yea i could probably just try and work something up today to unblock that, but not sure how needed it is
[17:13:19] <dcausse>	 true
[17:14:56] <dcausse>	 and worst case you can still run cookbooks from your own copy of the cookbook repo (IIRC that's how they test new unmerged cookbooks)?
[17:36:31] <dcausse>	 heading out, have a nice week-end
[17:48:59] <pfischer>	 Amir1: is this critical/blocking?
[17:49:21] <ebernhardson>	 it's not just amir, it's the entire cookbooks repo. nothing passes ci
[17:49:36] <ebernhardson>	 i have a fix half done, will push in a bit. Testing now
[17:49:50] <inflatador>	 <o/
[17:50:39] <inflatador>	 Sorry, I was out in my morning
[17:50:51] <ebernhardson>	 \o. no worries
[17:51:19] <inflatador>	 Amir1 the ban cookbook is not critical, we have an Ansible playbook that can do it https://gitlab.wikimedia.org/repos/search-platform/sre/ansible-playbooks/cirrussearch/-/blob/main/cirrussearch_ban.yml?ref_type=heads
[17:51:51] <ebernhardson>	 should we just drop the module then to unblock cookbooks?
[17:51:58] <ebernhardson>	 thats even easier to test :P
[17:53:22] <inflatador>	 yeah, we should just drop it. Sorry, I haven't been engaged on the cookbook stuff, I know Ryan and Luca made a lot of progress
[17:53:35] <Amir1>	 My patch per se is not important whatsoever but I think blocking merge of all cookbook patches can be really problematic 
[17:54:19] <inflatador>	 yeah, it's a bit of a sore spot for me since we did what we were asked (move logic into spicerack and out of the cookbook itself) and that's led to a ton of work
[17:54:26] <inflatador>	 plus blocking other teams
[17:54:54] <inflatador>	 anyway, I'm happy to write up a patch completely removing that cookbook if it will help
[17:55:13] <ebernhardson>	 yea it's probably important to unblock ci on the cookbooks
[17:55:55] <inflatador>	 ACK, will get started on that
[18:03:10] <inflatador>	 ebernhardson Amir1 patch up here if y'all wanna take a look: https://gerrit.wikimedia.org/r/c/operations/cookbooks/+/1205199
[18:14:56] <inflatador>	 Thanks for the +1, it has been merged
[18:16:51] <Amir1>	 just noting that +2 on the patch is enough, you don't need to force submit it. That's only for puppet
[18:17:44] <inflatador>	 ACK, it's merged now. LMK if that doesn't unblock you
[18:17:48] <jclark-ctr>	 T409938  If anyone is available from data platform Monday or Before. The Replacement drives arrived  at Eqiad 
[18:17:49] <stashbot>	 T409938: Degraded RAID on an-worker1208 - https://phabricator.wikimedia.org/T409938
[20:50:43] <ryankemper>	 Catching up on the elastic dependency stuff
[20:52:56] <ryankemper>	 Ah patch already merged, great
[21:01:04] <ryankemper>	 Restoring the actual cookbook shouldn’t be too difficult
[21:01:23] <ryankemper>	 jclark-ctr: yeah I’m around monday, what time are you envisioning?
[21:03:07] <jclark-ctr>	 I am usually on site around noon utc time
[21:12:03] <jclark-ctr>	 Usually Ben takes them out in advance. Thats really early for you and it’s not that urgent if  someone else from  data platform can help might be better thanks for offering Ryan
[22:04:57] <ryankemper>	 inflatador: in https://meet.google.com/qtf-yoqt-tmu if you'd like to sync
[22:27:30] <inflatador>	 ryankemper sorry, forgot I had to pick up Simon today. Let's wait 'till next week
[22:27:53] <ryankemper>	 inflatador: sounds good!
[22:29:28] <inflatador>	 cool, see ya Monday