[06:19:23] <Amir1>	 legoktm: Thanks <3
[06:21:16] <legoktm>	 yep :D
[06:24:15] <Amir1>	 We should start discussion of migration of mailing lists btw
[06:24:29] <Amir1>	 legoktm: shall I make a ticket? I actually got some numbers
[06:25:39] <legoktm>	 yes please
[09:26:50] <godog>	 PSA that NSDI21 was on this week, https://www.usenix.org/conference/nsdi21/
[09:33:51] <XioNoX>	 "In this paper, we present Facebook's BGP-based data center routing design and how it marries data center's stringent requirements with BGP's functionality." interesting stuff, thanks!
[09:35:43] <elukey>	 https://www.usenix.org/conference/nsdi21/presentation/qian-zhengping is also interesting for the Analytics folks
[09:36:25] <elukey>	 and also https://www.usenix.org/conference/nsdi21/presentation/ghigoff looks super cool
[09:36:29] <elukey>	 _joe_ effie --^
[09:37:07] <XioNoX>	 🇫🇷
[09:37:27] <elukey>	 "Memcached, one of the most popular key-value stores, suffers from performance limitations inherent to the Linux networking stack and fails to achieve high performance when using high-speed network interfaces"
[09:37:48] <elukey>	 I am curious about what "high-speed" means in this context :D
[09:37:51] <effie>	 ah nice, I will read
[09:37:56] <effie>	 tx tx
[09:38:26] <godog>	 yeah the whole program is intense but I'm interested in papers and/or talks worth watching for sure
[09:43:45] <elukey>	 +1 thanks!
[10:07:57] <arturo>	 elukey: if you +1 https://gerrit.wikimedia.org/r/c/operations/homer/public/+/679862 I'm ready to merge it now! :-)
[10:13:54] <arturo>	 thanks XioNoX !
[10:13:54] <XioNoX>	 arturo: I'm merging something else right now, wait before merging yours please
[10:14:01] <arturo>	 ack
[10:39:22] <XioNoX>	 arturo: all good
[10:40:11] <arturo>	 ack
[10:40:20] <arturo>	 merging
[10:45:12] <arturo>	 done
[11:48:33] <Amir1>	 godog: I met the chair of that program in a conference in Berlin. He is awesome
[12:05:50] <godog>	 Amir1: eheh easy to believe! I love mr Mickens papers, always a fun pleasure to read, is that who you met?
[12:06:43] <Amir1>	 yup
[12:06:53] <godog>	 fantastic
[12:06:54] <Amir1>	 let me find his presentation
[12:08:55] <Amir1>	 godog: Found it https://www.oreilly.com/radar/my-love-letter-to-computer-science-is-very-short/
[12:10:47] <Amir1>	 ugh the full presentation is paywalled
[12:11:32] <ema>	 love letter to O'Reilly also not very long
[12:12:42] <godog>	 Amir1: nice, thank you for the link, did not disappoint
[12:14:02] <Amir1>	 ema: godog viva youtube https://www.youtube.com/watch?v=4vd2rCBjHp8
[12:15:24] <godog>	 genius
[15:00:39] <dcaro>	 godog: hi, maybe you know from the top of your head, I'm trying to get the current size of the WAL directory from the prometheus metrics (for the prometheus host tools-prometheus-03), do you know which metric is it? (if there's one)
[15:04:08] <godog>	 dcaro: hi! I'm not sure there's a metric no, couldn't find one from a quick skim of https://grafana.wikimedia.org/d/GWvEXWDZk/prometheus-server
[15:06:56] <dcaro>	 ack, thanks
[15:44:08] <ottomata>	 godog: have there been any recent changes to kafka logging eqiad?
[15:44:13] <ottomata>	 like, new brokers or something?
[15:45:22] <godog>	 ottomata: yeah new brokers this week, finding the task
[15:45:34] <godog>	 ottomata: https://phabricator.wikimedia.org/T279342
[15:45:53] <ottomata>	 ah
[15:45:59] <ottomata>	 the helm values were not updated! :o
[15:46:03] <ottomata>	 herron: yt?
[15:47:16] <ottomata>	 is kafka totally off of logstash1010-1012 now?
[15:48:51] <godog>	 I'm not sure, looking at puppet it seems so though
[15:49:38] <ottomata>	 lots of errors in eventgate-logging-external, which is why i looked at this
[15:49:45] <ottomata>	 but i don't know how it is producing any events at all if so
[15:49:46] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/680375
[15:50:01] <ottomata>	 i really wish we had these things DRYed up and matched with puppet
[15:50:10] <ottomata>	 https://phabricator.wikimedia.org/T253058
[15:50:10] <ottomata>	 :/
[15:50:36] <herron>	 ottomata: yup that's moved over to kafka-logging100[123] now
[15:50:48] <ottomata>	 herron:  ya, i guess there's no way for you to know
[15:50:54] <ottomata>	 but helmfiles can't/don't use puppet to get hostnames
[15:51:18] <ottomata>	 does this look right?
[15:51:19] <ottomata>	 https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/680375
[15:51:22] <ottomata>	 codfw changed over too?
[15:51:40] <herron>	 alright I'll add to the migration checklist so its covered in the future
[15:51:49] <herron>	 not yet, but it will be
[15:51:57] <ottomata>	 oh ok, won't change that yet then
[15:52:06] <dcaro>	 volans|off: jynus for the downtime thingie (though will have to change when we upgrade icinga) https://gerrit.wikimedia.org/r/c/operations/puppet/+/680376
[15:52:21] <jynus>	 he he
[15:53:45] <dcaro>	 xd... it does not like the commit message with the long lines... will fix, review is still ok though
[15:53:57] <jynus>	 one thing, not sure what is the expected outcome, but I think when run- it creates downtimes for both host and services, but only clears the host ones?
[15:55:07] <dcaro>	 afaik it clears all, but let me try (it's like a search, the second parametere, empty in the script, is the service name match, and the third the comment)
[15:55:54] <jynus>	 I have no idea what DEL_DOWNTIME_BY_HOST_NAME does
[15:56:06] <jynus>	 so you may be right
[15:56:24] <ottomata>	 herron:  just updated https://phabricator.wikimedia.org/T253058
[15:56:40] <dcaro>	 yep, works for all
[15:56:44] <ottomata>	 not really sure how to solve all that
[15:57:30] <jynus>	 dcaro, I would add a "and all its services" to the help, just to make it 100% clear
[15:57:52] <dcaro>	 ack, I'm logging off, can you add a comment? I'll fix on monday
[15:58:01] <jynus>	 sure, thank you for working on that
[15:58:07] <dcaro>	 👍
[16:01:41] <herron>	 ottomata: looks like you predicted the future!  yeah am not 100% sure either off hand, but good food for thought.  also happy to get together and talk about it if that would be any help
[16:04:19] <RhinosF1>	 _joe_: didn't someone do https://phabricator.wikimedia.org/T280377 yesteday
[16:04:34] <RhinosF1>	 See https://phabricator.wikimedia.org/T279804
[16:05:34] <_joe_>	 lol i searched "floc" on phab and nothing came uo
[16:05:38] <_joe_>	 *up
[16:06:14] <cdanis>	 ottomata: so was this effectively a temporary eventgate-logging-external outage?
[16:06:25] <RhinosF1>	 Phab search is weird _joe_ at times
[16:07:58] <ottomata>	 cdanis: partial?  yes?  afaict it is still producing most(?) events?
[16:08:10] <ottomata>	 https://grafana.wikimedia.org/goto/VCODT0XGz
[16:08:21] <ottomata>	 although i'm not totally sure how
[16:09:46] <cdanis>	 interesting
[16:09:54] <cdanis>	 there wasn't an obvious dropout period on the NEL data
[16:13:59] <ottomata>	 cdanis i do see some being dropped
[16:14:02] <ottomata>	 but not all
[16:14:23] <ottomata>	 https://logstash.wikimedia.org/goto/210c2ded15bcf1c386a62420de0f3f7c
[16:15:56] <ottomata>	 herron:  fyI https://gerrit.wikimedia.org/r/c/operations/deployment-charts/+/680380 was also needeed
[16:16:05] <ottomata>	 ok deployed egl with new broker settingsg
[16:17:06] <herron>	 ack, thanks noted