[07:00:23] <wikibugs>	 Analytics-Cluster, Analytics-Kanban, Operations, ops-eqiad: analytics1049.eqiad.wmnet disk failure - https://phabricator.wikimedia.org/T137273#2408707 (elukey) @Cmjohnson the server is showing another disk failure, it seems that we are not lucky with this one:  ``` [1600281.300136] EXT4-fs error...
[07:01:24] <elukey>	 analytics1049 has another failed disk :/
[07:08:50] <elukey>	 stopped it completely to remove it from the cluster, the phab task has been updated :)
[08:38:26] <joal>	 Hi elukey
[08:38:32] <joal>	 Thanks for caring the disks
[08:38:53] <joal>	 a-team: Due to disk failure, some oozie jobs have failed yesterday, everything has been relaunched
[08:39:07] <elukey>	 thanks!
[08:39:38] <joal>	 elukey: any news on network for cassandra?
[08:41:45] <elukey>	 nope, need to chat with Alex
[08:41:56] <joal>	 k
[08:41:58] <joal>	 :)
[08:42:00] <joal>	 Thnaks
[09:29:16] <wikibugs>	 Analytics: Varnishkafka should auto-reconnect to abandoned VSM - https://phabricator.wikimedia.org/T138747#2408827 (elukey)
[09:33:03] <wikibugs>	 Analytics: Varnishkafka should auto-reconnect to abandoned VSM - https://phabricator.wikimedia.org/T138747#2408842 (elukey)
[09:57:45] <wikibugs>	 Analytics: Evaluate alternatives to varnishkafka: varnishevents - https://phabricator.wikimedia.org/T138426#2408902 (elukey)
[10:52:33] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations: Deployment of Gerrit Delays panel for engineering - https://phabricator.wikimedia.org/T138752#2408971 (Lcanasdiaz)
[10:57:13] <elukey>	 joal: https://phabricator.wikimedia.org/T138609#2408949 - you are free to test :)
[10:58:02] <elukey>	 elukey@analytics1030:~$ telnet aqs1006.eqiad.wmnet 9160
[10:58:02] <elukey>	 Trying 10.64.48.146...
[10:58:03] <elukey>	 telnet: Unable to connect to remote host: Connection refused
[10:58:33] <elukey>	 that is better elukey@analytics1030:~$ telnet aqs1006-a.eqiad.wmnet 9160
[10:58:36] <elukey>	 Trying 10.64.48.148...
[10:58:38] <elukey>	 \o/
[10:58:40] <elukey>	 Connected to aqs1006-a.eqiad.wmnet.
[10:58:42] <joal>	 YAY !
[10:58:46] <joal>	 Thanks elukey :)
[10:58:49] <joal>	 Testing again !
[11:00:56] <wikibugs>	 Analytics: Varnishkafka should auto-reconnect to abandoned VSM - https://phabricator.wikimedia.org/T138747#2408992 (ema) p:Triage>Normal
[11:01:44] <wikibugs>	 Analytics: Evaluate alternatives to varnishkafka: varnishevents - https://phabricator.wikimedia.org/T138426#2408994 (ema) p:Triage>Normal
[11:03:00] <joal>	 Wow elukey, huge drop in bytes from aqs100[123] since 20mins
[11:03:04] <joal>	 elukey: any idea?
[11:04:11] <joal>	 arf, my bad elukey, was because of loading not happening at expe3cted time
[11:04:16] <joal>	 elukey: sorry for false alarm
[11:05:21] <elukey>	 ah yes I was wondering looking at https://grafana.wikimedia.org/dashboard/db/aqs-cassandra-system
[11:05:24] <elukey>	 goooood
[11:05:43] <elukey>	 joal: https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings#Proposal
[11:05:56] <elukey>	 I am writing this document to recap everything
[11:06:05] <elukey>	 than I'll send it to ops for review
[11:06:05] <joal>	 elukey: looks like I still have issues:(
[11:06:10] <elukey>	 should be completed today
[11:06:33] <joal>	 awesome :)
[11:06:45] <elukey>	 joal: expected, whatever we do together can't be resolved in less than a week :P
[11:06:54] <joal>	 huhu
[11:07:07] <elukey>	 do we get a different error now?
[11:07:10] <elukey>	 or is it the same?
[11:07:15] <joal>	 no error yet
[11:07:19] <joal>	 waiting
[11:07:42] <elukey>	 ah sorry I thought "looks like I still have issues:(" was related to AQS loading
[11:08:07] <joal>	 yes it is, but error takes some time to show up (looks like similar to previous
[11:09:41] <joal>	 actually elukey yes, error is different: connection timed out
[11:23:09] <elukey>	 ah snap
[11:23:26] <joal>	 elukey: investigating more
[11:23:28] <elukey>	 theoretically the connection should work, timeout is a bit weird
[11:23:30] <elukey>	 checking logs
[11:23:39] <joal>	 great
[11:23:40] <elukey>	 is there a specific host ?
[11:23:47] <elukey>	 or instance
[11:23:55] <joal>	 elukey: It will inform me if I know that connection happenend :)
[11:23:57] <joal>	 elukey: any
[11:24:26] <joal>	 aqs1005-a as my first example, but each had the same error
[11:25:56] <elukey>	 could it be something client specific?
[11:26:05] <joal>	 That's what I'm after
[11:26:23] <elukey>	 super
[11:28:49] <elukey>	 joal: maybe write_request_timeout_in_ms=2000 ?
[11:30:52] <elukey>	 mm maybe not
[11:31:05] <elukey>	 didn't find any particular error msg in the logs
[11:31:23] <joal>	 elukey: trying again with another spec
[11:34:12] <elukey>	 not related, but from 2.1 datastax docs
[11:34:13] <elukey>	 The compaction strategy DateTieredCompactionStrategy precludes using read repair, because of the way timestamps are checked for DTCS compaction. In this case, you must set read_repair_chance to zero. For other compaction strategies, read repair should be enabled with a read_repair_chance value of 0.2 being typical.
[11:34:36] <elukey>	 https://docs.datastax.com/en/cassandra/2.1/cassandra/operations/opsRepairNodesReadRepair.html
[11:35:15] <wikibugs>	 Analytics, Revision-Slider, TCB-Team, WMDE-Analytics-Engineering, and 2 others: Data need:  Explore range of article revision comparisons  - https://phabricator.wikimedia.org/T134861#2409042 (WMDE-leszek)
[12:00:27] <joal>	 Maaaaaan
[12:01:10] <joal>	 elukey: I need to apologize, seems that 9160 is not the only one needed
[12:01:43] <joal>	 elukey: storage port is another one :(
[12:01:45] <joal>	 7000
[12:06:01] <mforns>	 hi team!!
[12:06:08] <joal>	 Hey mforns !
[12:06:24] <mforns>	 how are you doing guys :]
[12:06:39] <joal>	 not bad !
[12:06:41] <joal>	 you?
[12:06:50] <mforns>	 good :]
[12:07:43] * mforns catches up with email
[12:19:59] <elukey>	 mforns: helloooo
[12:20:05] <mforns>	 hi!
[12:20:06] <elukey>	 how are you?
[12:20:09] <mforns>	 goood :]
[12:20:11] <mforns>	 you?
[12:20:15] <elukey>	 goooood
[12:20:23] <elukey>	 joal: ouch also 7000
[12:20:25] <mforns>	 do you guys have news from madhuvishy?
[12:21:01] <joal>	 mforns: last news she had a passpotrt
[12:21:13] <mforns>	 aha I saw that, cool
[12:21:25] <joal>	 elukey: Actually doc is very obscure on which ports for what
[12:21:48] <joal>	 But from the code, looks like 7000 is used :(
[12:22:06] <joal>	 and I can't find 9160 used in code
[12:22:22] <joal>	 So, looks like I shouldn't have trusted docs :(
[12:22:26] <joal>	 elukey: --^
[12:22:48] * joal is tired of cassandra fight - Hopefully cassandra is even more
[12:23:29] <moritzm>	 joal: we can have a looked at dropped packets for a query if you want to have a definitive answer
[12:23:33] <moritzm>	 joal: we can have a look at dropped packets for a query if you want to have a definitive answer
[12:23:56] <joal>	 moritzm: that's be great
[12:24:18] <joal>	 moritzm: only issue is that it comes from hadoop, so changing hosts
[12:24:41] <joal>	 But I can tell you once the job has started
[12:26:00] <joal>	 moritzm: would that work?
[12:31:20] <moritzm>	 sure, but the logging rules need to be setup on the receiving hosts, if that one is non-volatile, that's no problem
[12:31:51] <elukey>	 the weird thing is that port 7000 seems used for internode communications https://docs.datastax.com/en/cassandra/2.1/cassandra/security/secureFireWall_r.html
[12:31:56] <joal>	 moritzm: issue is that there seem to be ACLs in the middle as well
[12:32:04] <joal>	 elukey: I know !!!
[12:32:37] <joal>	 elukey: I promise it's the last time I deal with a goddess !
[12:33:01] <elukey>	 maybe it is a trick to let the node believe that the sstables are coming from the cluster
[12:33:34] <elukey>	 I am interested in the logging rules too
[12:33:44] <elukey>	 (I guess that those will be iptables one right?)
[12:33:45] <joal>	 elukey: possible
[12:33:54] <moritzm>	 which hosts are we talking about here?
[12:34:02] <moritzm>	 port 7000 is already allowed in cassandra
[12:34:08] <joal>	 aqs100[456]-[a]
[12:34:16] <joal>	 aqs100[456]-[ab] sorry
[12:34:28] <moritzm>	 let me have a look, I think I know the problem
[12:34:34] <joal>	 moritzm: woth data coming from analytics network?
[12:38:25] <elukey>	 it is traffic from analytics* (hadoop) to cassandra port 7000
[12:38:32] <elukey>	 not sure if allowed by net ACL
[12:38:39] <elukey>	 and also by ferm
[12:38:43] <moritzm>	 so, in role::aqs port 7000 is allowed for cassandra_hosts_ferm
[12:38:51] <moritzm>	 which is generated from Hiera:
[12:38:55] <moritzm>	 cassandra::seeds
[12:39:48] <moritzm>	 but cassandra::seeds for aqs only contains the names of the actual aqs hosts
[12:39:50] <joal>	 moritzm: can't connect from an analytics machine
[12:40:11] <moritzm>	 but not the IP addresses/hostnames assigned to the -a and -b instances
[12:40:27] <moritzm>	 so this might be the problem
[12:40:40] <joal>	 k
[12:41:02] <moritzm>	 can we have a specific example, from where does the query originate and which hosts did it reach?
[12:42:35] <joal>	 moritzm: I tried to telnet from analytics1048 to aqs1004-a on port 7000
[12:50:25] <moritzm>	 so that won't work with the current rules, access to 7000 on aqs1004 is only allowed from the IP addresses of the Cassandra instances, not from anywhere else
[12:50:40] <moritzm>	 so aqs100[4-6]-[ab] at the moment
[12:51:05] <moritzm>	 which software package instead of telnet would access port 7000 from the hadoop cluster?
[12:54:49] <moritzm>	 port 7000 is documented for inter-node communication only, though, so I'm wondering whether we actually need this?
[12:55:31] <joal>	 moritzm: I'm trying to bulk load SStables into cassandra
[12:55:47] <elukey>	 same doubts that I have
[12:55:53] <elukey>	 feels wrong to use 7000
[12:56:11] <elukey>	 but I am sure that it is something to trick the nodes to accept sstables
[12:56:37] <elukey>	 urandom: Hi! Any thoughts about port 7000 used for SSTables bulk load?
[12:59:48] <elukey>	 (brb)
[13:17:06] <wikibugs>	 Analytics-Tech-community-metrics: Mediawiki support to be added to GrimoireLab - https://phabricator.wikimedia.org/T138007#2409146 (Lcanasdiaz) The Mediawiki support for Perceval is being finished this week.
[13:18:09] <wikibugs>	 Analytics-Tech-community-metrics, Developer-Relations: Deployment of Demography panel - https://phabricator.wikimedia.org/T138757#2409147 (Lcanasdiaz)
[13:24:58] <madhuvishy>	 Hey mforns I got my passport this morning. Just got US visa approved. They said they'll call tomorrow morning and I can go to the embassy and get it
[13:29:12] <elukey>	 madhuvishy: \o/
[13:31:50] <mforns>	 madhuvishy, cooooool...
[13:31:54] <mforns>	 finally
[13:32:45] <mforns>	 did you make it to Esino Lario?
[13:43:57] <madhuvishy>	 mforns: yeah! :) I did
[13:44:01] <madhuvishy>	 On wednesday
[13:44:07] <madhuvishy>	 Came to milan this morning
[13:44:38] <madhuvishy>	 elukey: :)
[14:22:03] <elukey>	 a-team: first draft of what we discussed in Berlin - https://wikitech.wikimedia.org/wiki/User:Elukey/Ops/AQS_Settings
[14:22:15] <elukey>	 probably still full of typos and things to add
[14:22:19] <elukey>	 early comments are welcome :)
[14:22:59] <mforns>	 elukey, at first glance looks amazing
[14:25:43] <elukey>	 thanksss!! I still need how to place those images since I am a wikitext newbie
[14:33:00] <urandom>	 elukey: context?
[14:33:09] <joal>	 Hi urandom
[14:33:14] <urandom>	 hi!
[14:33:34] <joal>	 I'm having trouble bulk feeding cassandra
[14:33:39] * urandom is reading the backlog
[14:33:48] <joal>	 ports to be used is one of the issue
[14:36:28] <elukey>	 urandom: o/
[14:38:28] <urandom>	 the bulk loading uses streaming
[14:38:39] <urandom>	 (sorry my internets is janky)
[14:39:06] <urandom>	 but yeah, 7000 sounds right
[14:39:16] <joal>	 mwarf
[14:39:24] <urandom>	 is this bad?
[14:39:35] * joal apologizes to elukey for having noticed the wrong port :(
[14:40:30] <joal>	 urandom: doc on streaming is not really proficient, so I've been fighting a bit on how it works, and asked elukey (and ops) to open 9160
[14:41:00] <urandom>	 mmm
[14:41:03] <joal>	 urandom: It's just about me asking too many things, nothing really wrong
[14:41:22] <urandom>	 bulk loading is an advanced subject i guess
[14:41:31] <elukey>	 urandom: we allow traffic coming from the other nodes of the AQS cluster to port 700
[14:41:34] <elukey>	 *7000
[14:41:46] <elukey>	 meanwhile we should allow traffic from hadoop to enable this
[14:41:53] <elukey>	 not a big deal but it felt weird
[14:41:54] <elukey>	 :)
[14:42:12] <joal>	  you're right urandom, and all the doc is about using sstableloader tool, not the java classes
[14:42:28] <joal>	 Thanks for the confirmation urandom :)
[14:42:42] <urandom>	 the hadoop node(s) streamig data are technically nodes for the duration of the stream
[14:42:51] <urandom>	 Cassandra nodes that is
[14:43:11] <urandom>	 from a messaging perspective anyway
[14:43:12] <joal>	 elukey: Do you think it's possible to get port 7000 openned?
[14:43:16] <moritzm>	 is that bulk loading one time thing or an ongoing task?
[14:43:19] <joal>	 makes sense urandom
[14:43:44] <joal>	 moritzm: currently one shots, we are learning, but in the end if it works as exepected, could become ongoing
[14:44:59] <moritzm>	 will that require access from any hadoop node or will that happen from a designated one?
[14:45:48] <joal>	 moritzm: any, as always with hadoop, we can't know which of the nodes will be designated to execute the task
[14:46:01] <moritzm>	 ok
[15:37:13] <joal>	 elukey, moritzm: sorry to bother again, but I'm not sure of what I can expect from you guys :)
[15:37:44] <joal>	 Are you opening the 7000 ports? if yes, when?
[15:41:35] <elukey>	 One thing that it was proposed before was to create a hiera config for the hadoop nodes only, to allow only those ones rather than the whole analytics network in ferm
[15:41:38] <elukey>	 might be an option
[15:44:53] <joal>	 Would work for me elukey :)
[15:45:38] <elukey>	 I think we should wait for moritzm advice since we might risk to keep opening ports to achieve a goal opening big security holes :)
[15:46:00] <joal>	 ok elukey :)
[15:46:15] <moritzm>	 elukey: yeah, that sounds good, but we need it tweakable via Hiera so that it only applies to the 7000 port of aqs, not to cassandra as used in restbase
[15:46:17] <joal>	 I'll be completely fine to close those holes once we find the working ones !
[15:48:10] <elukey>	 moritzm: would something like https://gerrit.wikimedia.org/r/#/c/295907/3/manifests/role/aqs.pp be ok? Maybe not with the whole analytics network but only the hadoop nodes
[15:56:08] <moritzm>	 yeah, let's rather limit to the hadoop nodes
[16:33:29] <elukey>	 all right, going offline team!
[16:33:33] <elukey>	 see you tomorroW1
[16:33:35] <elukey>	 w!
[16:33:36] <elukey>	 :)
[16:47:28] <joal>	 Bye elukey
[19:55:19] <mforns>	 bye a-team!!
[19:55:26] <joal>	 Good night mforns
[19:55:36] <mforns>	 night joal1
[19:55:38] <mforns>	 !
[21:32:16] <grrrit-wm>	 (PS5) Addshore: Add WikidataArticlePlaceholderMetrics [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295896 (https://phabricator.wikimedia.org/T138500)
[21:33:23] <grrrit-wm>	 (CR) Addshore: "@joal So AFAIK this is now correct." [analytics/refinery/source] - https://gerrit.wikimedia.org/r/295896 (https://phabricator.wikimedia.org/T138500) (owner: Addshore)