[00:06:27] 10Analytics, 10Analytics-Kanban, 10good first task: [reportupdater] Allow defaults for all config parameters - https://phabricator.wikimedia.org/T193171 (10Nuria) Thank you @paulkernfeld , let us know if you are interested on another task, you can ping us in #wikimedia-analytics on irc an example of one (i... [01:05:13] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10jwang) I have published the report to meta wiki. Feel free to comment. Link: https://meta.wikimedia.org/wiki/Research:Emerging_Technic... [06:17:31] 10Analytics: Ensure Puppet checks types as part of the build - https://phabricator.wikimedia.org/T261693 (10elukey) >>! In T261693#6428636, @Ottomata wrote: > Ya, PCC is good, but Razzi created this ticket hoping that an obvious incorrect type like this could be caught by the Jenkins tests that run with every pa... [06:18:03] good morning [06:24:48] 10Analytics: Ensure Puppet checks types as part of the build - https://phabricator.wikimedia.org/T261693 (10razzi) Yes, in my opinion there should be a nonzero exit code of the type checker, and that should propagate to the Jenkins job. [07:18:55] very interesting reading https://www.kernel.org/doc/Documentation/lockup-watchdogs.txt [07:19:15] we have a recurrent issue about soft cpu locks ups for hadoop workers [07:20:07] once every now and then one worker gets into a state that causes alarms (too busy to respond to icinga pings) and the mgmt serial console shows soft lockups in the tty with the login [07:20:28] (root login via serial console is not available, too slow even to create a session) [07:20:58] in this case, 1059 was reported as "down" from icinga [07:20:58] https://grafana.wikimedia.org/d/000000377/host-overview?orgId=1&refresh=5m&var-server=analytics1059&var-datasource=thanos&var-cluster=analytics&from=now-24h&to=now [07:21:26] and I had to powercycle the host to restore its functionality [07:22:59] (the watchdog is supposed to update a timestamp inside the kernel periodically, if it doesn't do it for 20s then a stall is detected) [07:24:57] this might be something that goes away with the migration to Bigtop + buster probably [07:38:28] -- [07:38:40] I am going to reimage jumbo1003 to buster [07:44:02] 10Analytics-Clusters: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['kafka-jumbo1003.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-reimage/202009020743_el... [07:49:56] so I wanted to also start the roll restart of hadoop for openjdk upgrades, buuut I see that sqoop is running so not a great idea :D [08:18:16] 10Analytics-Clusters, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1003.eqiad.wmnet'] ` and were **ALL** successful. [08:37:13] !log run kafka preferred-replica-election on jumbo after jumbo1003's reimage to buster [08:37:15] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [08:37:43] ok jumbo1004 is the only broker left to upgrade to buster [08:45:08] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10elukey) Today I tried to add the ` --header "X-Forwarded-For: XX.XX.XX.XX"` using my external ipv4 address, and I was able to get a short url from an-tool1005. I guess that we could think about us... [09:04:45] 10Analytics-Clusters, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by elukey on cumin1001.eqiad.wmnet for hosts: ` ['kafka-jumbo1004.eqiad.wmnet'] ` The log can be found in `/var/log/wmf-auto-r... [09:38:44] 10Analytics-Clusters, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['kafka-jumbo1004.eqiad.wmnet'] ` and were **ALL** successful. [09:41:47] all jumbo nodes on busteR! [09:43:10] 10Analytics-Clusters, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10elukey) https://gerrit.wikimedia.org/r/623742 didn't really solve the problem, but I wouldn't spend more time into this since all the Jumbo nodes are on buster now. Calling this task... [09:43:34] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10elukey) [09:43:53] 10Analytics-Clusters, 10Analytics-Kanban, 10Patch-For-Review: Upgrade Kafka Brokers to Debian Buster - https://phabricator.wikimedia.org/T255123 (10elukey) p:05Triage→03Medium a:03elukey [09:44:03] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [09:45:02] 10Analytics, 10Analytics-Kanban: Move the Analytics infrastructure to Debian Buster - https://phabricator.wikimedia.org/T234629 (10elukey) [10:09:09] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Joe) [10:09:34] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Joe) p:05Triage→03Unbreak! Setting priority to UBN! given the seriousness of the perf regression. [10:14:27] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Joe) It looks like kafka2003 is the culprit - its broker latencies are in the order of 1 seconds. [10:28:21] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Joe) So this is probably due to all the purges going through the codfw kafka2003 server, and that we still haven't partitioned the purge topic. In normal conditions, the pur... [10:45:16] * elukey lunch! [11:49:17] \o/ kaf-ster! [13:24:51] nice [13:24:53] heya [13:31:23] 10Analytics, 10Analytics-Kanban: Undo any temporary changes made while running in codfw - https://phabricator.wikimedia.org/T261865 (10Milimetric) [13:31:49] 10Analytics, 10Analytics-Kanban: Undo any temporary changes made while running in codfw - https://phabricator.wikimedia.org/T261865 (10Milimetric) p:05Triage→03High [13:34:21] elukey / ottomata: yall see the eventgate unbreak now? [13:34:33] https://phabricator.wikimedia.org/T261846 [13:34:47] we are discussing in #wikimedia-sre :) [13:35:42] ah! What?! I'm on wikimedia-operations like a loser and there's a new channel? [13:36:19] k, glad yall on it [13:36:28] ahahah nono we follow it as well, it is just too much noise with icigna [13:36:31] *icinga [13:36:36] so we created a new fancy chan [13:53:53] razzi: klausman there is an occasional Product Analytics + Analytics Engineering sync up meeting happening in 1h, then we have our weekly analytics ops sync meeting in 1.5h (thats just us 4 analytics eng SREs). [13:54:01] you should both be invited to the analytics eng ops sync [13:54:13] lemme know if you'd like to be invited to the PA + analytics eng one. [13:54:32] Product Analytics is a team of data scientists and analysts in the product dept. that use our infra [13:56:48] so both are in the ops sync meeting invite, but not in the PA one afaics [13:56:57] Correct [13:58:44] ya, i guess i'll add yall as optional invites [13:59:18] already done ottomata [13:59:27] klausman: can you check? [13:59:36] oh :p [13:59:43] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Goal, 10Services (watching): Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10mpopov) [13:59:51] 10Analytics, 10Analytics-Kanban, 10Event-Platform, 10Product-Infrastructure-Data, and 2 others: Streams with empty configs should be rendered as {} in the JSON returned by StreamConfig API - https://phabricator.wikimedia.org/T259917 (10mpopov) 05Open→03Resolved a:05fdans→03Mholloway Thanks, Michael... [14:01:04] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Goal, 10Services (watching): Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10Ottomata) @nuria, I think the work for this can be considered done. Should we close this parent task? [14:01:09] Yup, I see a ⇄ meeting overlapping with the ops sync [14:06:36] yeah, will go to the PA one and then leave early for ops sync [14:11:50] I think I'll give it a go, just see what it's like [14:17:29] 10Analytics, 10Analytics-EventLogging, 10Event-Platform, 10Goal, 10Services (watching): Modern Event Platform: Stream Configuration - https://phabricator.wikimedia.org/T205319 (10Nuria) 05Open→03Resolved [14:17:31] 10Analytics-EventLogging, 10Analytics-Kanban, 10Event-Platform, 10Goal, and 3 others: Modern Event Platform (TEC2) - https://phabricator.wikimedia.org/T185233 (10Nuria) [14:29:24] razzi: I'm going to deploy today, after standup, do you want to tag along? [14:49:42] ottomata: doesn't this catch the "Any unexpected error" mentioned in the comment just below? https://gerrit.wikimedia.org/g/mediawiki/services/eventstreams/deploy/+/dbc9bbbe7355b844a8c7e4455f0b1e5b5f45053f/node_modules/kafka-sse/lib/KafkaSSE.js#666 [14:50:49] Starting build #3 for job wikimedia-event-utilities-maven-release-docker [14:51:50] Project wikimedia-event-utilities-maven-release-docker build #3: 09SUCCESS in 1 min 1 sec: https://integration.wikimedia.org/ci/job/wikimedia-event-utilities-maven-release-docker/3/ [14:55:23] milimetric: i have some stuff i want to merge before deploy! [14:55:31] trying to get it in , but meetings are starting soon! [14:56:13] ottomata: no prob, I can wait [15:01:10] (03PS7) 10Ottomata: Use EventSchemaLoader and EventLoggingSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) [15:07:35] (03CR) 10Ottomata: [C: 03+2] Use EventSchemaLoader and EventLoggingSchemaLoader from org.wikimedia.eventutilities [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/622369 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [15:12:48] klausman: man sorry your intro to meetings at WMF so far is "how should we have meetings" [15:12:53] they aren't all like this! [15:14:06] milimetric: Yeah, it'd be great to follow the deploy [15:17:23] Well, the great thing about that is that I can always claim I know noooothing, since, well, I know nothing on the topic ;) [15:27:36] ottomata: did you have this locally?? https://gerrit.wikimedia.org/r/c/schemas/event/primary/+/623813 [15:27:45] it fixes the validation of examples I reported [15:37:07] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Joe) 05Open→03Resolved a:03Joe We added two additional partitions to resource_purge, and this seems to have solved the issue, mostly. [15:56:09] ottomata, klausman , razzi , elukey : can we move the SRE ops sync 30 mins earlier and we will have the sync up with PA 30 min later? [15:57:02] Like, starting next week, do Ops, PA, Standup, 30m each, starting at 17:00? [15:58:53] klausman: teh PA meeting is only every 2 weeks [15:59:09] Ah, right. So a 30m gap? [15:59:17] every other week ya [15:59:32] wfm, though I am usually not productive in such slots. [15:59:34] ottomata, klausman: is ops standup weekly? [15:59:45] klausman: ok, so more compression would be best [15:59:53] (I'll brt for standup, just a minute over) [15:59:53] klausman: compression of meetings that is [15:59:56] Ops Sync up is weekly, yes [16:00:28] klausman: ok, i can move our 1 on 1 to that "dangling " slot [16:00:38] klausman: for best utilization? [16:00:52] Sure, that works [16:10:16] 10Analytics, 10Analytics-Kanban: Create new mailing list for analytics systems users - https://phabricator.wikimedia.org/T260849 (10Nuria) [16:12:25] 10Analytics, 10VPS-Projects, 10Puppet: Puppet failing on wikistats.analytics.eqiad.wmflabs due to statistics::user - https://phabricator.wikimedia.org/T259307 (10Nuria) [16:21:45] 10Analytics, 10VPS-Projects, 10Puppet: Puppet failing on wikistats.analytics.eqiad.wmflabs due to statistics::user - https://phabricator.wikimedia.org/T259307 (10Nuria) a:03razzi [16:22:15] 10Analytics, 10Analytics-Kanban: Create new mailing list for analytics systems users - https://phabricator.wikimedia.org/T260849 (10Nuria) a:05Ottomata→03elukey [16:43:51] (03PS4) 10Ottomata: Add ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) [16:46:13] (03PS5) 10Ottomata: Add ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) [16:49:56] (03CR) 10Ottomata: Add ProduceCanaryEvents job (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [16:50:24] (03CR) 10Ottomata: Add ProduceCanaryEvents job (032 comments) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [16:51:28] (03CR) 10Joal: Add ProduceCanaryEvents job (031 comment) [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [16:55:03] (03CR) 10Ottomata: [C: 03+2] Add ProduceCanaryEvents job [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/623448 (https://phabricator.wikimedia.org/T251609) (owner: 10Ottomata) [16:57:42] thanks ottomata for accepting my nitpickyness :) [16:59:51] meh - denormalize failed :( [16:59:58] joal: never seen any sign of nitpickiness from you [17:00:06] :P [17:00:14] * elukey sends wikilove to joal [17:00:20] <3 [17:00:26] whatt it failed [17:00:27] sigh [17:02:04] razzi: I gotta eat quickly and then, deploy? [17:02:17] like maybe 20-30 min [17:02:35] milimetric: +1 [17:23:58] 10Analytics-Radar, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install an-worker11[02-17] - https://phabricator.wikimedia.org/T259071 (10Cmjohnson) a:05Cmjohnson→03ayounsi @ayounsi can you add the analytics vlan to cloudsw-d5 please and these 2 servers to it's v... [17:48:04] razzi: k, sorry, ran over [17:48:18] cave? [17:48:25] cya there [17:57:41] * elukey afk! [18:00:20] milimetric: did you kill the mediawiki denormalize job? mediawiki-history-denormalize-coord? [18:00:53] nuria: nope [18:01:07] (looking) [18:01:12] milimetric: it went kaput [18:01:21] (looking) [18:01:22] milimetric: i have a 1 on 1 now but can talk after [18:04:55] I'm looking into it now milimetric, nuria [18:05:19] The error raised by spark is one I see for the first time - Meh [18:05:22] joal: invalid operation, it's not your ops week :) [18:06:07] ack milimetric - I'll let you drive - let me help along the way as you need [18:06:47] well, I always welcome help, so I'll think out loud here on the chat [18:11:49] joal: ok, so reading a bit, it looks like errors fetching blocks. Some say this happens from time to time and Spark recovers by retrying, but I'm seeing a lot of retries and all failures. Someone on stackoverflow suspects a network problem and that sounds vaguely possible with the codfw switchover, but I'm not familiar enough with which of our hadoop boxes might be in codfw (ie not kafka clusters) [18:12:11] TransportRequestHandler - Error sending result [18:12:19] OneForOneBlockFetcher - Failed while starting block fetches [18:12:34] TransportResponseHandler - Still have 36 requests outstanding when connection from an-worker1091.eqiad.wmnet/10.64.36.115:7337 is closed [18:12:44] ShuffleBlockFetcherIterator - Failed to get block(s) from an-worker1082.eqiad.wmnet:7337 [18:13:03] milimetric: lately I have seen more errors in spark jobs, which I think are due to more people using spark and therefore more pressure being put on the shuffle-handler [18:13:17] hm, but this should have priority [18:13:36] but the shuffle handler doesn't care about that, only yarn, I see [18:13:38] the job ran in the wrong queue actually (probably my bad) - we should restastrt it [18:13:42] ah! [18:13:55] ok, I'll restart, easy first step. If it happens again, we think more about the network [18:14:12] and indeed - shuffle-handler manages stuff for all spark jobs, and if the cluster is busy with spark doing a lot reading/writing, well there is pressure [18:14:33] milimetric: other interesting finding - we're hitting https://issues.apache.org/jira/browse/SPARK-23243 [18:15:22] milimetric: failures in stages usually recover - but in our case, spark didn't want to recompute because it says there was an indeterministic step somewhere [18:15:37] hm... [18:16:16] joal: but this is in the production queue, no? https://hue.wikimedia.org/oozie/list_oozie_coordinator/0000553-200720135922440-oozie-oozi-C/ [18:16:44] milimetric: it should, but is not - look in the Configuration [18:16:51] tab, the queue_name value [18:16:55] oh default [18:16:59] I saw queue: production [18:17:03] indeed [18:17:10] but queue_name: default [18:17:21] and the one used b the job is queue_name [18:19:02] milimetric: I support the idea of restarting and hoping [18:19:12] If it fails again I'll investigate checkpointing [18:19:16] milimetric: --^ [18:19:30] ok, restarting now [18:19:50] milimetric: kill restart in prod queue? [18:20:03] yeah, that's what I was gonna do... having some problems [18:20:15] np - as long as we're on the same page :) [18:20:20] thanks a lot milimetric [18:20:24] https://www.irccloud.com/pastebin/2K6uojbI/ [18:20:32] reading [18:20:53] milimetric: LGTM! [18:21:42] ah, I have to kinit to run kerberos-run-command, I am fuzzy on that [18:22:03] anyway, https://hue.wikimedia.org/oozie/list_oozie_workflow/0074295-200720135922440-oozie-oozi-W/?coordinator_job_id=0074294-200720135922440-oozie-oozi-C [18:22:35] great milimetric - I'm gonne gently look at that execution (not late) [18:22:43] milimetric: would you please log the action? [18:23:51] for instance milimetric: there currently are 2 relatively big spark jobs running from users on the cluster - and this puts pressure [18:24:21] !log restarting mediawiki history denormalize coordinator in production queue, due to failed 2020-08 run [18:24:23] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [18:24:29] good point, thx jho [18:24:31] *jo [18:30:31] 10Analytics-Radar, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install an-worker11[02-17] - https://phabricator.wikimedia.org/T259071 (10Cmjohnson) [18:36:01] milimetric: read backscroll, got it [18:36:28] :thumb: [18:36:51] * milimetric mutters something about millenials and emojis [18:49:01] hello! how can I get ssh access to the analytics boxes? I'm on the search platform team and trying to use the flink cluster in hadoop [18:50:14] nice! [18:50:25] maryum: https://wikitech.wikimedia.org/wiki/Analytics/Data_access [18:50:32] thanks a ton! [18:50:37] you'll want to submit a ticket asking for analytics-privatedata-users [18:51:30] I don't want the one for the search platform team members? analytics-search-users [18:51:41] oh wait I see [18:51:51] hadoop [18:52:22] you probalby want that too [18:52:29] just in case :) [18:54:14] noted [18:59:16] 10Analytics-Radar, 10DC-Ops, 10Operations, 10ops-eqiad, 10Patch-For-Review: (Need By: TBD) rack/setup/install an-worker11[02-17] - https://phabricator.wikimedia.org/T259071 (10Cmjohnson) [19:02:51] 10Analytics-Clusters, 10DC-Ops, 10Operations, 10ops-eqiad: (Need By: TBD) rack/setup/install an-worker10[18-40] - https://phabricator.wikimedia.org/T260445 (10Cmjohnson) @elukey Are you trying to re-use hostnames? We should be using an-worker1118+ [19:04:03] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10CDanis) I think that idea could be reasonable... but is it too hard to get the original XFF header out of the user request made to Turnilo, and forward that? [19:08:37] 10Analytics, 10Operations: eventgate-main latencies very high since the failover to codfw - https://phabricator.wikimedia.org/T261846 (10Ottomata) FYI I also increased partitions to 3 for resource_change as well. [19:11:19] so I'm already in that analytics private data users group....it might just be an error on my end, but we're not supposed to use the kerberos passwords to ssh onto the machines, correct? I haven't ssh'd to an analytics machine since the kerberos changes [19:12:30] (03CR) 10Ottomata: [C: 03+1] "Just an idea, but +1 either way." (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) (owner: 10Joal) [19:18:35] I wanna 🚂🚃🚃🚃🚃🚃🚃🚃🚃🚃🚃🚃🚃🚃 but maybe it's a bit late... [19:19:41] 10Analytics, 10Patch-For-Review: Add urlshortener button to Turnilo - https://phabricator.wikimedia.org/T233336 (10Milimetric) That would be nice. We'd have to upstream a patch, but it would be a [[ https://github.com/allegro/turnilo/blob/8824d86d37e354baea45cff492e7e57154daab5d/src/server/routes/shorten/shor... [19:27:39] maryum: you can ssh normally [19:27:48] maryum: after you are in [19:27:52] in order to access data [19:27:58] you need to type kinit [19:28:15] maryum: to set your kerberos credentials [19:28:34] maryum: so kerberos only keeps track of your data access [19:28:41] maryum: not your ssh keys [19:29:52] 10Analytics, 10Analytics-Kanban, 10Product-Analytics: Technical contributors emerging communities metric definition, thick data - https://phabricator.wikimedia.org/T250284 (10Nuria) 05Open→03Resolved [19:29:52] 10Analytics-Kanban, 10Analytics-Radar, 10Product-Analytics: Technical contributors metrics definition - https://phabricator.wikimedia.org/T247419 (10Nuria) [19:33:56] (03CR) 10Joal: [V: 03+1] Update drop-mediawiki-snapshots parameters and datasets (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) (owner: 10Joal) [19:34:28] thanks for the review ottomata - I'll try to spend a minute with Luca tomorrow thinking about the idea written in my comment-responce [19:36:05] (03CR) 10Ottomata: [C: 03+1] Update drop-mediawiki-snapshots parameters and datasets (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/623586 (https://phabricator.wikimedia.org/T237047) (owner: 10Joal) [19:52:04] ottomata: hey I won't have time to deploy source today, but I noticed there's no refinery job or puppet timer or anything that would use it, is that in a different patch somewhere or not yet done? [19:52:57] not yet! [19:53:01] will do after is deployed [19:53:03] no thurry [19:53:13] its a new job [19:53:15] no puppet yet [19:54:04] ok, cool, I'll deploy it either tonight or tomorrow morning [20:01:42] ottomata: quick question, does/can eventgate add the client's IP address to a field? [20:08:02] cdanis: it is set up to if the http.client_ip field is in the schema, yes, from the X-Client-IP header [20:08:43] ah great ty [20:08:43] Pchelolo: did you create that ticket about the kafka_burrow not goign back to 0 on eqiad? I can't find it and ryankemper can't either [20:08:54] or do you have any more understanding of what's going on? [20:10:59] gehel: the issue is change-prop, not burrow. https://phabricator.wikimedia.org/T261691 [20:13:42] Pchelolo: cool, thanks! In the short term, is there something we should do? If I understand correctly, in the current situation, when we switch back to eqiad as main DC, we'l reprocess those 4k messages [20:13:56] in the case of Cirrus, that's not an issue [20:14:27] gehel: not really reprocess them - they will be deduplicated [20:14:41] so it's weird, but there's no consequences to this [20:15:11] they will have been processed in codfw and will be processed again in eqiad? [20:53:51] (03PS1) 10GoranSMilovanovic: minor [analytics/wmde/WD/WD_HumanEdits] - 10https://gerrit.wikimedia.org/r/623864 [20:54:06] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] minor [analytics/wmde/WD/WD_HumanEdits] - 10https://gerrit.wikimedia.org/r/623864 (owner: 10GoranSMilovanovic) [20:59:33] cdanis: sorry am in meetings and hangouts all day today! [20:59:51] if you add the http fragment schema to your schema, eventgate-wikimdia will fill in some defaults in it [21:00:15] https://gerrit.wikimedia.org/r/plugins/gitiles/eventgate-wikimedia/+/refs/heads/master/eventgate-wikimedia.js#361 [21:06:33] ottomata: oh, wonderful, I'll definitely do that [21:07:40] gotta run byeeeeEEE [21:09:10] (03PS1) 10GoranSMilovanovic: Init [analytics/wmde/WD/WD_referenceHunt] - 10https://gerrit.wikimedia.org/r/623868 [21:09:19] (03CR) 10GoranSMilovanovic: [V: 03+2 C: 03+2] Init [analytics/wmde/WD/WD_referenceHunt] - 10https://gerrit.wikimedia.org/r/623868 (owner: 10GoranSMilovanovic) [21:18:29] milimetric: did we deploy aqs? [21:45:49] * nuria answering my own question: no