[00:11:40] (03PS3) 10Dave Pifke: Add TLS support [analytics/statsv] - 10https://gerrit.wikimedia.org/r/721044 (https://phabricator.wikimedia.org/T290131) [00:27:49] 10Analytics-Radar, 10Product-Analytics (Kanban): [REQUEST] Investigate decrease in New Registered Users - https://phabricator.wikimedia.org/T289799 (10Iflorez) Wikistats numbers for New Registered Users are in line with the numbers seen in the logging table (where log_action = 'create') and SSAC table (where e... [01:37:10] (03CR) 10Krinkle: [C: 03+2] "Confirmed that the defaults (proto PLAINTEXT, cafile None) match the currently implied defaults per [1], and the rest we can find out in d" [analytics/statsv] - 10https://gerrit.wikimedia.org/r/721044 (https://phabricator.wikimedia.org/T290131) (owner: 10Dave Pifke) [01:38:09] (03CR) 10Krinkle: [C: 03+1] "No tests in this repo yet. +1 for Analytics to merge/deploy as they see fit." [analytics/statsv] - 10https://gerrit.wikimedia.org/r/721044 (https://phabricator.wikimedia.org/T290131) (owner: 10Dave Pifke) [02:28:45] 10Analytics-Radar, 10Wikipedia-iOS-App-Backlog, 10Product-Analytics (Kanban), 10User-Johan: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10GeneralNotability) Comment from enwiki - we routinely hardblock open proxies, including CDNs being used as proxies. Cloudflar... [04:12:48] (03PS2) 10Ladsgroup: Add script to get some data out of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/723613 (https://phabricator.wikimedia.org/T291276) [04:13:33] (03CR) 10jerkins-bot: [V: 04-1] Add script to get some data out of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/723613 (https://phabricator.wikimedia.org/T291276) (owner: 10Ladsgroup) [04:21:39] (03PS3) 10Ladsgroup: Add script to get some data out of wb_changes [analytics/wmde/scripts] - 10https://gerrit.wikimedia.org/r/723613 (https://phabricator.wikimedia.org/T291276) [07:20:11] o/ do you use SSL (port 9093) to connect to kafka in some places? [07:21:15] Hello! Yep :) [07:21:26] varnishkafka uses it for example [07:21:34] (cpxxxx nodes to kafka-jumbo) [07:21:45] and IIRC eventgate to kafka main [07:25:22] elukey: thanks! I tested this but I have to disable hostname verif and was wondering if this is a known issue [07:25:50] cert CN reports things like "kafka_main-codfw_broker" [07:26:28] dcausse: weird, the only thing that we usually do is force the client to use the puppet CA public cert to validate the kafka TLS certificate [07:27:26] elukey: tested with openssl s_client -servername kafka-main1001.eqiad.wmnet -CAfile /etc/ssl/certs/ca-certificates.crt -verify_hostname kafka-main2001.codfw.wmnet kafka-main2001.codfw.wmnet:9093 <<< "Q" [07:27:31] and the python client as well [07:27:45] they both require to disable hostname verification [07:28:40] damn sorry typo in the first command, should be: [07:28:50] openssl s_client -servername kafka-main2001.codfw.wmnet -CAfile /etc/ssl/certs/ca-certificates.crt -verify_hostname kafka-main2001.codfw.wmnet kafka-main2001.codfw.wmnet:9093 <<< "Q" [07:29:35] dcausse: mmmm it has been a while but IIRC they share the same TLS certificate, this is why it fails the hostname verification if you add it. But any kafka client should work fine on port 9093 (adding the path to the puppet public CA cert) [07:30:09] when you say "python client" what do you mean? [07:30:15] kafkapython? [07:30:18] yes [07:31:55] dcausse: can you copy/paste the code somewhere so I can test? [07:33:14] sure [07:36:16] elukey: https://phabricator.wikimedia.org/P17333 [07:43:16] I can repro yes, and it makes sense, the CN that we use follows this [07:43:20] (from puppet privatE) [07:43:22] kafka_main-eqiad_broker: authority: puppet_ca # profile::kafka::broker expects subjectless cert to use # User:CN=kafka_main-eqiad_broker as a simple DN # (distinguished name) for ACL principals. [07:43:47] ok so it's on purpose [07:44:10] in theory yes, IIRC for the kafka ACL stuff, but I need to double check [07:45:12] I think kafka-python>2 makes things a bit easier as you can provide all these options from the constructor instead of dealing with low-level ssl_context [07:48:38] it is weird that now TLS certificate validation fails [07:49:11] I am talking with Riccardo atm :) [07:49:19] ok :) [07:50:11] dcausse: I am curious to know if this happens also with confluent-kafka-python [07:50:23] it uses librdkafka IIRC, and we have never seen an issue with it [07:50:45] ah never tried this, can it be installed with conda? [07:51:00] will give it a try [07:51:16] no idea :( [08:13:23] it works, by default it seems to load /etc/ssl/certs/ca-certificates.crt without having to specify it [08:13:34] for hostname verif I have no clue what it does [08:18:33] I am wondering if it does a weak verification or not [08:19:42] http://kafka.apache.org/documentation.html#security_ssl see section about hostname verification perhaps we disabled that at the broker level? [08:23:14] The main reason for this check is to prevent man-in-the-middle attacks. For Kafka, this check has been disabled by default for a long time, but as of Kafka 2.0.0 host name verification of servers is enabled by default for client connections as well as inter-broker connections. [08:23:18] elukey: I think I found why [08:23:54] ssl.endpoint.identification.algorithm is set to none by default cf. https://docs.confluent.io/4.0.0/kafka/authentication_ssl.html [08:24:05] so hostname verification is disabled [08:25:00] this explains why [08:25:24] so to verify, if we add ssl.enable_verify to librdkafka's config it should fail [08:26:32] I have somethong to repro with the confluent client [08:27:21] elukey: https://phabricator.wikimedia.org/P17334 [08:27:50] dcausse: yeah +1, and it fails right? [08:28:12] yes adding a comment with the error [08:29:01] dcausse: at this point I think that we could avoid hostname verification for the time being in the code (sigh), and we can open a task to fix this in Kafka, even if it will not be super easy to roll out [08:29:31] sure [08:29:46] I can open one explaining the issue, or you can do it if you have all info handy [08:30:02] elukey: I will open one [08:30:08] ack :) [08:47:17] 10Analytics, 10Data-Engineering, 10Event-Platform: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10dcausse) [09:26:04] 10Analytics, 10Data-Engineering, 10Event-Platform: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10elukey) In puppet private's repo: ` kafka_main-eqiad_broker: authority: puppet_ca # profile::kafka::broker expects subjectless cert to use #... [09:33:25] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) **Snapshot Command** We intend to take a new snapshot of one keyspace from all 12 instances on the old aqs cluster.... [09:36:26] !log restart java daemons on an-test-coord1001 to pick up new openjdk [09:36:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:40:25] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) **Transfer Script** We are going to perform 6 rsyncs in sequence. Not the quickest approach, but it won't hammer the... [09:43:16] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) All of the destination servers have around 2.7 TB free on the destination volume, which is enough to hold the snapshot... [09:51:11] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Repair and reload all cassandra-2 data tables but the 2 big ones - https://phabricator.wikimedia.org/T291469 (10BTullis) Cleared these snapshots. ` btullis@aqs1007:~$ sudo nodetool-a clearsnapshot -t T291469 Requested clearing snapshot(s) for [all keyspace... [09:53:50] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) [x] Checked that the cassandra loading job is not running. Running the snapshot create task now. [09:55:08] !log btullis@cumin1001:~$ sudo cumin --mode async 'aqs100*.eqiad.wmnet' 'nodetool-a snapshot -t T291472 local_group_default_T_pageviews_per_article_flat' 'nodetool-b snapshot -t T291472 local_group_default_T_pageviews_per_article_flat' [09:55:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [09:55:14] T291472: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 [09:57:52] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_file data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) [10:13:29] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) [10:34:17] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) The transfer operation has started now. I've opted to omit the `-v` from rsync because there are too many files to... [10:58:28] 10Analytics-Clusters: Move the Data Engineering infrastructure to Debian Bullseye - https://phabricator.wikimedia.org/T288804 (10MoritzMuehlenhoff) [12:10:33] 10Analytics, 10Analytics-Kanban: Purge gobblin files - https://phabricator.wikimedia.org/T287084 (10JAllemandou) a:03JAllemandou [12:16:41] Hi btullis :) [12:17:26] btullis: today is deployment day - I have my usual childcare moment between 4/6pm CEST - Do you wish we do the deploy before, or maybe after? [12:26:40] Before please. I have extra-ordinary childcare today between about 3:00 and 5:30 pm BST so if we could o it before, that would be great. [12:26:55] btullis: now? [12:27:08] Yes, lets :-) [12:27:40] ack btullis - batcave? [12:27:58] I'm in. [12:34:42] !log deploying refinery [12:34:45] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:35:50] (03PS1) 10Joal: Remove /wmf/gobblin from hdfs_cleaner disallowlist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/724411 (https://phabricator.wikimedia.org/T287084) [12:39:53] (03PS2) 10Joal: Remove /wmf/gobblin from HDFSCleaner disallowlist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/724411 (https://phabricator.wikimedia.org/T287084) [12:44:46] (03PS1) 10Joal: Update hdfs-cleaner jar for disallowlist change [analytics/refinery] - 10https://gerrit.wikimedia.org/r/724412 (https://phabricator.wikimedia.org/T287084) [12:58:37] 10Analytics, 10Analytics-Kanban: Move WikimediaEventUtilities logging to Slf4j - https://phabricator.wikimedia.org/T284537 (10JAllemandou) 05Open→03Resolved [12:58:50] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: When gobblin fails, we should know about it - https://phabricator.wikimedia.org/T286559 (10JAllemandou) 05Open→03Resolved [12:58:52] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Gobblin Monitoring - https://phabricator.wikimedia.org/T287991 (10JAllemandou) [12:58:54] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10JAllemandou) [13:09:07] (03CR) 10Ottomata: "hostname verification context: https://phabricator.wikimedia.org/T291905" [analytics/statsv] - 10https://gerrit.wikimedia.org/r/721044 (https://phabricator.wikimedia.org/T290131) (owner: 10Dave Pifke) [13:13:32] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10JAllemandou) a:03JAllemandou [13:14:30] 10Analytics, 10Data-Engineering, 10Event-Platform: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10Ottomata) This is also relevant for https://gerrit.wikimedia.org/r/c/analytics/statsv/+/721044 > java clients have not been tested I'm not aware of... [13:14:30] !log Deployed refinery using scap, then deployed onto hdfs [13:14:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:14:34] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Gobblin Monitoring - https://phabricator.wikimedia.org/T287991 (10JAllemandou) [13:14:42] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Push Gobblin import metrics to Prometheus and add alerts on some critical imports - https://phabricator.wikimedia.org/T286503 (10JAllemandou) [13:14:44] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Gobblin Monitoring - https://phabricator.wikimedia.org/T287991 (10JAllemandou) [13:14:47] 10Analytics, 10Analytics-Kanban, 10Epic, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10JAllemandou) [13:14:49] 10Analytics, 10Analytics-Kanban, 10Epic, 10Patch-For-Review: Replace Camus by Gobblin - https://phabricator.wikimedia.org/T271232 (10JAllemandou) [13:17:05] 10Analytics: Make camus (or gobblin) jobs run in `essential` or `production` queue - https://phabricator.wikimedia.org/T274298 (10JAllemandou) 05Open→03Resolved a:03JAllemandou This has been done when moving to gobblin. [13:17:31] 10Analytics, 10Analytics-Kanban, 10Data-Engineering: Repair and reload cassandra2 mediarequest_per_file data table - https://phabricator.wikimedia.org/T291470 (10BTullis) This is now 50% of the way through the 4th and (hopefully) final repair. ` [2021-09-28 13:12:26,784] Repair session 9f49efa1-1f6a-11ec-8f2... [13:23:38] (03CR) 10Ottomata: [C: 03+2] Remove /wmf/gobblin from HDFSCleaner disallowlist [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/724411 (https://phabricator.wikimedia.org/T287084) (owner: 10Joal) [13:51:22] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Platform Engineering, 10tech-decision-forum: MediaWiki Events as Source of Truth - Decision Statement Overview - https://phabricator.wikimedia.org/T291120 (10Ottomata) [14:01:43] 10Analytics, 10Data-Engineering, 10Event-Platform, 10Platform Engineering, 10tech-decision-forum: MediaWiki Events as Source of Truth - Decision Statement Overview - https://phabricator.wikimedia.org/T291120 (10Jenlenfantwright) a:03Jenlenfantwright [14:14:47] 10Analytics, 10Data-Engineering, 10Event-Platform: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10elukey) Yes exactly, plus @jbond added the puppet CA to all the base truststore of WMF's jvms, so in theory for java-based clients it shouldn't be a... [14:27:37] ottomata, joal, btullis - Hadoop's status seems ok, anything against me starting the cookbook to roll restart daemons? [14:27:52] (so we do it before the beginning of the next month with sqoop etc...) [14:31:09] sure! [14:39:30] started :) [14:39:49] btullis: hi! as a heads-up, presto-test-1.analytics.eqiad1.wikimedia.cloud has local patches in /var/lib/git/operations/puppet that conflict with updates on gerrit according to wmcs magic experimental monitoring systems [14:50:05] 10Quarry, 10cloud-services-team (dcaro): [quarry] Fancy up the CI pipeline in Jenkins - https://phabricator.wikimedia.org/T289569 (10dcaro) [15:39:35] majavah: Thanks for the heads up. I'll sort that out tomorrow. I was aware that puppet wasn't running cleanly, but wasn't aware of the merge conflict monitoring. [15:54:44] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Growth-Team, and 6 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Ottomata) Tested in beta, I think this is working now. [16:05:00] 10Analytics, 10Data-Engineering, 10Event-Platform: Allow kafka clients to verify brokers hostnames when using SSL - https://phabricator.wikimedia.org/T291905 (10dpifke) I ran into this when implementing Kafka TLS support for Coal, Natiming, and statsv. Besides being inconvenient for developers, this does op... [16:07:15] 10Quarry: make automatically deploy-able staging quarry - https://phabricator.wikimedia.org/T291204 (10mdipietro) 05Open→03Resolved [16:08:02] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Growth-Team, and 6 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Ottomata) a:05Mholloway→03Ottomata [16:11:07] 10Quarry: add stop status to quarry - https://phabricator.wikimedia.org/T289349 (10mdipietro) 05Open→03Resolved [16:11:44] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10mdipietro) I believe this is fixed from T289349, please reopen if this is not the case. [16:11:52] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10mdipietro) 05Open→03Resolved [16:14:16] 10Quarry: Query queued for several days, can't be stopped - https://phabricator.wikimedia.org/T290743 (10mdipietro) Some changes from T289349 have been merged that may help with this. Please give stopping another try. Alternatively the fork button should give you a fresh session. [16:16:31] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10GeoffreyT2000) 05Resolved→03Open @mdipietro No, I still could not stop https://quarry.wmcloud.org/query/53420 or https://quarry.wmcloud.org/query/56363. [16:16:57] 10Quarry: Query queued for several days, can't be stopped - https://phabricator.wikimedia.org/T290743 (10GeoffreyT2000) [16:17:03] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10GeoffreyT2000) [16:33:28] folks the cookbook is about to finish, journal nodes almost done, will be afk right after it. All metrics are good :) [16:33:46] Great, thanks. [16:44:12] (03PS1) 10Bearloga: ETL test notebook [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/724469 (https://phabricator.wikimedia.org/T291958) [16:49:28] 10Quarry: Query queued for several days, can't be stopped - https://phabricator.wikimedia.org/T290743 (10GoingBatty) @GeoffreyT2000 - Clicking the Stop button still generated an error. Clicking the Fork button generated a fresh session, and allowed me to run the query. Thanks! [16:53:17] 10Analytics-Radar, 10Wikipedia-iOS-App-Backlog, 10iOS-app-v6.9-Carp-On-A-Zamboni: Metrics around existing Echo notifications volume - https://phabricator.wikimedia.org/T291663 (10JMinor) a:05JMinor→03SNowick_WMF [16:53:54] 10Quarry: Pressing the Stop button in Quarry results in a 500 error - https://phabricator.wikimedia.org/T290146 (10GoingBatty) @MarioGom - Clicking the Stop button twice quickly did the trick - thanks! [17:12:13] 10Analytics-Radar, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, 10User-Johan: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10ldelench_wmf) [17:23:00] 10Analytics-Radar, 10Product-Analytics: Do the messages left for unregistered or logged-out IP editors get read by those editors? - https://phabricator.wikimedia.org/T291297 (10ldelench_wmf) a:03jwang [17:25:43] PROBLEM - Check unit status of eventlogging_to_druid_netflow_hourly on an-launcher1002 is CRITICAL: CRITICAL: Status of the systemd unit eventlogging_to_druid_netflow_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [17:30:14] How critical is this to sort out immediately -^^ Is it likely to sort itself onl the next hourly run, or will the job likely need back-filling? [17:41:17] hm, i actually don't know; mforns might know more but i think if we fix it will backfill properly [17:57:09] OK, thanks. Will have a look now. [17:59:57] `Sep 28 17:19:56 an-launcher1002 eventlogging_to_druid_netflow_hourly[27880]: 21/09/28 17:19:56 ERROR DataFrameToDruid: Druid ingestion task index_hadoop_wmf_netflow_hccmadid_2021-09-28T17:00:33.62Z for wmf_netflow failed.` [18:00:11] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Growth-Team, and 6 others: Migrated Server-side EventLogging events recording http.client_ip as 127.0.0.1 - https://phabricator.wikimedia.org/T288853 (10Ottomata) Scheduled for a backport window tomrrow. [18:01:35] btullis: May I help in trying to understand what happens there? [18:01:47] Please. [18:02:18] btullis: the druid indexation task failed - let's try to find more info [18:02:25] `wmf_netflow`is the long-running job isn't it, the one that runs continuously in Druid. [18:02:32] btullis: actully - batcave could make it easier [18:05:36] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, and 2 others: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Script wmf-auto-reimage was launched by cmjohnson on cumin1001.eqiad.wmnet for hosts: ` an-db1001.eqiad.... [18:07:09] RECOVERY - Check unit status of eventlogging_to_druid_netflow_hourly on an-launcher1002 is OK: OK: Status of the systemd unit eventlogging_to_druid_netflow_hourly https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [18:10:31] 10Analytics-Clusters, 10DC-Ops, 10Data-Engineering, 10SRE, and 2 others: Q1:(Need By: ASAP) rack/setup/install an-db100[12].eqiad.wmnet - https://phabricator.wikimedia.org/T289632 (10ops-monitoring-bot) Completed auto-reimage of hosts: ` ['an-db1001.eqiad.wmnet'] ` Of which those **FAILED**: ` ['an-db1001... [18:24:56] 10Analytics-Radar, 10Fundraising-Backlog, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, 10User-Johan: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10DStrine) [18:37:33] 10Analytics-Radar, 10Fundraising-Backlog, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, 10User-Johan: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10JMinor) [18:40:57] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10Htriedman) Hi @Manuel — so sorry for the late response; my phabricator... [18:49:24] Fascinating. Thanks mforns and joal for that enlightenment. [19:03:11] I have learnt how the netflow indexing jobs work. We are going to decide tomorrow whether or not to backfill that one hour which failed, or wait three days for it to fix itself. [19:31:42] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10GoranSMilovanovic) @Htriedman Maybe I am missing something: > ... e... [19:50:06] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10Htriedman) Hi @GoranSMilovanovic — apologies for the confusion. I unde... [19:51:55] (03CR) 10Bearloga: "To see the notebook: https://gist.github.com/bearloga/70714d8d5345f52a4b54cd6967d01a41" [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/724469 (https://phabricator.wikimedia.org/T291958) (owner: 10Bearloga) [19:55:56] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10GoranSMilovanovic) @Htriedman Ok, so just let me check if I understa... [20:06:08] (03PS2) 10Bearloga: ETL test notebook [analytics/wmf-product/jobs] - 10https://gerrit.wikimedia.org/r/724469 (https://phabricator.wikimedia.org/T291958) [20:14:39] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10Htriedman) @GoranSMilovanovic Just took a look. Those files are alrig... [20:27:14] 10Analytics-Radar, 10Privacy Engineering, 10WMDE-Analytics-Engineering, 10Wikidata, 10Wikidata Analytics: Privacy Policy Review for Global South Wikidata edits and active editors datasets - https://phabricator.wikimedia.org/T291186 (10GoranSMilovanovic) @Htriedman Thank you for being so precise, it rea... [20:33:11] 10Analytics-Radar, 10Product-Analytics: Do the messages left for unregistered or logged-out IP editors get read by those editors? - https://phabricator.wikimedia.org/T291297 (10mpopov) @Whatamidoing-WMF: Quick question about "to decide whether we need to re-structure communication methods for IP editors" to he... [20:35:57] 10Analytics-Radar, 10Fundraising-Backlog, 10Product-Analytics, 10Wikipedia-iOS-App-Backlog, and 2 others: Understand impact of Apple's Relay Service - https://phabricator.wikimedia.org/T289795 (10Krinkle) [23:01:33] RECOVERY - Check unit status of hdfs-cleaner-tmp on an-launcher1002 is OK: OK: Status of the systemd unit hdfs-cleaner-tmp https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:22:49] RECOVERY - Check unit status of hdfs-cleaner-tmp-analytics on an-launcher1002 is OK: OK: Status of the systemd unit hdfs-cleaner-tmp-analytics https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [23:36:17] RECOVERY - Check unit status of hdfs-cleaner-tmp-druid on an-launcher1002 is OK: OK: Status of the systemd unit hdfs-cleaner-tmp-druid https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers