[02:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [05:00:02] RECOVERY - Check unit status of refinery-sqoop-mediawiki-production-daily on an-launcher1002 is OK: OK: Status of the systemd unit refinery-sqoop-mediawiki-production-daily https://wikitech.wikimedia.org/wiki/Analytics/Systems/Managing_systemd_timers [06:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [07:37:22] hello folks! [07:37:29] I am going to move presto to the new CA bundle with https://gerrit.wikimedia.org/r/c/operations/puppet/+/739477 [07:37:47] puppet enabled only in test, will deploy there before the rest [07:42:18] basically we are switching from using the system's cacert (provided by debian) that trusts a ton of other public CAs to a bundle that trusts only our internal ones [07:42:29] doesn't change much but it is a little more secure [07:47:40] of course I forgot one bit [08:02:46] namely https://gerrit.wikimedia.org/r/742910 [08:03:41] re-resting in test :D [08:08:21] interesting, it doesn't work in presto test [08:16:48] 10Data-Engineering, 10DBA, 10Infrastructure-Foundations, 10Patch-For-Review, 10Puppet: Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics) - https://phabricator.wikimedia.org/T296285 (10jcrespo) a:05jcrespo→03BTullis Reassigning to btullis, as he was the person t... [08:17:24] right in test we have the new discovery config [08:27:48] weird though, the new truststore contains the exact same internal CAs (puppet + root pki certs) with the same fingerprints [08:27:59] but it doesn't contain the debian ones of course [08:39:25] ah now I recall, the debian default one should be PKCS12 [09:00:04] ok all good, restored to old behavior [09:00:36] always a pleasure dealing with Presto [10:31:24] elukey: Thanks for all of your work on this. [10:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [10:39:26] btullis: o/ I made a mess and fixed, not a great start of the day :( will try to see why this is happening [10:50:32] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) The 12th snapshot has finished loading. ` progress: [/10.64.32.128]0:2090/2090 100% [/... [10:50:59] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Snapshot and Reload cassandra2 pageview_per_article data table from all 12 instances - https://phabricator.wikimedia.org/T291472 (10BTullis) a:05BTullis→03JAllemandou [11:21:23] 10Analytics, 10CheckUser, 10Patch-For-Review, 10Platform Team Workboards (Clinic Duty Team), 10Schema-change: Schema changes for `cu_changes` and `cu_log` table - https://phabricator.wikimedia.org/T233004 (10Ladsgroup) Sure thing. Can I know more about plumbing and sqooping infra in Data engineering? I'm... [11:33:39] 10Analytics, 10Data-Engineering, 10Data-Engineering-Kanban, 10Desktop Improvements, and 4 others: Sticky header: Add agent_type and access_method to sticky header instrumentation - https://phabricator.wikimedia.org/T294246 (10ovasileva) [11:38:08] 10Data-Engineering, 10Data-Engineering-Kanban, 10User-razzi: Superset SQL Lab fails to stop query - https://phabricator.wikimedia.org/T293083 (10KCVelaga_WMF) I am also facing this issue, a query has been running for almost 22 hrs. {F34800408} It doesn't seem to affect queries on other tabs though. [11:56:59] 10Data-Engineering, 10DBA, 10Infrastructure-Foundations, 10Patch-For-Review, 10Puppet: Split mariadb::dbstore_multiinstance into 2 separate roles (backup sources and analytics) - https://phabricator.wikimedia.org/T296285 (10Kormat) 05Open→03Resolved Cumin alias change is merged, so i'm going to optim... [12:04:53] 10Data-Engineering, 10Data-Engineering-Kanban: Move spark.local.dir to /srv on stat100x - https://phabricator.wikimedia.org/T295346 (10BTullis) I have verified that this works in Jupyter on an-test-client1001. {F34800233,width=80%} However, the only that the new settings can be applied is to stop each of the... [12:41:51] 10Data-Engineering, 10Data-Engineering-Kanban: Move spark.local.dir to /srv on stat100x - https://phabricator.wikimedia.org/T295346 (10BTullis) I have announced a maintenance window for Jupyter at 10:00 UTC tomorrow, where I will stop all notebooks unless people request that they be excluded or the maintenance... [13:05:23] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 5 others: Update EmbedPlayer to remove the use of Navigator.userAgent - https://phabricator.wikimedia.org/T296835 (10SimoneThisDot) [13:08:47] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 5 others: Investigate OGV library use of deprecated navigator.userAgent - https://phabricator.wikimedia.org/T296837 (10SimoneThisDot) [13:11:27] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 5 others: Investigate and update videoJs library from using deprecated navigator.userAgent - https://phabricator.wikimedia.org/T296838 (10SimoneThisDot) [13:12:43] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10MediaViewer, and 5 others: Update deprecated Navigator.userAgent from MultimediaViewer - https://phabricator.wikimedia.org/T296839 (10SimoneThisDot) [13:14:21] 10Data-Engineering, 10Data-Engineering-Kanban: Move spark.local.dir to /srv on stat100x - https://phabricator.wikimedia.org/T295346 (10BTullis) One more test item. I can verify that the temporary files are created under `/srv/spark` from Jupyter. I ran a simple query on an-test-client1001 and checked for the p... [13:16:49] mforns: Heya - let me know when you have a minute to talk about Airflow SQL templating :) [13:16:53] 10Analytics-Radar, 10Anti-Harassment, 10CheckUser, 10Privacy Engineering, and 4 others: Investigate and update jQuery.lazyload library from UploadWizard from using deprecated navigator.userAgent - https://phabricator.wikimedia.org/T296840 (10SimoneThisDot) [13:17:00] joal hi! now? [13:17:08] sure mforns! To the cave [13:17:12] ok [14:09:22] o/ [14:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [15:47:14] ottomata: o/ [15:47:16] hiiii [15:47:28] if you have a min I'd need a quick brainbounce about kafka main [15:47:39] hello! [15:47:41] elukey: ya am here! [15:47:52] thanksss [15:48:01] so I opened https://phabricator.wikimedia.org/T296641, we have to upgrade some nodes to buster [15:48:19] basically kafka-main100[1-3] are on stretch, 100[4-5] on buster (same thing for codfw) [15:48:33] I have created the usual partman work for /srv preservation etc.. [15:48:49] and the profile::kafka::broker code works nicely on buster without any extra config [15:49:00] so I am thinking to just to it node-by-node [15:49:17] is there anything that you can think of to double check etc..? [15:49:42] eventgate and friends should behave nicely when the host is under reimage, pulling from other nodes [15:50:17] (I'd like to avoid to change the kafka broker list for all clients everytime, seems a waste of time) [15:54:52] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: Make it possible to use anaconda + stacked conda envs for Airflow executors - https://phabricator.wikimedia.org/T288271 (10Ottomata) 05Open→03Declined going to decline this, i don't think this is quite right. Discussion in {T296543} [15:54:54] 10Data-Engineering, 10Airflow, 10Epic, 10Platform Team Workboards (Image Suggestion API): Airflow collaborations - https://phabricator.wikimedia.org/T282033 (10Ottomata) [17:30:47] elukey: sorry ah just realized i didn't respond! [17:30:50] yes that looks good! [17:30:51] it should be fine [17:32:22] elukey: come back! [17:34:15] ottomata: trying my internet conn is broken :( [17:53:30] 10Analytics-Clusters, 10Data-Engineering: Define a list of exactly which alerts should page the Analytics team in VictorOps - https://phabricator.wikimedia.org/T296552 (10BTullis) Hi @Aklapper - Many thanks for that. I simply forgot to tag it. Adding to #data-engineering and #analytics-clusters for discussion. [18:28:44] 10Analytics, 10Analytics-Kanban, 10Data-Engineering, 10Data-Engineering-Kanban: Write document about "Fast Enough Superset" - https://phabricator.wikimedia.org/T294046 (10JAllemandou) 05Open→03Resolved [18:28:46] 10Data-Engineering, 10Data-Engineering-Kanban, 10Epic: Presto/Superset User Experience Improvement - https://phabricator.wikimedia.org/T294259 (10JAllemandou) [18:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [20:29:53] heya ottomata you still there? can you help me with airflow metastore connection? [20:32:38] yup here [20:32:39] mforns: [20:32:42] what's up? [20:32:45] razzi: also am here [20:32:58] hi ottomata ! [20:33:20] ottomata: trying to configure the metastore conn in airflow, for HivePartitionSensor [20:33:56] I'm hoping to wrap up the deployment train in the next half hour or so, then I'd like your input on data governance [20:34:02] ottomata: [20:34:28] mforns: i think we merged the change to make connections mananged by puppet, not sure what happens if you do them in the UI too [20:34:39] hmm... now I see it's trying to use the hive principal??? [20:34:39] hmm thought i wrote dcos on that, guess not? [20:34:57] right [20:35:17] ottomata: but I'm in the dev instance [20:35:23] oh locally? [20:35:29] or you mean yeah stats box [20:35:29] ok [20:35:31] what's up? [20:35:36] razzi: ok [20:35:41] I'm in an-launcher1002 [20:35:51] ok [20:36:06] meanwhile for deployment train one of the patches to deploy is https://gerrit.wikimedia.org/r/c/analytics/refinery/+/742672 [20:36:21] razzi: ya [20:36:37] when the dag runs, there's the error: thrift.transport.TTransport.TTransportException: Could not start SASL: b'Error in sasl_client_start (-1) SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor code may provide more information (Server hive/10.64.21.104@WIKIMEDIA not found in Kerberos database)' [20:36:39] its merged so when you deploy refinery, the next gobblin run should use it [20:36:47] mforns: that sounds like kerberos issue for sure [20:36:50] sooo [20:36:50] hm [20:36:55] hmmm [20:37:01] seems airflow is trying to use hive's principal to connect to the metastore? [20:37:23] yeah hm [20:38:06] how is the metastore_conn defined? [20:39:26] ottomata: points to 10.64.21.104, port 9083, extra = {"authMechanism": "GSSAPI"} [20:39:43] that's how I used it in the POC [20:40:00] !log deploy refinery for T296089 patch https://gerrit.wikimedia.org/r/c/analytics/refinery/+/742672 [20:40:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [20:40:04] T296089: Unify WMF internal CA certs bundle generation - https://phabricator.wikimedia.org/T296089 [20:40:28] very stupid question: What is the unit for time_to_first byte in https://wikitech.wikimedia.org/wiki/Analytics/Data_Lake/Traffic/Webrequest ? [20:40:41] seconds, milliseconds? [20:41:33] Not a stupid question Amir1 :) (I don't know the answer off the top of my head tho xD) [20:41:42] :D :D [20:41:51] all good, nothing urgent [20:42:29] I'm guessing it's miliseconds tho, since time to first byte is often sub-second [20:43:13] https://varnish-cache.org/docs/trunk/reference/varnishncsa.html [20:43:19] Varnish:time_firstbyte [20:43:19] Time from when the request processing starts until the first byte is sent to the client, in seconds. For backend mode: Time from the request was sent to the backend to the entire header had been received. [20:43:28] sooo seconds? [20:43:56] oh but it is a double, so it should have decimal precision [20:43:58] of some kind [20:44:05] yeah it's seconds [20:44:19] with a quick query: usually values oscillate around 2.0E-4, so I think seconds makes sense [20:44:25] ```time_firstbyte [20:44:25] 7.2E-5 [20:44:25] 2.43E-4``` [20:44:38] (from `SELECT time_firstbyte from webrequest_sampled_128 limit 2`) [20:44:49] interesting, I see 300 something for some as well which are GET requests and mediawiki supposed to terminate those after 60 [20:45:21] mforns: can i see the full log [20:45:24] but the excimer has limitations so [20:45:45] i don't really know what the right settings are, but maybe it needs to have the users' kerb ticket / keytab? [20:47:13] ottomata: https://pastebin.com/48mMwRHe [20:54:41] ottomata: question for you that may go better over voice call than me typing [20:54:54] But in the time it takes you to read this message, I'll start typing: [20:54:58] https://gerrit.wikimedia.org/r/c/analytics/refinery/+/742672/2/gobblin/common/kafka_to_hdfs_hourly.properties is the change in question [20:55:12] it's changing source.kafka.ssl.truststore.location to be /etc/ssl/localcerts/wmf-java-cacerts [20:55:24] Currently scap has deployed to the canary, stat1007, but I'm not sure how to test this change [20:55:40] gobblin is launched from an-launcher1002 [20:55:49] but, i'm pretty sure joseph and elukey have tested it [20:55:55] but, it is almost end of my day too [20:56:09] so, if you want to wait until tomorrow, it won't hurt [20:56:22] cool, I'm going to proceed, I see the new certs on an-launcher1002 [20:56:45] it's already deployed the canary but I suppose it's not testable on just stat1007 [20:57:12] In terms of actually testing it, I'll keep an ear out for any problems with the next gobblin run [20:58:57] ottomata: do you know where https://gerrit.wikimedia.org/r/c/operations/puppet/+/739923/ is in the deployment pipeline? [20:59:13] I guess it's done actually [20:59:24] it's just marked under "interim" in the train etherpad https://etherpad.wikimedia.org/p/analytics-weekly-train [21:04:18] oh, should be done razzi [21:04:21] i merged that for dan [21:04:26] mforns: [21:04:29] Cool yeah I added the date to it [21:04:29] can you make that error happen again? [21:04:43] i was trying to repro via ipython cli but i think i'm getting a different error [21:04:48] i'm watching some hive logs [21:05:00] ottomata: I'm seeing sth very weird, my credentials change in an-launcher1002, is that you? [21:05:22] your credentials? [21:05:34] i was executing some things as you to see and edit connections in your airflow instance [21:06:01] ottomata: if I do klist and then launch the dev instance, then the error happens in airflow... [21:06:04] so far so good [21:06:27] then I execute klist again, and it shows as if I was the hive user! [21:06:47] oh that is strange [21:06:49] that is not me! [21:07:10] mforns: can we try this in the hadoop-test-instance? [21:07:17] wondering if it is akeytab thing [21:07:20] yes [21:07:30] one sec [21:07:35] you can just copy/paste your dag into airflow-dags/analytics/dags there [21:08:55] an-test-client1001 [21:09:22] uou, but there's the libraries and all [21:09:45] hm [21:09:46] ok, will move some stuff [21:10:38] 10Data-Engineering, 10Data-Engineering-Kanban, 10User-razzi: Run Atlas on test cluster - https://phabricator.wikimedia.org/T296670 (10razzi) Progress: using a public docker-compose configuration, https://github.com/sonnyhcl/apache-atlas-docker, I have gotten atlas 2.2 running on my local machine: {F34803414... [21:10:47] ottomata: hm, but I'll have to move files into plugins, to be in the PYTHONPATH and change all imports [21:12:20] ok... [21:13:40] Any python qs I can help with? Train is done for now (https://gerrit.wikimedia.org/r/c/analytics/wikistats2/+/742764 will come later) [21:14:20] hehhe, we are trying to figure out why airflow can't connect to hive metastore. some kerberos issue [21:14:28] actually mforns ...i might be able to make a simple way to reporo [21:14:30] lemme try [21:14:44] gotcha [21:14:52] ottomata: I can repro, just did [21:15:06] oh! [21:15:07] ok [21:15:09] on test client? [21:15:20] ah, no no [21:15:36] or, on an lancher analytics instance [21:15:43] i just want something that is running airflow using hte analytics keytab [21:15:47] on an-launcher1002 [21:16:03] I see [21:16:32] suggested an-test-client just to not mess with stuff on our 'prod' instance, buuut [21:16:35] the script that launches the dev instance has a bug when using szudo -u analytics, as analytics has no home folder [21:16:35] i mean, nothing is there yet [21:16:41] so you can copy/paste into dags and plugins there if you like [21:17:15] that probably wouldn't work anyway without kerberos settings [21:17:17] ok, I'll need to change some import paths, will take me a bit [21:20:49] hm, there's no plugins dir in our instance [21:22:00] ottomata: ^ [21:22:53] mforns: on launcher or test client? [21:23:03] on launcher [21:23:29] aye [21:23:32] mforns: i can make it work on test client [21:23:36] but i can't with your dev instance [21:23:45] i think maybe we need to configure kerveros properly for your user [21:23:50] which, i'm not sure we can do without a keytab [21:24:24] oh you have kerberos configs in your airflow instance [21:24:27] hm [21:24:33] yes [21:25:44] To enable kerberos you will need to generate a (service) key tab. [21:25:49] yeah, mforns i think it won't work without a keytab [21:26:02] oh [21:26:09] https://airflow.apache.org/docs/apache-airflow/stable/security/kerberos.html [21:27:13] I wonder why didn't I need a keytab with the POC [21:27:16] sooo, i think we could make your dev thing still work, but it would actually only be usable from the airflow isntance the people have access too, since that is where the keytab for the airflow user lives [21:27:20] maybe I was executing as analytics? [21:27:34] maybe? weren't you using the airflow instance we set up? [21:27:38] or no? [21:27:53] the second POC yes, the first one no [21:27:55] hm [21:28:11] well, lets verify. [21:28:17] lets see if we can make your instance work as analytics with its keytab [21:28:34] ok [21:28:38] make this your kerberos conf in airflow.cfg [21:28:40] [kerberos] [21:28:40] ccache = /srv/airflow-analytics/airflow_analytics_krb5_ccache [21:28:40] keytab = /etc/security/keytabs/analytics/analytics.keytab [21:28:40] kinit_path = kinit [21:28:40] principal = analytics/an-launcher1002.eqiad.wmnet@WIKIMEDIA [21:28:40] reinit_frequency = 3600 [21:28:44] https://www.irccloud.com/pastebin/1rh7ZqRo/ [21:28:56] then, when you launch scheduler [21:29:17] sudo -u analytics kerberos-run-command /bin/airflow scheduler [21:29:28] sudo -u analytics kerberos-run-command analytics /bin/airflow scheduler [21:37:33] ottomata: I can not, the scheduler is trying to mkdir within /nonexistent ? [21:38:05] arfff [21:41:23] sorry ottomata, my head is about to explode, I started early today and am tired.. [21:41:33] I will look at this tomorrow [21:41:44] thank you for the help! [21:42:47] huh [21:42:52] mkdir in nonexistent? [21:42:54] weird [21:42:55] whyu [21:43:06] oh mforns maybe [21:43:12] export AIRFLOW_HOME=... [21:43:21] mm? [21:43:25] path do dir where you have airflow.cfg [21:43:31] oh, ok [21:43:33] tying [21:43:34] oh, but you'll have to do that within the sudo comnand [21:43:38] you might need a script then [21:43:43] make a script that exports that [21:43:47] and then launches airflow [21:43:51] and pass that to kerberos-run-command [21:46:09] ottomata:: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) attempt to write a readonly database [21:46:43] mforns: your sqlitefile [21:46:48] is probably still owned and writable only by you [21:50:04] I changed that to readable by all, and same error [21:50:51] *writable [21:52:05] hm [21:52:48] i'll chown it maybe? [21:53:07] owned it by analytics, try again mforns [21:53:20] k [21:55:02] ottomata: same [21:55:44] hmm, how are you launching scheduler? [21:56:36] /home/mforns/.conda/envs/airflow_development/bin/airflow scheduler [21:57:26] mforns: kerberos-run-command? [21:57:38] yes [21:57:39] sudo -u analytics kerberos-run-command analytics home/mforns/.conda/envs/airflow_development/bin/airflow scheduler [21:57:41] like that? [21:57:53] I did as you suggested [21:58:04] created a script [21:58:07] with: [21:58:07] oh right [21:58:17] where script? [21:58:20] https://www.irccloud.com/pastebin/z8k8KoHZ/ [21:58:54] and then called it with sudo -u analytics kerberos-run-command analytics ./script [21:59:15] ottomata: /home/mforns/airflow/test [22:02:01] hmm [22:02:03] yeah i get that too now [22:02:05] weird [22:02:08] maybe we shoudl remove the db then [22:02:12] and let airflow create it anew [22:02:31] rm airflow.db and then do the db init stuff? [22:03:05] oh maybe mforns make your airflow directory writable by analytics too? [22:20:49] 10Data-Engineering, 10Data-Engineering-Kanban, 10Airflow: [Airflow] Automate sync'ing archiva packages to HDFS - https://phabricator.wikimedia.org/T294024 (10Ottomata) ALRIGHT! Got some stuff working on the hadoop-test instance. https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/blob/main/c... [22:26:41] 10Analytics, 10Analytics-Wikistats: Wikistats feature: run http server in npm run dev - https://phabricator.wikimedia.org/T296890 (10razzi) [22:35:00] (EventgateLoggingExternalLatency) firing: (2) Elevated latency for GET events on eventgate-logging-external in codfw. - https://wikitech.wikimedia.org/wiki/Event_Platform/EventGate - https://grafana.wikimedia.org/d/ZB39Izmnz/eventgate?viewPanel=79&orgId=1&var-service=eventgate-logging-external - https://alerts.wikimedia.org [23:10:54] 10Data-Engineering, 10Data-Engineering-Kanban, 10Patch-For-Review, 10User-razzi: Increase Superset Timeout - https://phabricator.wikimedia.org/T294771 (10razzi) @odimitrijevic and I have been testing this; the change is not doing what we want (increase timeout to 3 minutes), and we came across a new error... [23:24:05] (03PS4) 10Jenniferwang: Add skin_diff schema to sanitize allowlist [analytics/refinery] - 10https://gerrit.wikimedia.org/r/737471 (https://phabricator.wikimedia.org/T287255) [23:25:43] (03CR) 10Jenniferwang: Add skin_diff schema to sanitize allowlist (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/737471 (https://phabricator.wikimedia.org/T287255) (owner: 10Jenniferwang) [23:40:24] 10Analytics-Radar, 10Product-Analytics, 10Campaign-Registration: Develop a consistent rule for which special pages count as pageviews - https://phabricator.wikimedia.org/T240676 (10Iflorez)