[01:57:48] 10Analytics, 10Analytics-EventLogging, 10Performance: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4107359 (10AndyRussG) >>! In T187207#4106874, @Nuria wrote: > @AndyRussG what is the impact on performance of EL code that you see? (cc @krinkle)... [02:52:01] 10Analytics, 10Analytics-EventLogging, 10Performance: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4107386 (10Krinkle) >>! In T187207#4107359, @AndyRussG wrote: > Hi! I'm seeing 3.13 bytes transferred (with compression) for the full El client-side... [04:51:21] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Google-Summer-of-Code (2018): GSoC Proposal 2018 : [Analytics] Improvements to Wikistats2 front-end - https://phabricator.wikimedia.org/T189964#4107429 (10sahil505) [04:57:15] 10Analytics, 10Analytics-EventLogging, 10Performance: Spin out a tiny EventLogging RL module for lightweight logging - https://phabricator.wikimedia.org/T187207#4107433 (10Nuria) Thanks @Krinkle for providing details [06:09:56] (03CR) 10Amitjoki: "> Indeed Amit, there's something weird happening with the graphModel" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423144 (https://phabricator.wikimedia.org/T182990) (owner: 10Amitjoki) [07:20:44] Hi elukey :) [07:21:06] o [07:21:08] o/ [08:19:05] (03CR) 10Elukey: "> Looks good, did we tested that -skipTrash parameter actually does" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/423844 (https://phabricator.wikimedia.org/T189051) (owner: 10Elukey) [08:19:16] joal: are you ok if I merge --^ [08:21:22] elukey: +1P [08:21:23] elukey: +1! [08:21:39] ack! [08:21:47] elukey: dumb question: [08:21:54] (03CR) 10Elukey: [V: 032 C: 032] Append '-skipTrash' to all the hdfs -rm invocations [analytics/refinery] - 10https://gerrit.wikimedia.org/r/423844 (https://phabricator.wikimedia.org/T189051) (owner: 10Elukey) [08:21:59] elukey: --skipTrash doesn't fail in case no trash is there, right? [08:22:37] joal: so if no user/something/.Trash is there? [08:23:33] or if the Trash is not enabled? [08:23:47] in both cases it doesn't fail :) [08:23:48] elukey: trash not enabled [08:24:02] elukey: cause trash is not enabled as of nopw, right? [08:24:08] yep [08:25:34] I'll enable it only after the refinery deployment [08:25:40] yup [09:12:38] 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10aberjak) Created a ticket in Opera's bug tracking system, internal reference: MT-3735. Thanks for all the info - we will investigate. [09:16:47] 10Analytics-Kanban: Make 'metric' field not a partition in mediawiki_metrics - https://phabricator.wikimedia.org/T190058#4107847 (10JAllemandou) a:03JAllemandou [09:50:37] (03PS1) 10Joal: Update oozie mediawiki-history-metrics job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) [09:51:06] (03CR) 10Joal: "To be tested this afternoon." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [10:22:10] * elukey lunch + errand! be back in ~2h (available via hangouts for any ping if needed!) [11:26:12] hi guys, we are currently upgrading libssl1.1 on cp*, varnishkafka is linked against it so we need to restart it, are the instructions here https://wikitech.wikimedia.org/wiki/Service_restarts#Cache_proxies_(varnish)_(cp) up to date regarding varnishkafka restart? [11:31:12] Hi vgutierrez - I can't say - elukey willbe bac soon and is way more knowledgeable than I am [12:28:05] (03PS2) 10Joal: Update oozie mediawiki-history-metrics job [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) [12:45:27] vgutierrez: o/ [12:45:36] sorry just seen the msg [12:46:14] so you can do it anytime, if possible in batches and not all in once, but no big precautions [12:46:18] https://grafana.wikimedia.org/dashboard/db/varnishkafka is there to help [12:46:41] (in case a restart triggers unwanted changes etc..) [12:47:08] vgutierrez: but better to test in one first, since vk now pushes to kafka using TLS [12:47:53] the service restart page is not up to date since we completed the migration to TLS only recently [12:51:22] btw, same applies for varnishkafka-statsv and varnishkafka-eventlogging? [12:51:33] 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 3 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4108511 (10herron) 05Open>03Resolved These users have been removed from `analytics-privatedata-users`. If there's any other... [12:51:54] 10Analytics, 10Operations, 10Ops-Access-Requests, 10Research, and 2 others: Restricting access for a collaboration nearing completion - https://phabricator.wikimedia.org/T189341#4108513 (10herron) [12:53:30] vgutierrez: yep (statsv doesn't use TLS yet so easier) [12:53:55] vgutierrez: if you want to restart a couple I can check kafka logs and tell you how it goes [12:54:21] ack, I restarted an hour ago cp1008 [12:54:21] (or I can take care of the restart myself later on if you are busy) [12:54:39] ah ok but no live traffic in there right? [12:54:42] right [12:54:44] I'll hit cp2025 now [12:54:51] super [12:55:39] done [12:56:35] ah that's misc, I was tailing text kafka events and wondering why cp2025 wasn't there P [12:56:38] :P [12:57:38] looks good [13:00:24] awesome.. -b 1 -s 15 on cumin looks enough to run the restart across the misc cluster? [13:00:36] 1 at a time, 15 seconds between each one [13:02:11] vgutierrez: yeah seems good, even if the best thing ever would be that the varnish frontend was drained before doing the restart (vk starts reading from the tail of the varnish logs). But we have never done it so we can proceed :) [13:02:19] I'd need to discuss this with my team [13:03:14] (namely I am wondering if the restart could trigger a small data loss due to vk restarting and reading the tail of the shm log, rather than its last event read [13:03:17] ) [13:04:58] that also reminds me of https://phabricator.wikimedia.org/T128374 [13:06:17] hmm funny, on some instances it's linked against libssl1.1 and in others against libssl1.0 [13:06:54] hey teaaam :] [13:09:40] o/ [13:10:01] ===== NODE GROUP ===== [13:10:01] (31) cp[5001-5005,5007-5012].eqsin.wmnet,cp[3007-3008,3010,3034-3037].esams.wmnet,cp[4021-4032].ulsfo.wmnet,cp1008.wikimedia.org [13:10:04] ----- OUTPUT of 'ldd /usr/bin/var... | cut -f1 -d"="' ----- [13:10:06] libssl.so.1.1 [13:10:10] (60) cp[2001-2002,2004-2008,2010-2014,2016-2020,2023-2026].codfw.wmnet,cp[1045,1048-1055,1058,1061-1068,1071-1074,1099].eqiad.wmnet,cp[3030-3033,3038-3049].esams.wmnet [13:10:13] ----- OUTPUT of 'ldd /usr/bin/var... | cut -f1 -d"="' ----- [13:10:16] libssl.so.1.0.0 [13:11:30] but same version on the whole cluster: 1.0.12-2 [13:16:21] (03CR) 10Mforns: [V: 032 C: 032] "Cool! LGTM" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/423351 (https://phabricator.wikimedia.org/T188268) (owner: 10Sahil505) [13:18:39] vgutierrez: I am pretty sure that we also have 1.1 on cp hosts no? [13:18:57] weird indeed though [13:21:17] maybe the build dependency was changed for the recent builds? if you build-depend on libssl1.1-dev it will pick 1.1, with libssl-dev 1.0.2 [13:22:09] so we've the same package version with two different builds? [13:25:04] I was checking control https://github.com/wikimedia/varnishkafka/blob/debian/debian/control [13:25:32] maybe it comes from librdkafka-dev ? [13:25:42] (it must be) [13:26:09] yes [13:26:27] elukey@cp5001:~$ ldd /usr/lib/x86_64-linux-gnu/librdkafka.so.1 | grep libssl libssl.so.1.1 => /usr/lib/x86_64-linux-gnu/libssl.so.1.1 [13:26:45] and it makes sense, it is librdkafka that does all the work [13:27:12] now what version is installed? [13:27:19] 0.11.3-1~bpo8+1+wikimedia1 on cp5001 [13:27:41] 0.9.4-1~jessie1 on cp1045 [13:27:43] vgutierrez: --^ [13:27:45] there you go [13:27:56] we have two different librdkafka versions :( [13:28:17] that explains the differences [13:28:56] so there was a problem a while ago using 0.11 with a kafka cluster "speaking" 0.9 [13:29:01] since it was causing timeouts etc.. [13:29:56] but now we have this config [13:30:30] varnishkafka-(webrequest|eventlogging) -> send events to a kafka 1.0 cluster (librdkafka 0.11 is ok) [13:30:54] varnishkafka-statsv -> sends events to a kafka 0.9 cluster [13:31:15] but I think that for --^ we have a special config in place that would make it work with librdkafka 0.11 [13:31:18] checking [13:32:13] yes [13:32:13] kafka.api.version.request=false [13:32:14] kafka.broker.version.fallback=0.9.0.1 [13:32:48] so summary: we can, in my opinion, upgrade librdkafka to 0.11 everywhere (maybe gradually just to avoid fires) [13:36:32] Hey mforns - Job finished yesterday - can you move the data and flag it for the history job to start? [13:37:31] hello joal, I already launched the mv command [13:37:34] it just finished [13:37:41] :] [13:37:46] awesome mforns :) [13:38:11] !log copied sqooped data for mediawiki history from /user/mforns over to /wmf/data/raw/mediawiki/tables/ for enwiki, table: revision [13:38:13] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:38:28] will create _SUCCESS files now, joal [13:39:05] Thanks mforns - Don't forget to log :) [13:39:06] BTW joal, hdfs wouldn't let me use mv, because there was already a folder named like that [13:39:23] just logged ^ [13:39:24] elukey: ack, I'm going to update cp2001 and let's see how it behaves after restarting varnishkafka-* [13:39:34] joal, in the end I used cp command [13:40:01] will log success files as well [13:43:32] !log created success files in /wmf/data/raw/mediawiki/tables//snapshot=2018-03 for
in revision, logging, pagelinks [13:43:33] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [13:43:40] joal, done [13:43:58] Cool mforns - I'll monitor the jobs [13:44:06] cool [13:50:25] ottomata: o/ [13:50:47] whenever you are caffeinated and ready I'd need to ask you a qs about librdkafka on cp nodes [13:57:55] (03CR) 10Mforns: [C: 031] "LGTM!" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [14:12:46] omg, I just accidentally broke the html of analytics SAL with html injection... [14:13:17] I logged a message with the word
in it [14:13:26] https://tools.wmflabs.org/sal/analytics [14:16:23] looool [14:23:50] huhu mforns :) [14:24:03] Gone to catch the kids folks - Will be back for standup [14:24:09] k :] [14:32:47] 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10mbaluta) Hi! I'm from Opera Mini server team. We did not move any traffic between DCs on specified dates and I don't see any changes on Mini server si... [14:41:13] elukey: hi sorry! [14:41:14] i missed that ping! [14:41:21] elukey: ask me! [14:43:58] np! [14:44:24] ottomata: so I had a chat with vgutierrez earlier on and it seems that on cp hosts we have deployed two versions of librdkafka (0.9 and 0.11) [14:44:34] at the same time? [14:44:34] that in turn link to two different openssl versions [14:44:49] i'm pretty sure everything should be 0.11 [14:44:55] where is it 0.9? [14:45:13] I think on these [14:45:14] cp[2001-2002,2004-2008,2010-2014,2016-2020,2023-2026].codfw.wmnet,cp[1045,1048-1055,1058,1061-1068,1071-1074,1099].eqiad.wmnet,cp[3030-3033,3038-3049].esams.wmnet [14:45:27] (they link to libssl.so.1.0.0) [14:45:50] huh. [14:45:51] weird [14:45:53] my understanding as well is that we can move all the cps to 0.11 but I wanted to ask you first [14:46:09] yeah, especially if other cps are 0.11, those will work like that too [14:46:14] i can see no reason they shouldnt' all be 0.11 [14:46:19] vk-statsv has its safeguard for the 0.9 cluster so I can't think about anyhing else [14:46:20] lookking for phab ticket though... [14:46:29] ? [14:46:44] the kafka.broker.version.fallback=0.9.0.1 etc.. [14:46:48] oh [14:46:52] yes [14:46:55] but that is not related [14:47:03] how so? [14:47:05] we need that til we update the brokers [14:47:10] yeah sure [14:47:10] right? [14:47:15] even on 0.11 librdkafka [14:47:18] yep [14:47:50] I brought it up since without that upgrading a vk-statsv instance could cause troubles to main eqiad [14:48:05] it pushes in there right? [14:48:20] yes [14:48:23] just checked :) [14:48:29] yes [14:49:14] hm yeah, but i'm pretty sure it shouldn't matter, 0.11 librdkafka should be able to produce to 0.9 cluster. hmm i will double check a test message with kafkacat [14:49:40] yes it will be fine, but it needs that option until we migrate [14:49:59] so everything is good, that one was my only concern [14:50:33] otherwise a 0.11 client will try to auto-negotiate the API version and kafka 0.9 will not like it [14:50:58] yes [14:56:20] elukey@burrow:~$ curl localhost:6667/v3/kafka/analytics-eqiad/consumer [14:56:23] {"error":false,"message":"consumer list returned","consumers":["kafka-mirror-main-analytics_to_jumbo-analytics","kafka-mirror-main-eqiad_to_jumbo-eqiad"],"request":{"url":"/v3/kafka/analytics-eqiad/consumer","host":"burrow"}} [14:56:27] \o/ [14:56:34] (this is labs) [14:56:52] yeahHH! [14:58:06] need to play a bit more on it but it seems working [14:58:13] I hate the --config-dir parameter [15:01:30] elukey, ottomata then I'll proceed with the upgrade + varnishkafka restarts across cp*, thanks :D [15:01:54] vgutierrez: thanks a lot! [15:02:09] let me know if I can help [15:07:11] vgutierrez: all cache hosts are still jessie, is that correct? [15:07:23] i'm looking into librdkakfa (nad kafkacat elsewhere now), and this is a bit all over the place [15:07:36] thinking of building some updated packages for strectch and jessie for both of those now [15:07:53] re T182163 [15:07:54] T182163: Update to latest kafkacat - https://phabricator.wikimedia.org/T182163 [15:07:57] ottomata: right [15:19:10] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Services (doing): Add support for catch-all rule in ChangeProp - https://phabricator.wikimedia.org/T191238#4109024 (10Pchelolo) The multi-topic rule support was added to change-prop ad deployed for the JobQueue. What's left is to make that exclusive so that... [15:19:23] 10Analytics, 10New-Readers, 10Operations, 10Traffic, and 2 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4109027 (10Nuria) Twitter is the best! Thanks for taking time to look into this: @mbaluta: how about we setup a short meeting and we can go over data changes we... [15:29:20] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban: Spike: Consider alternatives to MirrorMaker: uReplicator, Confluent Replicator - https://phabricator.wikimedia.org/T190049#4061233 (10Ottomata) Moving this to done. I think it is best to stick with MirrorMaker for now. We know why (mostly) it wasn't wor... [15:34:40] 10Analytics, 10Analytics-Cluster, 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Fix Mirror Maker erratic behavior when replicating from main-eqiad to jumbo - https://phabricator.wikimedia.org/T189464#4109087 (10Ottomata) Alright! Since the changes we made a couple of days ago, main -> jumbo has... [15:41:39] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4109112 (10Ottomata) [15:44:47] I'm gonna be a few minutes late at standup folks sorry [15:55:12] a-team: I am doing another meeting (cassandra standup), I may be late to standup [16:02:17] ping joal [16:22:40] elukey, ottomata update finished, no issues on my side, thanks again guys :D [16:23:32] awesome, thank you [16:26:38] 10Analytics-Kanban, 10Patch-For-Review: Spark 2 as cluster default (working with oozie) - https://phabricator.wikimedia.org/T159962#4109272 (10Ottomata) [16:27:04] vgutierrez: just checked the vk dashboard, all good, thanks a lot! [16:45:33] 10Analytics, 10Analytics-EventLogging, 10Reading Epics (Analytics): Bulk/Batch event endpoint - https://phabricator.wikimedia.org/T166249#4109424 (10Ottomata) I think we should lump this task in as a feature of the Event Data Platform program next FY [16:45:49] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4109425 (10Nuria) [16:46:41] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#3961955 (10Nuria) p:05Triage>03High [16:46:55] 10Analytics, 10Analytics-Dashiki: Paginate table-timeseries visualization - https://phabricator.wikimedia.org/T191270#4099283 (10Nuria) p:05Triage>03Low [16:47:21] 10Analytics, 10Contributors-Analysis: Add raw sites table to Analytics Data Lake - https://phabricator.wikimedia.org/T191412#4104521 (10JAllemandou) Hi @Neil - Would `wmf_raw.mediawiki_project_namespace_map` saisfy the need ? This table is updated every month (snapshot partition) and is defined as explained [... [16:50:39] 10Analytics: Provide historical redirect flag in Data Lake edit data - https://phabricator.wikimedia.org/T161146#3122912 (10Nuria) p:05Normal>03Low [16:52:12] 10Analytics, 10Easy: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604#3534404 (10Nuria) p:05Triage>03Normal [16:52:26] 10Analytics, 10Easy: Reportupdater: do not write execution control files in source directories - https://phabricator.wikimedia.org/T173604#3534404 (10Nuria) p:05Normal>03Low [16:53:32] 10Analytics: Adding top counts for wiki projects (ex: WikiProject:Medicine) to pageview API - https://phabricator.wikimedia.org/T141010#4109479 (10Nuria) p:05Normal>03Low [16:53:44] 10Analytics, 10Pageviews-API: Adding top counts for wiki projects (ex: WikiProject:Medicine) to pageview API - https://phabricator.wikimedia.org/T141010#2484223 (10Nuria) [16:59:36] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Consistently preserve settings when a user switches to a new metric (especially on the same page). - https://phabricator.wikimedia.org/T183181#4109507 (10mforns) [16:59:39] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats: Wikistats Beta – Put view settings in URL so it can be shared. bookmarks - https://phabricator.wikimedia.org/T179444#4109510 (10mforns) [16:59:49] 10Analytics, 10Analytics-Kanban: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#4109511 (10Nuria) a:03Ottomata [16:59:59] 10Analytics, 10Analytics-Kanban: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#3613231 (10Nuria) p:05Low>03Normal [17:02:09] 10Analytics, 10Operations, 10netops, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#4109537 (10Nuria) 05stalled>03Resolved [17:05:28] 10Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#4109556 (10Nuria) We think this can be inferred from pivot data: https://pivot.wikimedia.org/#webrequest Do we want a more permanent dashboard rather than an exploratory one (for last week of data)? [17:05:39] 10Analytics: Create ops dashboard with info like ipv6 traffic split - https://phabricator.wikimedia.org/T138396#4109557 (10Nuria) p:05Normal>03Low [17:06:05] 10Analytics, 10Operations, 10netops, 10User-Elukey: Review ACLs for the Analytics VLAN - https://phabricator.wikimedia.org/T157435#4109561 (10elukey) Since this task has been open for a long time, I'll open a new one when we'll be ready to create the analytics-in6 filter. [17:09:23] 10Analytics, 10Trash: --- Items above are triaged ----------------------- - https://phabricator.wikimedia.org/T115634#4109572 (10Nuria) 05stalled>03Invalid [17:11:02] 10Analytics, 10Analytics-Wikistats: Can't combine 'Editor type' and editor 'Activity level' filters to narrow results (in WikiStats 2.0) - https://phabricator.wikimedia.org/T183316#3849995 (10Nuria) p:05Triage>03Normal [17:13:44] 10Analytics: Create new table for 'referer' aggregated data - https://phabricator.wikimedia.org/T112284#1630965 (10Nuria) p:05Normal>03Low [17:14:45] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772#4109610 (10Nuria) a:03Nuria [17:15:08] 10Analytics, 10MediaWiki-Releasing: Create dashboard showing MediaWiki tarball download statistics - https://phabricator.wikimedia.org/T119772#1835400 (10Nuria) p:05High>03Low [17:16:34] 10Analytics: Transform and Import Qualtrics Survey data - https://phabricator.wikimedia.org/T184626#4109622 (10Nuria) p:05Triage>03Low [17:17:30] 10Analytics, 10Analytics-Cluster, 10Operations: Clean up permissions for privatedata files on stat1005 - they should be group readable by statistics-privatedata-users - https://phabricator.wikimedia.org/T89887#1048013 (10Nuria) There are not many private files anymore, declining. [17:17:40] 10Analytics, 10Analytics-Cluster, 10Operations: Clean up permissions for privatedata files on stat1005 - they should be group readable by statistics-privatedata-users - https://phabricator.wikimedia.org/T89887#4109632 (10Nuria) 05Open>03declined [17:18:12] 10Analytics, 10User-Elukey: Secure hue and other private data access sites with 2FA - https://phabricator.wikimedia.org/T159584#4109633 (10Nuria) p:05Normal>03Low [17:22:23] 10Analytics, 10Discovery-Analysis, 10Reading-analysis: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828#4109650 (10Nuria) Our preferred path here is that the data analysts team initiates this work and we support them. Given queries are done it just r... [17:22:45] 10Analytics, 10Discovery-Analysis, 10Reading-analysis: Productionize per-country daily & monthly active app user stats - https://phabricator.wikimedia.org/T186828#3956689 (10Nuria) p:05Triage>03Low [17:23:17] 10Analytics, 10User-Elukey: Refactor analytics cronjobs to alarm on failure with (maybe) cronic - https://phabricator.wikimedia.org/T172532#3501241 (10Nuria) p:05High>03Normal [17:24:56] 10Analytics, 10EventBus, 10Services (watching): Enable multiple topics in EventStreams URL - https://phabricator.wikimedia.org/T187418#3974509 (10Nuria) p:05Triage>03Low [17:26:11] 10Analytics, 10EventBus, 10Patch-For-Review, 10Services (watching): EventBus schema validation should report the name of the failed property - https://phabricator.wikimedia.org/T188027#4109683 (10Nuria) 05Open>03Resolved [17:28:57] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Hadoop jobs that generate large temporary files can take down nodes - https://phabricator.wikimedia.org/T187139#3965657 (10Nuria) Yarn supports proper containers, it is probably a lot of work but the root cause solution should be "containerize" the execut... [17:29:03] 10Analytics, 10Analytics-Cluster, 10Patch-For-Review: Hadoop jobs that generate large temporary files can take down nodes - https://phabricator.wikimedia.org/T187139#3965657 (10Nuria) p:05Triage>03Low [17:32:39] 10Analytics, 10Operations, 10User-Elukey: Import some Analytics git puppet submodules to operations/puppet - https://phabricator.wikimedia.org/T188377#4109729 (10Nuria) p:05Normal>03Low [17:35:13] * elukey off! [17:35:56] 10Analytics, 10Analytics-Kanban, 10EventBus, 10Patch-For-Review, 10Services (watching): Use --new.consumer for main codfw <-> eqiad Kafka MirrorMaker - https://phabricator.wikimedia.org/T190940#4109745 (10Ottomata) I'd like to try this early next week. [17:51:05] (03CR) 10Nuria: Update oozie mediawiki-history-metrics job (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [17:56:21] (03CR) 10Joal: "Comments inline" (032 comments) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [18:29:44] madhuvishy: yt? [18:29:50] having trouble with dumps mount on notebook servers [18:34:14] Gone for diner team [18:36:35] 10Analytics: stats.wikimedia.org home page should link to wikistats 2 - https://phabricator.wikimedia.org/T191555#4109929 (10Nuria) [18:37:56] (03CR) 10Nuria: Update oozie mediawiki-history-metrics job (031 comment) [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [18:45:41] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mount dumps on SWAP machines (notebook1001.eqiad.wmnet / notebook1002.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#4109973 (10Ottomata) @madhuvishy, I did as described, and things seem fine, but there aren't any files in the mounted directories. [18:45:55] 10Analytics, 10Analytics-Kanban, 10Patch-For-Review: Mount dumps on SWAP machines (notebook1003.eqiad.wmnet / notebook1004.eqiad.wmnet) - https://phabricator.wikimedia.org/T176091#4109975 (10Ottomata) [19:03:02] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4110036 (10Nuria) >Normally users from Nigeria connect to a data center in Europe, however between February 17th - March 10th I see a small... [19:09:43] Hey ottomata - Will you work tomorrow? [19:10:16] joal: ya [19:10:26] usually no but i am going to switch days with next friday [19:10:37] ottomata: do you mind if we postpone testing refine-spark2 to tomorrow? [19:10:52] joal: sounds good! [19:10:57] ottomata: I could do wih an early evening today :) [19:11:01] Thanks mate [19:11:07] i'll get spark 2.3 installed today [19:11:22] Great - Tomorrow, testing day [19:11:25] ya [19:11:38] I'll make sure to actually re-test the other ones [19:19:12] (03CR) 10Joal: [V: 031] "tested in cluster." [analytics/refinery] - 10https://gerrit.wikimedia.org/r/424253 (https://phabricator.wikimedia.org/T190058) (owner: 10Joal) [19:24:24] !log upgrading spark2 to spark 2.3 [19:24:26] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [19:24:29] Done for today team - See ou tomorrow :) [19:28:08] 10Analytics, 10EventBus, 10MediaWiki-JobQueue, 10Goal, and 2 others: FY17/18 Q4 Program 8 Services Goal: Complete the JobQueue transition to EventBus - https://phabricator.wikimedia.org/T190327#4110068 (10Pchelolo) Here's the proposed bulk for the next iteration: - LocalGlobalUserPageCacheUpdateJob - Glob... [19:36:00] 10Analytics-Kanban, 10Patch-For-Review: Spark 2 as cluster default (working with oozie) - https://phabricator.wikimedia.org/T159962#4110077 (10Ottomata) [19:36:53] 10Analytics-Kanban, 10Patch-For-Review: Spark 2 as cluster default (working with oozie) - https://phabricator.wikimedia.org/T159962#3084705 (10Ottomata) Spark 2.3 installed fleet wide. I also updated the spark2 spark-assemply.zip file and added a new oozie sharelib spark2.3.0: ``` sudo -u spark hdfs dfs -put... [20:05:47] ottomata: question [20:05:52] if you are there [20:14:25] ya hey nuria_ [20:14:28] sorry missed poing [20:14:56] ottomata: can we give access to superset to people that just have an account on wikimediafoundation wiki? [20:15:04] ottomata: via oauth? [20:15:17] ottomata: is there any way to not have to do ldap? [20:18:12] nuria_: maybe, but not easily [20:18:57] ottomata: ok [20:19:47] 10Analytics: Research how hard it would be to do oauth with wkimediafoundation.org as an option for access to superset - https://phabricator.wikimedia.org/T191563#4110183 (10Nuria) [20:34:42] nuria_: why? [20:34:43] :) [20:35:22] ottomata: cause i can think of a bunch of people that work here that would like to look at geowiki data that probably cannot get themselves shell access [20:35:36] ottomata: coms team, asaf's team (except asaf himself) [20:36:21] nuria_: just for viewing, right? [20:36:26] ottomata: yes [20:36:37] there is an anonymous permission level in superset [20:36:47] i'm not sure how certain things are restricted [20:37:02] we've got it behind that apache ldap authenicator, so that's a little wierd [20:37:11] we only did that because of some bugs upstream that maybe will be fixed soon? [20:37:25] if we didn't have to do that, then we could probably allow access to (some?) dashboards without authenticating [20:39:12] ottomata: we need authentication cause data is private [20:39:27] ottomata: but it should -ideally- not require ldap [20:40:12] ah [20:40:39] hm, nuria_ they should be able to sign in with ldap perms in the wmf or nda group [20:40:45] without having a shell account [20:40:56] (they'd have to use a 'shell' username, which is also assigned to them in ldap) [20:41:06] they'd just not be able to do any hadoop queries, etc. [20:41:24] we'd still probably ahve to manually create superset accounts for them [20:41:27] but should be possible [20:42:41] ottomata: but how do they create that account outside of the process of requesting access with ssh keys [20:43:29] nuria_: ldap access != ssh access [20:43:45] i think they just need a wikitech account [20:43:49] ottomata: right, but how do they request ldap access alone? [20:44:12] ottomata: i do not think i have seen docs on that [20:44:18] people def do it, looking [20:45:04] here's instructions for 'volunteers' https://wikitech.wikimedia.org/wiki/Volunteer_NDA#Volunteer_NDA_for_privileged_LDAP_access_or_shell_access [20:46:00] 10Analytics, 10Analytics-Wikistats: Updating Wikistats 2.0 github Readme - https://phabricator.wikimedia.org/T191567#4110264 (10sahil505) [20:47:00] yeah, nuria_ they just need a wikitech account and need to be in either the nda or wmf grou [20:47:00] pso [20:47:01] so [20:47:14] this would just be requesting to put their wikitech account in the wmf group i guess [20:49:06] ottomata: i guess we would need to create a wiki about that cause all wikis i see link a wikitech account with creation of ssh keys: https://wikitech.wikimedia.org/wiki/Help:Getting_Started we will need to do that [20:49:13] ottomata: will get something [20:49:44] yeah [20:49:57] ottomata: i think this would do: https://wikitech.wikimedia.org/wiki/Analytics/Systems/Pivot#Access_to_Pivot [20:50:06] nuria_: we could add it on https://wikitech.wikimedia.org/wiki/Analytics/Data_access maybe? [20:50:12] oh sure! [20:50:12] yeah [20:50:20] that'll do [20:52:41] chelsyx: heya! just updated to spark 2.3, and along the way included a spark2-thriftserver executable [20:52:47] on stat1005, you can now just do [20:52:48] spark2-thriftserver [20:52:59] or [20:52:59] spark2-thriftserver --master yarn ... [20:54:05] ottomata: Cool! Thanks ottomata ! I will try later [21:00:32] 10Analytics, 10Analytics-Wikistats: Updating Wikistats 2.0 github Readme - https://phabricator.wikimedia.org/T191567#4110289 (10Nuria) Sounds great, please work on this item cc @mforns [21:05:37] ottomata: https://wikitech.wikimedia.org/wiki/Analytics/Data_access#Dashboards:_Superset_and_pivot [21:21:51] gr8 nuria_! [22:03:24] Confirm that the spark2-thriftserver executable works! Thank you ottomata !!! [22:08:13] (03PS1) 10Sahil505: Updated Readme & corrected syntax errors [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/424464 (https://phabricator.wikimedia.org/T191567) [22:09:56] (03CR) 10Sahil505: "Please do let me know if something else can be a good addition to this." [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/424464 (https://phabricator.wikimedia.org/T191567) (owner: 10Sahil505) [22:23:32] (03CR) 10Nuria: Updated Readme & corrected syntax errors (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/424464 (https://phabricator.wikimedia.org/T191567) (owner: 10Sahil505) [22:24:00] 10Analytics, 10Analytics-Kanban, 10Analytics-Wikistats, 10Patch-For-Review: Updating Wikistats 2.0 github Readme - https://phabricator.wikimedia.org/T191567#4110473 (10Nuria) [22:54:42] 10Analytics, 10Data-release, 10Research, 10Privacy: An expert panel to produce recommendations on open data sharing for public good - https://phabricator.wikimedia.org/T189339#4110555 (10DarTar) [23:00:04] 10Analytics, 10Analytics-Data-Quality, 10Analytics-Kanban, 10New-Readers, and 4 others: Opera mini IP addresses reassigned - https://phabricator.wikimedia.org/T187014#4110582 (10Nuria) Varnish5 rollout might have something to do with this? https://gerrit.wikimedia.org/r/#/c/409047/ cc @ema