[00:28:19] 10Analytics-Kanban, 10Analytics-Wikistats: SEO-friendly HTML titles for Wikistats 2.0 - https://phabricator.wikimedia.org/T182718#3888638 (10Nuria) The titles of pages shoudl be pushed to piwik so we can see traffic paths through the site [00:51:09] 10Analytics-Kanban: Create scala-spark job to ingest simple data sets from Hive-EventLogging to Druid to Pivot - https://phabricator.wikimedia.org/T179976#3742504 (10Nuria) Is this the changeset: https://gerrit.wikimedia.org/r/#/c/386882/? [00:52:32] (03PS23) 10Nuria: Add core class and job to import EL hive tables to Druid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/386882 (https://phabricator.wikimedia.org/T166414) (owner: 10Mforns) [00:54:26] (03CR) 10Nuria: "Can this be merged?" [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/401814 (https://phabricator.wikimedia.org/T183192) (owner: 10Nuria) [00:58:55] (03CR) 10jerkins-bot: [V: 04-1] Add core class and job to import EL hive tables to Druid [analytics/refinery/source] - 10https://gerrit.wikimedia.org/r/386882 (https://phabricator.wikimedia.org/T166414) (owner: 10Mforns) [07:01:43] 10Analytics-Kanban: Create scala-spark job to ingest simple data sets from Hive-EventLogging to Druid to Pivot - https://phabricator.wikimedia.org/T179976#3888988 (10mforns) @Nuria Yes. Sorry, this change was started as part of the other task mentioned in the description, and I reused it. [08:49:43] morningggg [08:50:32] db1107 is still processing/sanitizing data, it will probably finish on Friday/Saturday if it keeps going in this way [09:11:47] Hi elukey - forgot to remind you yesterday, today I'm off day-wide [09:12:31] joal: <3 - have a good day [09:17:28] elukey: camus and webrequest have been successfully tested yesterday, but cluster seems in an unstable mode this morning [09:18:10] elukey: gone again ;) Later friend [09:27:58] ah nice! Will investigate then when you'll be back, there are probably a lot of things to tweak [10:11:09] 10Analytics-Kanban, 10Operations, 10ops-eqiad: dbstore1002 possibly MEMORY issues - https://phabricator.wikimedia.org/T183771#3889328 (10jcrespo) Please send us a 15 minute meeting invite, there are some things that we need to discuss regarding dbstores for you to talk to analytics and other dbstore users. T... [11:03:09] !log reboot analytics1040->43 for kernel updates [11:03:10] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:03:23] I'll try to do 40->59 today [11:11:14] https://grafana.wikimedia.org/dashboard/db/prometheus-analytics-hadoop ported to prometheus!! [11:11:17] \o/ [11:11:58] I can see that the jvm metrics are more precise [11:12:08] and we'll probably need some follow ups [11:26:27] !log reboot analytics1044->47 for kernel updates [11:26:30] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:27:23] !log re-run webrequest-load-wf-text-2018-1-10-10 (failed due to reboots) [11:27:24] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [11:51:05] !log re-run webrequest-load-wf-upload-2018-1-10-10 (failed due to reboots) [11:51:06] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [12:26:16] applied https://gerrit.wikimedia.org/r/#/c/403123/ with the recent standard that we adopted for jmx/jvm prometheus metrics [12:27:05] basically the jmx_exporter grabs by default a list of jvm metrics [12:27:15] without duplicates etc.. so we don't need to specify them in the configs [12:27:35] the goal is to eventually have the same set of metrics for all the wikimedia jvms [12:27:49] and have a common/shared dashboard for them (far down the future) [12:28:00] anyhow, let me know if you see any issue [12:28:23] I've ran puppet on an1001,1002,1003, druid1001/4 and a couple of hadoop worker nodes, all good [12:28:28] then I've re-enabled puppet [12:28:37] so in ~30m the config will be everywhere [12:28:40] now, lunch :) [12:51:16] (03CR) 10Fdans: [C: 032] Replacing JSON download with CSV download [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/401814 (https://phabricator.wikimedia.org/T183192) (owner: 10Nuria) [12:58:29] hi a-team :] [12:59:12] morning [12:59:13] whatuuuup mforns [12:59:28] hellooo [12:59:56] mforns: YOU GET A CR! [13:00:59] fdans, xD for me to look at? will do! [13:01:33] a-team we got by country pageviews!!! :D [13:01:35] * fdans dances [13:01:47] https://wikimedia.org/api/rest_v1/metrics/pageviews/top-by-country/en.wikipedia/all-access/2017/10 [13:02:39] congrats fdans :) Awesome work :) [13:02:53] * joal sings for fdans to dance :) [13:03:17] woo!! this is really cool! [13:03:42] so it's for privacy reasons that we only show a range of pageviews, and not an exact number? [13:04:03] joal: it's weird because I tested the URL just as this song was about to break https://youtu.be/hLE8EFVHLDY?t=3m20s [13:04:08] musikanimal: that's right :) [13:04:18] I'm documenting that now in the Pageview API docs [13:04:36] cool thanks :) [13:07:17] (03CR) 10Fdans: "Just a couple of dissents but LGTM" (032 comments) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/402466 (https://phabricator.wikimedia.org/T183188) (owner: 10Milimetric) [13:07:20] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats, 10RESTBase-API, 10Services (done): Add "Pageviews by Country" AQS endpoint - https://phabricator.wikimedia.org/T181520#3889684 (10mobrovac) The public API endpoint has been deployed. [13:08:50] fdans: congratulations man, this is very cool [13:08:55] People gonna love it [13:10:52] thank youuuu, I'm really happy [13:58:48] fdans, it's suuuper, I learned of the existence of a couple countries like Northfolk Island :] [13:58:55] kudos! [14:29:20] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Hadoop daemons - https://phabricator.wikimedia.org/T177458#3889830 (10elukey) I am testing in labs why oozie/hive daemons are not starting up with the -javaagent. For oozie I keep seeing the log (in catalina.ou... [14:31:11] so nice --^ [14:31:22] oozie duplicates the java command line options [14:31:36] so my -javaagent explodes when started two times [14:32:09] oozie does? [14:32:22] ottomata: o/ [14:32:35] hiii [14:32:35] not sure, I don't find it in our config afaics [14:32:52] (I am testing in the labs coordinator atm) [14:33:08] what env var do you set it in? [14:33:39] CATALINA_OPTS [14:33:57] but even without it (so with the current config) things are repeated [14:34:17] like -Doozie.https.port=11443 -Xmx2048m -Doozie.https.port=11443 [14:38:04] hmm [14:38:42] wow multiple times [14:40:41] elukey: i think its some combo of oozied.sh and oozie-env.sh [14:40:44] in /usr/lib/oozie/bin [14:41:11] I was checking those [14:41:28] i see both setting things like https.port [14:42:09] yes you are right.. so is the init.d script wrong? [14:42:15] ya, oozied.sh sources oozie-sys.sh, which sources oozie-env.sh [14:42:29] yeah, looks like an oozie [14:42:30] bug [14:44:16] hmmm, but we also have export CATALINA_OPTS="$CATALINA_OPTS -Doozie.https.port=${OOZIE_HTTPS_PORT}" in /etc/oozie/conf/oozie-env.sh [14:44:27] looks like we could reove those lines [14:45:19] i think we don't need to export CATALINA_OPTS in /etc/ ... oozie-env.sh [14:45:36] elukey: in which file do you set -javaagent? [14:45:43] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3889866 (10BBlack) That looks about right (disable all hashes older than SHA256, disable RSA+DSA), although it's hard to suss exactly what th... [14:45:44] -env.sh [14:45:48] in /etc/oozie/conf ? [14:46:03] in /etc/oozie/conf/oozie-env.sh, where the other things are [14:46:10] not sure where to do it otherwise [14:46:48] aye [14:46:51] ya that sounds right [14:46:52] hmm [14:48:11] it seems a bug in how the init.d script is layed out though [14:48:33] the init.d script? [14:49:39] yes, it might not be done correctly (by cloudera?) and hence triggering this, or maybe we are using the env conf wrong, either one [14:52:37] it might work setting JAVA_OPTS, trying [14:53:34] yes it works :D [14:53:34] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3889872 (10Ottomata) > Does that mean SHA1 is disabled, except in the cases that it's the root cert of a chain stored in the jdkCA's default... [14:53:48] it does? [14:53:57] yep, oozie starts and I can see the metrics [14:54:05] not sure how... [14:54:06] oozied.sh: export JAVA_OPTS=${CATALINA_OPTS} [14:54:37] only in stop no? [14:54:40] not in start [14:54:59] ? [14:55:04] i'm looking in oozied.sh [14:55:22] oh [14:55:22] yes [14:55:23] # A bug in catalina.sh script does not use CATALINA_OPTS for stopping the server [14:55:26] export JAVA_OPTS=${CATALINA_OPTS} [14:55:28] ohhh [14:55:30] hmm [14:55:34] where is JAVA_OPTS set then? [14:55:35] mmm [14:56:06] AH [14:58:50] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, 10User-Elukey: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3889885 (10BBlack) Yeah, seems reasonable to just set it system-wide on these systems. [14:59:17] in journalctl at some point I can see Jan 10 14:55:45 hadoop-coordinator-1 oozie[21911]: setting JAVA_OPTS=etc.. [14:59:43] yeah its in /usr/lib/bigtop-tomcat/bin/catalina.sh [15:00:16] "$_RUNJAVA" "$LOGGING_CONFIG" $LOGGING_MANAGER $JAVA_OPTS $CATALINA_OPTS [15:00:18] sounds fine elukey :) [15:01:22] it looks like a big mess to me [15:01:28] but yeah it might be ok [15:01:36] I am frustrated that I don't find any clear doc about it [15:06:56] ottomata: sorry my test was not right, I re-tried with export JAVA_OPTS="$JAVA_OPTS -Xmx2048m" and it yields the same duplication [15:08:22] so in the init.d seems to first call /usr/lib/oozie/bin/oozie-env.sh [15:08:47] then, it checks what function to call (start/stop/etc..) and calls oozied.sh right? [15:08:50] ya [15:09:19] but JAVA_OPTS shouldn't be mangled anywhere by oozie scripts, (except by stop) [15:09:45] so oozie is def duplicating crap all over the place, but it shouldn't duplicate anythign in JAVA_OPTS, no? [15:10:28] but it does afaics [15:22:44] I am also trying to break hive now :) [15:24:00] and hive with the new settings works [15:24:22] great! [15:24:22] :) [15:24:22] (setting HADOOP_OPTS in hive-env.sh) [15:24:25] aye [15:25:34] eheh spoken too soon, it doesn't bind its port [15:25:46] now I remember, this was the issue [15:25:55] the javaagent works, but the daemon does not bind the port [15:26:03] (hive server/metastore I mean) [15:26:46] hmm [15:41:41] Hi - are there any opportunities for tech volunteers? I've been reading various wikis and looking for guidance on this but so far I'm not clear on how to get on board or whether volunteering for tech folks is an option [15:43:28] Looking at Phabricator there are quite a few tickets I would love to help with. I have experience in many elements of your stack and I could spend 5 to 8 hours a week [15:49:20] 10Analytics, 10Operations, 10hardware-requests, 10ops-eqiad: Decommission db104[67] - https://phabricator.wikimedia.org/T181784#3890027 (10Cmjohnson) 05Open>03Resolved [15:55:46] (03PS2) 10Fdans: Translate g according to the y-axis width [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/403184 (https://phabricator.wikimedia.org/T184138) [16:02:04] ralgara: hiya! Some of the things we are are difficult to get volunteers to work on, because of certain access requirements, but some things are easier. What type of work are you looking to do? [16:02:23] what elements of the stack are you familiar with? [16:06:47] ottomata: a few: Hadoop MR, Hive, Kafka, Elasticsearch, Cassandra, MariaDB. I'm also fluent in Scala, Java, Python, C++ and others and have background in machine learning [16:07:50] hm! [16:07:52] cool [16:07:54] All backend, though I've done a good amount of front end work and could help there. At this point I just want to get involved and familiar with your stack [16:07:57] joal: ^^ any ideas? [16:08:14] a-team: from now on, https://grafana.wikimedia.org/dashboard/db/analytics-hadoop?orgId=1 points to the new prometheus metrics (the old one is available at https://grafana.wikimedia.org/dashboard/db/analytics-hadoop-graphite) [16:09:52] ottomata: I've been pulling and building analyses on your pagecount for a while. I do NLP and sentiment analysis in my day job and I'm particularly interested in how Wikipedia traffic reflects social trends [16:11:04] ralgara: we'll try and think of something, a lot of that type of work is hard to do without cluster access [16:11:21] which, is not impossible to get, but it might be hard unless you have a project or university or something that wants to partner with someone here [16:11:29] Hi folks! [16:11:32] we have a daily meeting in a bit, we'll discuss :) [16:11:37] in the meantime, have you seen https://phabricator.wikimedia.org/tag/analytics/ ? [16:11:45] it is a bit disorganized but contains our huge backlog of things do to [16:12:08] Yes, I've been going through Phab tickets and definitely see lots of interesting stuff [16:12:50] My company would probably support some kind of more official involvement. I suppose NDAs and other formalities are needed. I'll check on my end too [16:14:02] nuria_, ottomata, milimetric: I'd like to continue the discussion on getting banner views in near-realtime to the payments cluster, without pushing all the logs through kafkatee and filtering at our end [16:14:37] Are one or more of you free for a meeting friday at 10AM PST / 1AM EST ? [16:14:46] 10Analytics-Kanban, 10Patch-For-Review, 10User-Elukey: Add the prometheus jmx exporter to all the Hadoop daemons - https://phabricator.wikimedia.org/T177458#3890145 (10elukey) The hive server/metastore issue is more subtle: everything starts, the jmx agent returns metrics correctly but the daemons do not bin... [16:14:47] Or would some other time work better? [16:14:58] ejegg: this friday not so good, i wont' be working, but another time def! [16:15:51] ralgara: if you want to volunteer with us our easiest opportunities are not in hadoop/scala hive [16:16:03] ralgara: they are going to be in python/javscript to start [16:16:55] ottomata: is 10:30 AM PST Monday better? [16:17:27] ha ejegg that's good [16:17:32] thanks! [16:19:50] nuria_: I'll be happy to start with Python/JS [16:20:19] I'll just need some guidance on where to start [16:21:01] ottomata: oops, that's MLK day! [16:21:09] ejegg: yeah, just about to say [16:21:23] if you want to meet Friday, I can do that, and we can talk more next week (Tuesday)? [16:21:31] gcal suggests Tuesday at 9AM PST [16:21:40] works for me [16:21:51] ohhh mlk day [16:21:51] great! [16:21:53] ya that's fine [16:22:11] thanks folks [16:23:02] !log restarting kafka jumbo brokers to apply java.security certpath restrictions [16:23:03] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:25:07] I checked off 'let guests modify' in case SF folks want to add a meeting room. Not sure how many work from the office, or how big those rooms are. [16:29:49] ralgara: ok, let me find somethings where you can take a look , we use gerrit for crs so to commit code you will need credentials for it but that came late [16:30:34] milimetric: are you good to go to meeting with edward by yourself? [16:30:38] ejegg: nobody from our team is in SF :) [16:31:02] Yes nuria_ [16:31:10] well that makes it easy! Neither are we [16:33:04] 10Analytics-Cluster, 10Analytics-Kanban, 10Operations, 10Traffic, and 2 others: TLS security review of the Kafka stack - https://phabricator.wikimedia.org/T182993#3890215 (10Ottomata) Oook, I've set this on all jumbo Kafka brokers. @bblack anything else? [16:48:39] (03PS4) 10Mforns: [WIP] Improve WikiSelector [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/402387 (https://phabricator.wikimedia.org/T179530) [16:55:37] ejegg: i responded, let's please have a more targeted meeting to start [16:56:27] ejegg: i am afraid a meeting with that many people might not be very productive, we can have a smaller meeting to start and include more folks if needed later. [16:56:56] !log reboot analytics1048->50 for kernel updates [16:56:57] Logged the message at https://www.mediawiki.org/wiki/Analytics/Server_Admin_Log [16:58:00] * fdans is wondering what happens when our backend doesn't know about Kosovo but our UI does [16:58:00] OK, I'm about to attend daily stand-up with the FR-tech team, I'll ask who wants to be part of an initial smaller group [16:58:33] ralgara: sorry wanted to message here, how about: https://phabricator.wikimedia.org/T138505 [16:58:48] ralgara: this requires just github and will help the world at large [17:02:25] nuria_: Sounds like a good start. I'll take a look later today. [17:02:37] ralgara: excellent, many thanks [17:03:11] Though there seem to be many uap-* versions. The ticket doesn't specify which. Python? [17:03:17] nuria_ ^^ [17:04:05] 10Analytics, 10New-Readers, 10Easy: Split opera mini in proxy or turbo mode - https://phabricator.wikimedia.org/T138505#3890397 (10Nuria) We use UA parser in both python and java [17:04:09] ralgara: both python and java, added now to TT [17:06:35] nuria_: OK, sure. Should I just send a PR when ready? [17:06:44] ralgara: please [17:06:50] ralgara: and update ticket [17:06:59] ralgara: for which you will need a phab user [17:07:02] 10Analytics: Transform and Import Qualtrics Survey data - https://phabricator.wikimedia.org/T184626#3890422 (10Milimetric) [17:07:08] nuria_: will do. Thank you! [17:07:19] ralgara: https://www.mediawiki.org/wiki/Phabricator/Help [17:09:57] 10Analytics: Estimate how long a new Dashiki Layout for Qualtrics Survey data would take - https://phabricator.wikimedia.org/T184627#3890442 (10Milimetric) [17:14:31] 10Analytics-Cluster, 10Analytics-Kanban, 10Analytics-Wikistats, 10RESTBase-API, 10Services (done): Add "Pageviews by Country" AQS endpoint - https://phabricator.wikimedia.org/T181520#3890474 (10Nuria) Let's add a link to docs to this ticket [17:23:44] hi people. I am trying to access superset.wikimedia.org via my wikitech account and I can't. Can you help me figure out what's going on? (I get "The page isn't redirecting properly.) [17:24:32] (03CR) 10Milimetric: Translate g according to the y-axis width (031 comment) [analytics/wikistats2] - 10https://gerrit.wikimedia.org/r/403184 (https://phabricator.wikimedia.org/T184138) (owner: 10Fdans) [17:26:23] leila: use your shell login [17:26:26] not your wikitech name [17:26:28] ldap* [17:26:35] shell login, ldap password [17:26:37] same you'd use for hue [17:28:20] oh wait leila, i didn't make you an account [17:28:21] doing... [17:28:24] :D [17:29:00] username: leila [17:29:01] go for it! [17:33:46] fantastic. it works, ottomata, now. [18:22:49] 10Analytics: Private geo wiki data in new analytics stack - https://phabricator.wikimedia.org/T176996#3890738 (10Nuria) See notes: https://etherpad.wikimedia.org/p/analytics-geowiki [18:41:58] milimetric, mforns , joal : we have to task the geowiki in our next groskin, and hopefully get started in a week from now [18:42:07] k [18:42:27] definitely, we should get started asap [18:42:42] 10Analytics: Private geo wiki data in new analytics stack - https://phabricator.wikimedia.org/T176996#3890814 (10Nuria) Moving notes here: We're still setting up logins for that but we can look at it together in the meeting and see if it meets your needs The UI is not important, the numbers are important * L... [19:07:14] nuria_, milimetric +1 [19:07:40] I'm gonna head-off a-team [19:07:43] See you tomorrow [19:09:58] laterrrs [19:57:56] 10Analytics: Estimate how long a new Dashiki Layout for Qualtrics Survey data would take - https://phabricator.wikimedia.org/T184627#3890442 (10Nuria) Ping @egalvezwmf can you add mocks and also the audiences for this tool? [21:14:35] (03PS1) 10Milimetric: Point data and success flags to the same directory [analytics/refinery] - 10https://gerrit.wikimedia.org/r/403484 (https://phabricator.wikimedia.org/T170764) [22:11:27] fdans: let me know if you are arround [22:14:00] nuria_: holaa [22:14:49] fdans: hola, i was looking at pageview data [22:14:54] fdans: do we know that http://wikimedia.org/api/rest_v1/metrics/pageviews/top/pt.wikipedia/mobile-app/2015/11/01 [22:15:14] fdans: redirects to https://wikimedia.org/api/rest_v1/metrics/pageviews/top/pt.wikipedia/mobile-app/2015/11/01 [22:15:29] fdans: sorry, trying again [22:16:39] fdans: nevermind i think examples just need https [22:20:51] fdans: things are looking good [22:21:10] nuria_: glad I could rubberduck :) [22:21:38] nuria_: those urls are top articles tho, right? [22:22:49] fdans: top country examples do same, just all examples need https [22:23:21] makes sense [22:29:04] (03CR) 10Nuria: "I think the only way i have been able to test this is using hdfs user to run the job 1 time in prod setup" [analytics/refinery] - 10https://gerrit.wikimedia.org/r/403484 (https://phabricator.wikimedia.org/T170764) (owner: 10Milimetric) [22:29:14] (03CR) 10Nuria: [V: 032 C: 032] Point data and success flags to the same directory [analytics/refinery] - 10https://gerrit.wikimedia.org/r/403484 (https://phabricator.wikimedia.org/T170764) (owner: 10Milimetric)